2023-11-16T16:16:42.794 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-16T16:16:42.794 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_choffman@teuthology --name choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} --archive /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460916 --verbose -- /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460916/orig.config.yaml 2023-11-16T16:16:42.854 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-16T16:16:42.944 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460916 branch: wip-choffman-62793 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} email: null first_in_suite: false job_id: '7460916' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-choffman-62793 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 5e4a743ad7fd1422ebd6c4dc950a21a303279207 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 5e4a743ad7fd1422ebd6c4dc950a21a303279207 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-choffman-62793 sha1: 5e4a743ad7fd1422ebd6c4dc950a21a303279207 owner: scheduled_choffman@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4433 sha1: 5e4a743ad7fd1422ebd6c4dc950a21a303279207 sleep_before_teardown: 0 suite: fs:workload suite_branch: wip-choffman-62793 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_5e4a743ad7fd1422ebd6c4dc950a21a303279207/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 5e4a743ad7fd1422ebd6c4dc950a21a303279207 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi002.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi141.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi155.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - check-counter: {} - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - fs/misc teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-16_16:09:17 tube: smithi user: choffman verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1224587 2023-11-16T16:16:42.944 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_5e4a743ad7fd1422ebd6c4dc950a21a303279207/qa; will attempt to use it 2023-11-16T16:16:42.945 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_5e4a743ad7fd1422ebd6c4dc950a21a303279207/qa/tasks 2023-11-16T16:16:42.945 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-16T16:16:42.947 INFO:teuthology.task.internal:Checking packages... 2023-11-16T16:16:42.972 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '5e4a743ad7fd1422ebd6c4dc950a21a303279207' 2023-11-16T16:16:42.972 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-16T16:16:42.972 INFO:teuthology.packaging:ref: None 2023-11-16T16:16:42.972 INFO:teuthology.packaging:tag: None 2023-11-16T16:16:42.972 INFO:teuthology.packaging:branch: wip-choffman-62793 2023-11-16T16:16:42.972 INFO:teuthology.packaging:sha1: 5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:16:42.972 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-choffman-62793 2023-11-16T16:16:43.346 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-3905.g5e4a743a 2023-11-16T16:16:43.346 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-16T16:16:43.352 INFO:teuthology.task.internal:no buildpackages task found 2023-11-16T16:16:43.352 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-16T16:16:43.371 INFO:teuthology.task.internal:Saving configuration 2023-11-16T16:16:43.391 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-16T16:16:43.396 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T16:16:43.422 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460916', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 16:10:25.045193', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T16:16:43.447 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi141.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460916', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 16:10:25.046973', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T16:16:43.469 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460916', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 16:10:25.048360', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T16:16:43.469 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-16T16:16:43.476 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-11-16T16:16:43.476 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-11-16T16:16:43.476 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-11-16T16:16:43.476 INFO:teuthology.run_tasks:Running task console_log... 2023-11-16T16:16:43.601 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4793006b80>, signals=[15]) 2023-11-16T16:16:43.602 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-16T16:16:43.609 INFO:teuthology.task.internal:Opening connections... 2023-11-16T16:16:43.609 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T16:16:43.612 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T16:16:43.682 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi141.front.sepia.ceph.com 2023-11-16T16:16:43.684 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T16:16:43.751 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2023-11-16T16:16:43.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T16:16:43.818 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-16T16:16:43.825 DEBUG:teuthology.orchestra.run.smithi002:> uname -m 2023-11-16T16:16:43.847 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2023-11-16T16:16:43.848 DEBUG:teuthology.orchestra.run.smithi002:> cat /etc/os-release 2023-11-16T16:16:43.908 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="Red Hat Enterprise Linux" 2023-11-16T16:16:43.908 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="8.6 (Ootpa)" 2023-11-16T16:16:43.908 INFO:teuthology.orchestra.run.smithi002.stdout:ID="rhel" 2023-11-16T16:16:43.908 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE="fedora" 2023-11-16T16:16:43.908 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="8.6" 2023-11-16T16:16:43.908 INFO:teuthology.orchestra.run.smithi002.stdout:PLATFORM_ID="platform:el8" 2023-11-16T16:16:43.909 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-16T16:16:43.909 INFO:teuthology.orchestra.run.smithi002.stdout:ANSI_COLOR="0;31" 2023-11-16T16:16:43.909 INFO:teuthology.orchestra.run.smithi002.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-16T16:16:43.909 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://www.redhat.com/" 2023-11-16T16:16:43.909 INFO:teuthology.orchestra.run.smithi002.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-16T16:16:43.909 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-16T16:16:43.909 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:16:43.909 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-16T16:16:43.909 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-16T16:16:43.909 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-16T16:16:43.909 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-16T16:16:43.910 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2023-11-16T16:16:43.938 DEBUG:teuthology.orchestra.run.smithi141:> uname -m 2023-11-16T16:16:43.959 INFO:teuthology.orchestra.run.smithi141.stdout:x86_64 2023-11-16T16:16:43.959 DEBUG:teuthology.orchestra.run.smithi141:> cat /etc/os-release 2023-11-16T16:16:44.019 INFO:teuthology.orchestra.run.smithi141.stdout:NAME="Red Hat Enterprise Linux" 2023-11-16T16:16:44.019 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION="8.6 (Ootpa)" 2023-11-16T16:16:44.019 INFO:teuthology.orchestra.run.smithi141.stdout:ID="rhel" 2023-11-16T16:16:44.019 INFO:teuthology.orchestra.run.smithi141.stdout:ID_LIKE="fedora" 2023-11-16T16:16:44.019 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION_ID="8.6" 2023-11-16T16:16:44.020 INFO:teuthology.orchestra.run.smithi141.stdout:PLATFORM_ID="platform:el8" 2023-11-16T16:16:44.020 INFO:teuthology.orchestra.run.smithi141.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-16T16:16:44.020 INFO:teuthology.orchestra.run.smithi141.stdout:ANSI_COLOR="0;31" 2023-11-16T16:16:44.020 INFO:teuthology.orchestra.run.smithi141.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-16T16:16:44.020 INFO:teuthology.orchestra.run.smithi141.stdout:HOME_URL="https://www.redhat.com/" 2023-11-16T16:16:44.020 INFO:teuthology.orchestra.run.smithi141.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-16T16:16:44.020 INFO:teuthology.orchestra.run.smithi141.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-16T16:16:44.020 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-11-16T16:16:44.020 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-16T16:16:44.020 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-16T16:16:44.020 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-16T16:16:44.021 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-16T16:16:44.021 INFO:teuthology.lock.ops:Updating smithi141.front.sepia.ceph.com on lock server 2023-11-16T16:16:44.046 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2023-11-16T16:16:44.066 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2023-11-16T16:16:44.066 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2023-11-16T16:16:44.125 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="Red Hat Enterprise Linux" 2023-11-16T16:16:44.125 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="8.6 (Ootpa)" 2023-11-16T16:16:44.125 INFO:teuthology.orchestra.run.smithi155.stdout:ID="rhel" 2023-11-16T16:16:44.125 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="fedora" 2023-11-16T16:16:44.125 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="8.6" 2023-11-16T16:16:44.125 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el8" 2023-11-16T16:16:44.125 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-16T16:16:44.126 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2023-11-16T16:16:44.126 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-16T16:16:44.126 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://www.redhat.com/" 2023-11-16T16:16:44.126 INFO:teuthology.orchestra.run.smithi155.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-16T16:16:44.126 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-16T16:16:44.126 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-11-16T16:16:44.126 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-16T16:16:44.126 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-16T16:16:44.127 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-16T16:16:44.127 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-16T16:16:44.127 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2023-11-16T16:16:44.146 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-16T16:16:44.156 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-16T16:16:44.161 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-16T16:16:44.162 DEBUG:teuthology.orchestra.run.smithi002:> test '!' -e /home/ubuntu/cephtest 2023-11-16T16:16:44.164 DEBUG:teuthology.orchestra.run.smithi141:> test '!' -e /home/ubuntu/cephtest 2023-11-16T16:16:44.166 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2023-11-16T16:16:44.184 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-16T16:16:44.191 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-16T16:16:44.191 DEBUG:teuthology.orchestra.run.smithi002:> test -z $(ls -A /var/lib/ceph) 2023-11-16T16:16:44.224 DEBUG:teuthology.orchestra.run.smithi141:> test -z $(ls -A /var/lib/ceph) 2023-11-16T16:16:44.227 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2023-11-16T16:16:44.263 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-16T16:16:44.341 INFO:teuthology.run_tasks:Running task kernel... 2023-11-16T16:16:44.355 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2023-11-16T16:16:44.355 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2023-11-16T16:16:44.355 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2023-11-16T16:16:44.355 DEBUG:teuthology.orchestra.run.smithi002:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-16T16:16:44.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T16:16:44.375 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2023-11-16T16:16:44.605 DEBUG:teuthology.task.kernel:sha1 for client.0 is 5b73517e0e2fa132ea79290e5fc06ea5349b2288 2023-11-16T16:16:44.605 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5b73517e0e2fa132ea79290e5fc06ea5349b2288"... 2023-11-16T16:16:44.606 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2023-11-16T16:16:44.626 INFO:teuthology.orchestra.run.smithi002.stdout:4.18.0-372.9.1.el8.x86_64 2023-11-16T16:16:44.626 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-372.9.1.el8.x86_64 vs 5b73517e0e2fa132ea79290e5fc06ea5349b2288 2023-11-16T16:16:44.626 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-11-16T16:16:44.626 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum upgrade -y linux-firmware 2023-11-16T16:16:45.885 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2023-11-16T16:16:45.885 INFO:teuthology.orchestra.run.smithi002.stdout:Unable to read consumer identity 2023-11-16T16:16:47.281 INFO:teuthology.orchestra.run.smithi002.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2023-11-16T16:16:48.415 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-11-16T16:16:52.140 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 460 kB/s | 24 kB 00:00 2023-11-16T16:16:52.990 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-11-16T16:16:52.991 INFO:teuthology.orchestra.run.smithi002.stdout:Nothing to do. 2023-11-16T16:16:52.991 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-11-16T16:16:53.124 INFO:teuthology.task.kernel:Downloading kernel 5b73517e0e2fa132ea79290e5fc06ea5349b2288 on client.0... 2023-11-16T16:16:53.124 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5b73517e0e2fa132ea79290e5fc06ea5349b2288 2023-11-16T16:16:53.267 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/5b73517e0e2fa132ea79290e5fc06ea5349b2288/centos/8/flavors/default/x86_64/ 2023-11-16T16:16:53.268 DEBUG:teuthology.orchestra.run.smithi002:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_rc5_g5b73517e0e2f-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/5b73517e0e2fa132ea79290e5fc06ea5349b2288/centos/8/flavors/default/x86_64/ --input-file=- 2023-11-16T16:16:53.271 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi002.front.sepia.ceph.com to complete... 2023-11-16T16:17:17.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16 16:17:17 URL:https://4.chacra.ceph.com/r/kernel/testing/5b73517e0e2fa132ea79290e5fc06ea5349b2288/centos/8/flavors/default/x86_64/kernel-6.6.0_rc5_g5b73517e0e2f-1.x86_64.rpm [383129572/383129572] -> "/tmp/kernel.x86_64.rpm" [1] 2023-11-16T16:17:17.308 INFO:teuthology.orchestra.run.smithi002.stderr:FINISHED --2023-11-16 16:17:17-- 2023-11-16T16:17:17.308 INFO:teuthology.orchestra.run.smithi002.stderr:Total wall clock time: 24s 2023-11-16T16:17:17.308 INFO:teuthology.orchestra.run.smithi002.stderr:Downloaded: 1 files, 365M in 23s (15.6 MB/s) 2023-11-16T16:17:17.309 INFO:teuthology.task.kernel:Installing kernel 5b73517e0e2fa132ea79290e5fc06ea5349b2288 on client.0... 2023-11-16T16:17:17.309 DEBUG:teuthology.orchestra.run.smithi002:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-11-16T16:17:18.989 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying... ######################################## 2023-11-16T16:17:19.041 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing... ######################################## 2023-11-16T16:17:19.123 INFO:teuthology.orchestra.run.smithi002.stdout:Updating / installing... 2023-11-16T16:18:00.000 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-6.6.0_rc5_g5b73517e0e2f-1 ######################################## 2023-11-16T16:18:01.630 INFO:teuthology.orchestra.run.smithi002.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2023-11-16T16:18:26.357 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi002.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-11-16T16:18:26.357 DEBUG:teuthology.orchestra.run.smithi002:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-11-16T16:18:26.424 INFO:teuthology.orchestra.run.smithi002.stdout:/boot/System.map-6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:26.424 INFO:teuthology.orchestra.run.smithi002.stdout:/boot/config-6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:26.424 INFO:teuthology.orchestra.run.smithi002.stdout:/boot/vmlinuz-6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:26.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:26.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel 2023-11-16T16:18:26.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch 2023-11-16T16:18:26.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86 2023-11-16T16:18:26.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto 2023-11-16T16:18:26.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-11-16T16:18:26.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-11-16T16:18:26.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-11-16T16:18:26.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-11-16T16:18:26.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-11-16T16:18:26.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-11-16T16:18:26.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-11-16T16:18:26.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/crc32c-intel.ko 2023-11-16T16:18:26.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-11-16T16:18:26.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-11-16T16:18:26.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-11-16T16:18:26.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-11-16T16:18:26.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/serpent-avx2.ko 2023-11-16T16:18:26.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-11-16T16:18:26.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-11-16T16:18:26.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-11-16T16:18:26.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-11-16T16:18:26.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-11-16T16:18:26.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-11-16T16:18:26.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-11-16T16:18:26.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kernel 2023-11-16T16:18:26.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kernel/cpu 2023-11-16T16:18:26.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kernel/cpu/mce 2023-11-16T16:18:26.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-11-16T16:18:26.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kvm 2023-11-16T16:18:26.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kvm/kvm-amd.ko 2023-11-16T16:18:26.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kvm/kvm-intel.ko 2023-11-16T16:18:26.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/arch/x86/kvm/kvm.ko 2023-11-16T16:18:26.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto 2023-11-16T16:18:26.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/ansi_cprng.ko 2023-11-16T16:18:26.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/anubis.ko 2023-11-16T16:18:26.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/arc4.ko 2023-11-16T16:18:26.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx 2023-11-16T16:18:26.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx/async_memcpy.ko 2023-11-16T16:18:26.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx/async_pq.ko 2023-11-16T16:18:26.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx/async_raid6_recov.ko 2023-11-16T16:18:26.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx/async_tx.ko 2023-11-16T16:18:26.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx/async_xor.ko 2023-11-16T16:18:26.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/async_tx/raid6test.ko 2023-11-16T16:18:26.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/authenc.ko 2023-11-16T16:18:26.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/authencesn.ko 2023-11-16T16:18:26.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/blake2b_generic.ko 2023-11-16T16:18:26.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/blowfish_common.ko 2023-11-16T16:18:26.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/blowfish_generic.ko 2023-11-16T16:18:26.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/camellia_generic.ko 2023-11-16T16:18:26.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/cast5_generic.ko 2023-11-16T16:18:26.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/cast6_generic.ko 2023-11-16T16:18:26.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/cast_common.ko 2023-11-16T16:18:26.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/ccm.ko 2023-11-16T16:18:26.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/cmac.ko 2023-11-16T16:18:26.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/crc32_generic.ko 2023-11-16T16:18:26.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/crypto_engine.ko 2023-11-16T16:18:26.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/crypto_user.ko 2023-11-16T16:18:26.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/cts.ko 2023-11-16T16:18:26.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/des_generic.ko 2023-11-16T16:18:26.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/dh_generic.ko 2023-11-16T16:18:26.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/echainiv.ko 2023-11-16T16:18:26.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/essiv.ko 2023-11-16T16:18:26.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/fcrypt.ko 2023-11-16T16:18:26.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/khazad.ko 2023-11-16T16:18:26.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/lz4.ko 2023-11-16T16:18:26.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/lz4hc.ko 2023-11-16T16:18:26.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/md4.ko 2023-11-16T16:18:26.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/michael_mic.ko 2023-11-16T16:18:26.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/pcbc.ko 2023-11-16T16:18:26.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/pcrypt.ko 2023-11-16T16:18:26.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/rmd160.ko 2023-11-16T16:18:26.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/seed.ko 2023-11-16T16:18:26.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/serpent_generic.ko 2023-11-16T16:18:26.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/tcrypt.ko 2023-11-16T16:18:26.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/tea.ko 2023-11-16T16:18:26.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/twofish_common.ko 2023-11-16T16:18:26.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/twofish_generic.ko 2023-11-16T16:18:26.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/vmac.ko 2023-11-16T16:18:26.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/wp512.ko 2023-11-16T16:18:26.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/xcbc.ko 2023-11-16T16:18:26.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/xor.ko 2023-11-16T16:18:26.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/crypto/xxhash_generic.ko 2023-11-16T16:18:26.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers 2023-11-16T16:18:26.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi 2023-11-16T16:18:26.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/acpi_ipmi.ko 2023-11-16T16:18:26.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/acpi_pad.ko 2023-11-16T16:18:26.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/custom_method.ko 2023-11-16T16:18:26.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/ec_sys.ko 2023-11-16T16:18:26.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/platform_profile.ko 2023-11-16T16:18:26.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/sbs.ko 2023-11-16T16:18:26.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/sbshc.ko 2023-11-16T16:18:26.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/acpi/video.ko 2023-11-16T16:18:26.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata 2023-11-16T16:18:26.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/acard-ahci.ko 2023-11-16T16:18:26.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/ahci_platform.ko 2023-11-16T16:18:26.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/ata_generic.ko 2023-11-16T16:18:26.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/libahci_platform.ko 2023-11-16T16:18:26.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_acpi.ko 2023-11-16T16:18:26.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_ali.ko 2023-11-16T16:18:26.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_amd.ko 2023-11-16T16:18:26.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_artop.ko 2023-11-16T16:18:26.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_atiixp.ko 2023-11-16T16:18:26.433 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_atp867x.ko 2023-11-16T16:18:26.433 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_cmd640.ko 2023-11-16T16:18:26.433 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_cmd64x.ko 2023-11-16T16:18:26.433 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_cypress.ko 2023-11-16T16:18:26.433 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_efar.ko 2023-11-16T16:18:26.433 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_hpt366.ko 2023-11-16T16:18:26.433 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_hpt37x.ko 2023-11-16T16:18:26.433 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_hpt3x2n.ko 2023-11-16T16:18:26.433 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_hpt3x3.ko 2023-11-16T16:18:26.433 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_it8213.ko 2023-11-16T16:18:26.433 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_it821x.ko 2023-11-16T16:18:26.433 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_jmicron.ko 2023-11-16T16:18:26.433 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_marvell.ko 2023-11-16T16:18:26.434 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_mpiix.ko 2023-11-16T16:18:26.434 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_netcell.ko 2023-11-16T16:18:26.434 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_ninja32.ko 2023-11-16T16:18:26.434 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_ns87410.ko 2023-11-16T16:18:26.434 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_ns87415.ko 2023-11-16T16:18:26.434 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_oldpiix.ko 2023-11-16T16:18:26.434 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_opti.ko 2023-11-16T16:18:26.434 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_optidma.ko 2023-11-16T16:18:26.434 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_pcmcia.ko 2023-11-16T16:18:26.434 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_pdc2027x.ko 2023-11-16T16:18:26.435 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-11-16T16:18:26.435 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_piccolo.ko 2023-11-16T16:18:26.435 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_rdc.ko 2023-11-16T16:18:26.435 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_sch.ko 2023-11-16T16:18:26.435 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_serverworks.ko 2023-11-16T16:18:26.435 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_sil680.ko 2023-11-16T16:18:26.435 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_sis.ko 2023-11-16T16:18:26.435 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_sl82c105.ko 2023-11-16T16:18:26.435 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_triflex.ko 2023-11-16T16:18:26.435 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pata_via.ko 2023-11-16T16:18:26.435 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/pdc_adma.ko 2023-11-16T16:18:26.435 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_inic162x.ko 2023-11-16T16:18:26.435 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_mv.ko 2023-11-16T16:18:26.436 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_nv.ko 2023-11-16T16:18:26.436 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_promise.ko 2023-11-16T16:18:26.436 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_qstor.ko 2023-11-16T16:18:26.436 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_sil.ko 2023-11-16T16:18:26.436 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_sil24.ko 2023-11-16T16:18:26.436 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_sis.ko 2023-11-16T16:18:26.436 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_svw.ko 2023-11-16T16:18:26.436 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_sx4.ko 2023-11-16T16:18:26.436 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_uli.ko 2023-11-16T16:18:26.436 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_via.ko 2023-11-16T16:18:26.436 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ata/sata_vsc.ko 2023-11-16T16:18:26.436 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/auxdisplay 2023-11-16T16:18:26.437 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/auxdisplay/ks0108.ko 2023-11-16T16:18:26.437 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/base 2023-11-16T16:18:26.437 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/base/regmap 2023-11-16T16:18:26.437 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/base/regmap/regmap-i2c.ko 2023-11-16T16:18:26.437 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/base/regmap/regmap-mmio.ko 2023-11-16T16:18:26.437 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/bcma 2023-11-16T16:18:26.437 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/bcma/bcma.ko 2023-11-16T16:18:26.437 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block 2023-11-16T16:18:26.437 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/aoe 2023-11-16T16:18:26.437 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/aoe/aoe.ko 2023-11-16T16:18:26.437 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/brd.ko 2023-11-16T16:18:26.437 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/drbd 2023-11-16T16:18:26.438 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/drbd/drbd.ko 2023-11-16T16:18:26.438 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/floppy.ko 2023-11-16T16:18:26.438 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/loop.ko 2023-11-16T16:18:26.438 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/mtip32xx 2023-11-16T16:18:26.438 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-11-16T16:18:26.438 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/nbd.ko 2023-11-16T16:18:26.438 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/null_blk 2023-11-16T16:18:26.438 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/null_blk/null_blk.ko 2023-11-16T16:18:26.438 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/pktcdvd.ko 2023-11-16T16:18:26.438 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/rbd.ko 2023-11-16T16:18:26.438 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/virtio_blk.ko 2023-11-16T16:18:26.438 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/xen-blkback 2023-11-16T16:18:26.439 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-11-16T16:18:26.439 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/xen-blkfront.ko 2023-11-16T16:18:26.439 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/zram 2023-11-16T16:18:26.439 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/block/zram/zram.ko 2023-11-16T16:18:26.439 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char 2023-11-16T16:18:26.439 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hangcheck-timer.ko 2023-11-16T16:18:26.439 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hw_random 2023-11-16T16:18:26.439 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hw_random/amd-rng.ko 2023-11-16T16:18:26.439 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hw_random/intel-rng.ko 2023-11-16T16:18:26.439 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-11-16T16:18:26.439 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hw_random/via-rng.ko 2023-11-16T16:18:26.439 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/hw_random/virtio-rng.ko 2023-11-16T16:18:26.440 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi 2023-11-16T16:18:26.440 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-11-16T16:18:26.440 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-11-16T16:18:26.440 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-11-16T16:18:26.440 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi/ipmi_si.ko 2023-11-16T16:18:26.440 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-11-16T16:18:26.440 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-11-16T16:18:26.440 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/lp.ko 2023-11-16T16:18:26.440 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/mwave 2023-11-16T16:18:26.440 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/mwave/mwave.ko 2023-11-16T16:18:26.440 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/ppdev.ko 2023-11-16T16:18:26.440 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tlclk.ko 2023-11-16T16:18:26.441 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm 2023-11-16T16:18:26.441 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm/tpm.ko 2023-11-16T16:18:26.441 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm/tpm_atmel.ko 2023-11-16T16:18:26.441 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm/tpm_infineon.ko 2023-11-16T16:18:26.441 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm/tpm_nsc.ko 2023-11-16T16:18:26.441 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm/tpm_tis.ko 2023-11-16T16:18:26.441 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-11-16T16:18:26.441 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/uv_mmtimer.ko 2023-11-16T16:18:26.441 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/char/virtio_console.ko 2023-11-16T16:18:26.441 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq 2023-11-16T16:18:26.441 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-11-16T16:18:26.441 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-11-16T16:18:26.442 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq/p4-clockmod.ko 2023-11-16T16:18:26.442 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-11-16T16:18:26.442 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq/powernow-k8.ko 2023-11-16T16:18:26.442 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/cpufreq/speedstep-lib.ko 2023-11-16T16:18:26.442 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto 2023-11-16T16:18:26.442 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/ccp 2023-11-16T16:18:26.442 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-11-16T16:18:26.442 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/ccp/ccp.ko 2023-11-16T16:18:26.442 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/intel 2023-11-16T16:18:26.442 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/intel/qat 2023-11-16T16:18:26.442 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/intel/qat/qat_common 2023-11-16T16:18:26.442 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-11-16T16:18:26.443 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-11-16T16:18:26.443 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-11-16T16:18:26.443 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/padlock-aes.ko 2023-11-16T16:18:26.443 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/padlock-sha.ko 2023-11-16T16:18:26.443 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/virtio 2023-11-16T16:18:26.443 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-11-16T16:18:26.443 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dca 2023-11-16T16:18:26.443 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dca/dca.ko 2023-11-16T16:18:26.443 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/devfreq 2023-11-16T16:18:26.443 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-11-16T16:18:26.443 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dma 2023-11-16T16:18:26.443 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dma/dw 2023-11-16T16:18:26.444 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dma/dw/dw_dmac.ko 2023-11-16T16:18:26.444 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dma/ioat 2023-11-16T16:18:26.444 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/dma/ioat/ioatdma.ko 2023-11-16T16:18:26.444 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac 2023-11-16T16:18:26.444 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/amd64_edac.ko 2023-11-16T16:18:26.444 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/e752x_edac.ko 2023-11-16T16:18:26.444 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/edac_mce_amd.ko 2023-11-16T16:18:26.444 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i3000_edac.ko 2023-11-16T16:18:26.444 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i3200_edac.ko 2023-11-16T16:18:26.444 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i5100_edac.ko 2023-11-16T16:18:26.444 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i5400_edac.ko 2023-11-16T16:18:26.444 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i7300_edac.ko 2023-11-16T16:18:26.445 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i7core_edac.ko 2023-11-16T16:18:26.445 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/i82975x_edac.ko 2023-11-16T16:18:26.445 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/ie31200_edac.ko 2023-11-16T16:18:26.445 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/sb_edac.ko 2023-11-16T16:18:26.445 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/edac/x38_edac.ko 2023-11-16T16:18:26.445 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/firmware 2023-11-16T16:18:26.445 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/firmware/edd.ko 2023-11-16T16:18:26.445 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/firmware/iscsi_ibft.ko 2023-11-16T16:18:26.445 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/gpio 2023-11-16T16:18:26.445 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/gpio/gpio-ich.ko 2023-11-16T16:18:26.445 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/gpio/gpio-viperboard.ko 2023-11-16T16:18:26.445 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid 2023-11-16T16:18:26.446 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-appleir.ko 2023-11-16T16:18:26.446 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-aureal.ko 2023-11-16T16:18:26.446 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-axff.ko 2023-11-16T16:18:26.446 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-dr.ko 2023-11-16T16:18:26.446 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-elecom.ko 2023-11-16T16:18:26.446 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-elo.ko 2023-11-16T16:18:26.446 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-emsff.ko 2023-11-16T16:18:26.446 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-gaff.ko 2023-11-16T16:18:26.446 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-gt683r.ko 2023-11-16T16:18:26.446 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-gyration.ko 2023-11-16T16:18:26.446 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-holtek-kbd.ko 2023-11-16T16:18:26.447 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-holtek-mouse.ko 2023-11-16T16:18:26.447 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-holtekff.ko 2023-11-16T16:18:26.447 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-hyperv.ko 2023-11-16T16:18:26.447 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-icade.ko 2023-11-16T16:18:26.447 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-keytouch.ko 2023-11-16T16:18:26.447 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-kye.ko 2023-11-16T16:18:26.447 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-lcpower.ko 2023-11-16T16:18:26.447 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-led.ko 2023-11-16T16:18:26.447 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-lenovo.ko 2023-11-16T16:18:26.447 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-logitech-dj.ko 2023-11-16T16:18:26.447 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-11-16T16:18:26.447 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-multitouch.ko 2023-11-16T16:18:26.448 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-ortek.ko 2023-11-16T16:18:26.448 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-penmount.ko 2023-11-16T16:18:26.448 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-petalynx.ko 2023-11-16T16:18:26.448 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-picolcd.ko 2023-11-16T16:18:26.448 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-pl.ko 2023-11-16T16:18:26.448 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-plantronics.ko 2023-11-16T16:18:26.448 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-primax.ko 2023-11-16T16:18:26.448 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-rmi.ko 2023-11-16T16:18:26.448 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-arvo.ko 2023-11-16T16:18:26.448 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-common.ko 2023-11-16T16:18:26.448 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-isku.ko 2023-11-16T16:18:26.448 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-kone.ko 2023-11-16T16:18:26.448 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-11-16T16:18:26.449 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-konepure.ko 2023-11-16T16:18:26.449 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-11-16T16:18:26.449 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-lua.ko 2023-11-16T16:18:26.449 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-pyra.ko 2023-11-16T16:18:26.449 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-ryos.ko 2023-11-16T16:18:26.449 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat-savu.ko 2023-11-16T16:18:26.449 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-roccat.ko 2023-11-16T16:18:26.449 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-saitek.ko 2023-11-16T16:18:26.449 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-samsung.ko 2023-11-16T16:18:26.449 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-sensor-hub.ko 2023-11-16T16:18:26.449 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-sjoy.ko 2023-11-16T16:18:26.449 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-sony.ko 2023-11-16T16:18:26.450 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-speedlink.ko 2023-11-16T16:18:26.450 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-steelseries.ko 2023-11-16T16:18:26.450 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-sunplus.ko 2023-11-16T16:18:26.450 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-thrustmaster.ko 2023-11-16T16:18:26.450 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-tivo.ko 2023-11-16T16:18:26.450 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-tmff.ko 2023-11-16T16:18:26.450 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-topseed.ko 2023-11-16T16:18:26.450 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-twinhan.ko 2023-11-16T16:18:26.450 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-uclogic.ko 2023-11-16T16:18:26.450 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-waltop.ko 2023-11-16T16:18:26.450 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-wiimote.ko 2023-11-16T16:18:26.451 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-xinmo.ko 2023-11-16T16:18:26.451 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-zpff.ko 2023-11-16T16:18:26.451 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/hid-zydacron.ko 2023-11-16T16:18:26.451 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/uhid.ko 2023-11-16T16:18:26.451 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hid/wacom.ko 2023-11-16T16:18:26.451 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hv 2023-11-16T16:18:26.451 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hv/hv_balloon.ko 2023-11-16T16:18:26.451 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hv/hv_utils.ko 2023-11-16T16:18:26.451 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hv/hv_vmbus.ko 2023-11-16T16:18:26.451 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon 2023-11-16T16:18:26.451 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/abituguru.ko 2023-11-16T16:18:26.451 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/abituguru3.ko 2023-11-16T16:18:26.451 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/acpi_power_meter.ko 2023-11-16T16:18:26.452 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ad7414.ko 2023-11-16T16:18:26.452 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ad7418.ko 2023-11-16T16:18:26.452 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adc128d818.ko 2023-11-16T16:18:26.452 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adm1025.ko 2023-11-16T16:18:26.452 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adm1026.ko 2023-11-16T16:18:26.452 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adm1029.ko 2023-11-16T16:18:26.452 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adm1031.ko 2023-11-16T16:18:26.452 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adm9240.ko 2023-11-16T16:18:26.452 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ads7828.ko 2023-11-16T16:18:26.452 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adt7410.ko 2023-11-16T16:18:26.452 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adt7411.ko 2023-11-16T16:18:26.452 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adt7462.ko 2023-11-16T16:18:26.453 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adt7470.ko 2023-11-16T16:18:26.453 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adt7475.ko 2023-11-16T16:18:26.453 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/adt7x10.ko 2023-11-16T16:18:26.453 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/amc6821.ko 2023-11-16T16:18:26.453 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/applesmc.ko 2023-11-16T16:18:26.453 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/asb100.ko 2023-11-16T16:18:26.453 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/asc7621.ko 2023-11-16T16:18:26.453 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/asus_atk0110.ko 2023-11-16T16:18:26.453 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/atxp1.ko 2023-11-16T16:18:26.453 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/coretemp.ko 2023-11-16T16:18:26.453 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-11-16T16:18:26.453 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/dme1737.ko 2023-11-16T16:18:26.454 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ds1621.ko 2023-11-16T16:18:26.454 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ds620.ko 2023-11-16T16:18:26.454 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/emc1403.ko 2023-11-16T16:18:26.454 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/emc6w201.ko 2023-11-16T16:18:26.454 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/f71805f.ko 2023-11-16T16:18:26.454 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/f71882fg.ko 2023-11-16T16:18:26.454 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/f75375s.ko 2023-11-16T16:18:26.454 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/fam15h_power.ko 2023-11-16T16:18:26.454 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/fschmd.ko 2023-11-16T16:18:26.454 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/g760a.ko 2023-11-16T16:18:26.454 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/g762.ko 2023-11-16T16:18:26.454 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/gl518sm.ko 2023-11-16T16:18:26.455 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/gl520sm.ko 2023-11-16T16:18:26.455 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/hwmon-vid.ko 2023-11-16T16:18:26.455 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/i5500_temp.ko 2023-11-16T16:18:26.455 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/i5k_amb.ko 2023-11-16T16:18:26.455 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ibmaem.ko 2023-11-16T16:18:26.455 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ibmpex.ko 2023-11-16T16:18:26.455 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ina209.ko 2023-11-16T16:18:26.455 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ina2xx.ko 2023-11-16T16:18:26.455 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/it87.ko 2023-11-16T16:18:26.455 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/k10temp.ko 2023-11-16T16:18:26.455 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/k8temp.ko 2023-11-16T16:18:26.455 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lineage-pem.ko 2023-11-16T16:18:26.456 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm63.ko 2023-11-16T16:18:26.456 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm73.ko 2023-11-16T16:18:26.456 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm75.ko 2023-11-16T16:18:26.456 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm77.ko 2023-11-16T16:18:26.456 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm78.ko 2023-11-16T16:18:26.456 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm80.ko 2023-11-16T16:18:26.456 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm83.ko 2023-11-16T16:18:26.456 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm85.ko 2023-11-16T16:18:26.456 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm87.ko 2023-11-16T16:18:26.456 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm90.ko 2023-11-16T16:18:26.456 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm92.ko 2023-11-16T16:18:26.456 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm93.ko 2023-11-16T16:18:26.457 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm95234.ko 2023-11-16T16:18:26.457 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm95241.ko 2023-11-16T16:18:26.457 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/lm95245.ko 2023-11-16T16:18:26.457 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc2945.ko 2023-11-16T16:18:26.457 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc4151.ko 2023-11-16T16:18:26.457 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc4215.ko 2023-11-16T16:18:26.457 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc4222.ko 2023-11-16T16:18:26.457 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc4245.ko 2023-11-16T16:18:26.457 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc4260.ko 2023-11-16T16:18:26.457 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/ltc4261.ko 2023-11-16T16:18:26.457 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max16065.ko 2023-11-16T16:18:26.457 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max1619.ko 2023-11-16T16:18:26.457 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max1668.ko 2023-11-16T16:18:26.458 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max197.ko 2023-11-16T16:18:26.458 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max6639.ko 2023-11-16T16:18:26.458 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max6650.ko 2023-11-16T16:18:26.458 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/max6697.ko 2023-11-16T16:18:26.458 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/mcp3021.ko 2023-11-16T16:18:26.458 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/nct6683.ko 2023-11-16T16:18:26.458 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/nct6775-core.ko 2023-11-16T16:18:26.458 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/nct6775.ko 2023-11-16T16:18:26.458 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/nct7802.ko 2023-11-16T16:18:26.458 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pc87360.ko 2023-11-16T16:18:26.458 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pc87427.ko 2023-11-16T16:18:26.458 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pcf8591.ko 2023-11-16T16:18:26.459 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus 2023-11-16T16:18:26.459 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-11-16T16:18:26.459 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-11-16T16:18:26.459 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-11-16T16:18:26.459 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/max16064.ko 2023-11-16T16:18:26.459 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/max34440.ko 2023-11-16T16:18:26.459 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/max8688.ko 2023-11-16T16:18:26.459 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-11-16T16:18:26.459 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-11-16T16:18:26.459 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-11-16T16:18:26.459 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-11-16T16:18:26.459 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-11-16T16:18:26.460 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-11-16T16:18:26.460 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/powr1220.ko 2023-11-16T16:18:26.460 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/sch5627.ko 2023-11-16T16:18:26.460 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/sch5636.ko 2023-11-16T16:18:26.460 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/sch56xx-common.ko 2023-11-16T16:18:26.460 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/sht15.ko 2023-11-16T16:18:26.460 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/sht21.ko 2023-11-16T16:18:26.460 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/shtc1.ko 2023-11-16T16:18:26.460 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/sis5595.ko 2023-11-16T16:18:26.460 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/smsc47b397.ko 2023-11-16T16:18:26.460 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/smsc47m1.ko 2023-11-16T16:18:26.460 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/smsc47m192.ko 2023-11-16T16:18:26.461 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/thmc50.ko 2023-11-16T16:18:26.461 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/tmp102.ko 2023-11-16T16:18:26.461 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/tmp103.ko 2023-11-16T16:18:26.461 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/tmp401.ko 2023-11-16T16:18:26.461 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/tmp421.ko 2023-11-16T16:18:26.461 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/via-cputemp.ko 2023-11-16T16:18:26.461 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/via686a.ko 2023-11-16T16:18:26.461 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/vt1211.ko 2023-11-16T16:18:26.461 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/vt8231.ko 2023-11-16T16:18:26.461 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83627ehf.ko 2023-11-16T16:18:26.461 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83627hf.ko 2023-11-16T16:18:26.461 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83781d.ko 2023-11-16T16:18:26.461 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83791d.ko 2023-11-16T16:18:26.462 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83792d.ko 2023-11-16T16:18:26.462 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83793.ko 2023-11-16T16:18:26.462 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83795.ko 2023-11-16T16:18:26.462 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83l785ts.ko 2023-11-16T16:18:26.462 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/hwmon/w83l786ng.ko 2023-11-16T16:18:26.462 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c 2023-11-16T16:18:26.462 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/algos 2023-11-16T16:18:26.462 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-11-16T16:18:26.462 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-11-16T16:18:26.462 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses 2023-11-16T16:18:26.462 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-11-16T16:18:26.462 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-11-16T16:18:26.463 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-11-16T16:18:26.463 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-11-16T16:18:26.463 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-11-16T16:18:26.463 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-11-16T16:18:26.463 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-11-16T16:18:26.463 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-11-16T16:18:26.463 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-i801.ko 2023-11-16T16:18:26.463 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-isch.ko 2023-11-16T16:18:26.463 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-11-16T16:18:26.463 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-11-16T16:18:26.463 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-11-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-parport.ko 2023-11-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-11-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-11-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-11-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-11-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-11-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-11-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-via.ko 2023-11-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-11-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-11-16T16:18:26.465 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/i2c-dev.ko 2023-11-16T16:18:26.466 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/i2c-mux.ko 2023-11-16T16:18:26.466 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-11-16T16:18:26.466 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/i2c-smbus.ko 2023-11-16T16:18:26.466 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/i2c/i2c-stub.ko 2023-11-16T16:18:26.466 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband 2023-11-16T16:18:26.466 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core 2023-11-16T16:18:26.466 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/ib_cm.ko 2023-11-16T16:18:26.466 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/ib_core.ko 2023-11-16T16:18:26.466 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/ib_umad.ko 2023-11-16T16:18:26.466 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-11-16T16:18:26.466 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/iw_cm.ko 2023-11-16T16:18:26.467 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/rdma_cm.ko 2023-11-16T16:18:26.467 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-11-16T16:18:26.467 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw 2023-11-16T16:18:26.467 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/cxgb4 2023-11-16T16:18:26.467 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-11-16T16:18:26.467 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/mlx4 2023-11-16T16:18:26.467 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-11-16T16:18:26.467 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/mlx5 2023-11-16T16:18:26.467 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-11-16T16:18:26.467 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/mthca 2023-11-16T16:18:26.467 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-11-16T16:18:26.467 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp 2023-11-16T16:18:26.468 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/ipoib 2023-11-16T16:18:26.468 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-11-16T16:18:26.468 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/iser 2023-11-16T16:18:26.468 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-11-16T16:18:26.468 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/isert 2023-11-16T16:18:26.468 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-11-16T16:18:26.468 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/srp 2023-11-16T16:18:26.468 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-11-16T16:18:26.468 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/srpt 2023-11-16T16:18:26.468 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-11-16T16:18:26.468 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input 2023-11-16T16:18:26.468 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/gameport 2023-11-16T16:18:26.469 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-11-16T16:18:26.469 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/gameport/fm801-gp.ko 2023-11-16T16:18:26.469 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/gameport/gameport.ko 2023-11-16T16:18:26.469 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/joydev.ko 2023-11-16T16:18:26.469 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/keyboard 2023-11-16T16:18:26.469 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/keyboard/gpio_keys.ko 2023-11-16T16:18:26.469 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse 2023-11-16T16:18:26.469 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/appletouch.ko 2023-11-16T16:18:26.469 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/bcm5974.ko 2023-11-16T16:18:26.469 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/cyapatp.ko 2023-11-16T16:18:26.469 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/elan_i2c.ko 2023-11-16T16:18:26.469 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/sermouse.ko 2023-11-16T16:18:26.469 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-11-16T16:18:26.470 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/synaptics_usb.ko 2023-11-16T16:18:26.470 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/mouse/vsxxxaa.ko 2023-11-16T16:18:26.470 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/rmi4 2023-11-16T16:18:26.470 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/rmi4/rmi_core.ko 2023-11-16T16:18:26.470 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/serio 2023-11-16T16:18:26.470 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/serio/altera_ps2.ko 2023-11-16T16:18:26.470 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/serio/arc_ps2.ko 2023-11-16T16:18:26.470 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-11-16T16:18:26.470 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/serio/serio_raw.ko 2023-11-16T16:18:26.470 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/input/sparse-keymap.ko 2023-11-16T16:18:26.470 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/iommu 2023-11-16T16:18:26.470 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/iommu/amd 2023-11-16T16:18:26.471 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/iommu/amd/iommu_v2.ko 2023-11-16T16:18:26.471 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds 2023-11-16T16:18:26.471 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/leds-blinkm.ko 2023-11-16T16:18:26.471 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/leds-lm3530.ko 2023-11-16T16:18:26.471 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/leds-lp3944.ko 2023-11-16T16:18:26.471 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/leds-ss4200.ko 2023-11-16T16:18:26.471 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger 2023-11-16T16:18:26.471 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-11-16T16:18:26.471 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-11-16T16:18:26.471 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-11-16T16:18:26.471 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-11-16T16:18:26.471 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-11-16T16:18:26.471 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-11-16T16:18:26.472 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-11-16T16:18:26.472 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-11-16T16:18:26.472 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md 2023-11-16T16:18:26.472 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/bcache 2023-11-16T16:18:26.472 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/bcache/bcache.ko 2023-11-16T16:18:26.472 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-bio-prison.ko 2023-11-16T16:18:26.472 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-cache-smq.ko 2023-11-16T16:18:26.472 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-cache.ko 2023-11-16T16:18:26.472 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-crypt.ko 2023-11-16T16:18:26.472 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-delay.ko 2023-11-16T16:18:26.472 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-flakey.ko 2023-11-16T16:18:26.472 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-log-userspace.ko 2023-11-16T16:18:26.473 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-multipath.ko 2023-11-16T16:18:26.473 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-queue-length.ko 2023-11-16T16:18:26.473 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-raid.ko 2023-11-16T16:18:26.473 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-round-robin.ko 2023-11-16T16:18:26.473 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-service-time.ko 2023-11-16T16:18:26.473 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-switch.ko 2023-11-16T16:18:26.473 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-thin-pool.ko 2023-11-16T16:18:26.473 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/dm-verity.ko 2023-11-16T16:18:26.473 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/faulty.ko 2023-11-16T16:18:26.473 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/linear.ko 2023-11-16T16:18:26.473 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/multipath.ko 2023-11-16T16:18:26.473 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/persistent-data 2023-11-16T16:18:26.473 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-11-16T16:18:26.474 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/raid0.ko 2023-11-16T16:18:26.474 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/raid1.ko 2023-11-16T16:18:26.474 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/raid10.ko 2023-11-16T16:18:26.474 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/md/raid456.ko 2023-11-16T16:18:26.474 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message 2023-11-16T16:18:26.474 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion 2023-11-16T16:18:26.474 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptbase.ko 2023-11-16T16:18:26.474 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptctl.ko 2023-11-16T16:18:26.474 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptfc.ko 2023-11-16T16:18:26.474 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptlan.ko 2023-11-16T16:18:26.474 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptsas.ko 2023-11-16T16:18:26.474 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptscsih.ko 2023-11-16T16:18:26.475 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/message/fusion/mptspi.ko 2023-11-16T16:18:26.475 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd 2023-11-16T16:18:26.475 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/lpc_ich.ko 2023-11-16T16:18:26.475 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/lpc_sch.ko 2023-11-16T16:18:26.475 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/mfd-core.ko 2023-11-16T16:18:26.475 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/sm501.ko 2023-11-16T16:18:26.475 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/viperboard.ko 2023-11-16T16:18:26.475 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/vx855.ko 2023-11-16T16:18:26.475 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/mfd/wl1273-core.ko 2023-11-16T16:18:26.475 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc 2023-11-16T16:18:26.475 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/altera-stapl 2023-11-16T16:18:26.475 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-11-16T16:18:26.475 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/apds9802als.ko 2023-11-16T16:18:26.476 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/apds990x.ko 2023-11-16T16:18:26.476 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/bh1770glc.ko 2023-11-16T16:18:26.476 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/cb710 2023-11-16T16:18:26.476 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/cb710/cb710.ko 2023-11-16T16:18:26.476 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/eeprom 2023-11-16T16:18:26.476 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/eeprom/at24.ko 2023-11-16T16:18:26.476 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/eeprom/eeprom.ko 2023-11-16T16:18:26.476 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-11-16T16:18:26.476 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/eeprom/max6875.ko 2023-11-16T16:18:26.476 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/enclosure.ko 2023-11-16T16:18:26.476 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/hpilo.ko 2023-11-16T16:18:26.476 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/isl29003.ko 2023-11-16T16:18:26.476 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/isl29020.ko 2023-11-16T16:18:26.477 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/lis3lv02d 2023-11-16T16:18:26.477 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-11-16T16:18:26.477 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-11-16T16:18:26.477 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/mei 2023-11-16T16:18:26.477 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/mei/mei-me.ko 2023-11-16T16:18:26.477 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/mei/mei-txe.ko 2023-11-16T16:18:26.477 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/mei/mei.ko 2023-11-16T16:18:26.477 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/sgi-gru 2023-11-16T16:18:26.477 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/sgi-gru/gru.ko 2023-11-16T16:18:26.477 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/sgi-xp 2023-11-16T16:18:26.477 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/sgi-xp/xp.ko 2023-11-16T16:18:26.477 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/sgi-xp/xpc.ko 2023-11-16T16:18:26.477 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-11-16T16:18:26.478 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/tifm_7xx1.ko 2023-11-16T16:18:26.478 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/tifm_core.ko 2023-11-16T16:18:26.478 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/tsl2550.ko 2023-11-16T16:18:26.478 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/vmw_balloon.ko 2023-11-16T16:18:26.478 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/vmw_vmci 2023-11-16T16:18:26.478 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-11-16T16:18:26.478 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net 2023-11-16T16:18:26.478 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/bonding 2023-11-16T16:18:26.478 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/bonding/bonding.ko 2023-11-16T16:18:26.478 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa 2023-11-16T16:18:26.478 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa/b53 2023-11-16T16:18:26.479 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa/b53/b53_common.ko 2023-11-16T16:18:26.479 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa/bcm-sf2.ko 2023-11-16T16:18:26.479 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa/mv88e6060.ko 2023-11-16T16:18:26.479 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa/mv88e6xxx 2023-11-16T16:18:26.479 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-11-16T16:18:26.479 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/dummy.ko 2023-11-16T16:18:26.479 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/eql.ko 2023-11-16T16:18:26.479 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet 2023-11-16T16:18:26.479 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/3com 2023-11-16T16:18:26.479 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-11-16T16:18:26.479 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-11-16T16:18:26.479 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-11-16T16:18:26.480 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-11-16T16:18:26.480 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/8390 2023-11-16T16:18:26.480 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/8390/8390.ko 2023-11-16T16:18:26.480 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-11-16T16:18:26.480 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-11-16T16:18:26.480 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-11-16T16:18:26.480 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/adaptec 2023-11-16T16:18:26.480 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-11-16T16:18:26.480 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/agere 2023-11-16T16:18:26.480 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/agere/et131x.ko 2023-11-16T16:18:26.480 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/alteon 2023-11-16T16:18:26.480 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-11-16T16:18:26.480 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/altera 2023-11-16T16:18:26.481 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-11-16T16:18:26.481 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/amd 2023-11-16T16:18:26.481 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-11-16T16:18:26.481 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-11-16T16:18:26.481 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-11-16T16:18:26.481 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros 2023-11-16T16:18:26.481 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/alx 2023-11-16T16:18:26.481 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-11-16T16:18:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atl1c 2023-11-16T16:18:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-11-16T16:18:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atl1e 2023-11-16T16:18:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-11-16T16:18:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atlx 2023-11-16T16:18:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-11-16T16:18:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-11-16T16:18:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom 2023-11-16T16:18:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-11-16T16:18:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-11-16T16:18:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-11-16T16:18:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-11-16T16:18:26.482 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-11-16T16:18:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/genet 2023-11-16T16:18:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-11-16T16:18:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-11-16T16:18:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/brocade 2023-11-16T16:18:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/brocade/bna 2023-11-16T16:18:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-11-16T16:18:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio 2023-11-16T16:18:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb 2023-11-16T16:18:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-11-16T16:18:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-11-16T16:18:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-11-16T16:18:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-11-16T16:18:26.483 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-11-16T16:18:26.484 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-11-16T16:18:26.484 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-11-16T16:18:26.484 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-11-16T16:18:26.484 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-11-16T16:18:26.484 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/cisco 2023-11-16T16:18:26.484 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/cisco/enic 2023-11-16T16:18:26.484 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-11-16T16:18:26.484 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec 2023-11-16T16:18:26.484 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip 2023-11-16T16:18:26.484 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-11-16T16:18:26.484 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-11-16T16:18:26.484 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-11-16T16:18:26.485 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-11-16T16:18:26.485 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-11-16T16:18:26.485 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-11-16T16:18:26.485 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dlink 2023-11-16T16:18:26.485 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-11-16T16:18:26.485 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-11-16T16:18:26.485 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/dnet.ko 2023-11-16T16:18:26.485 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/emulex 2023-11-16T16:18:26.485 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/emulex/benet 2023-11-16T16:18:26.485 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-11-16T16:18:26.485 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/ethoc.ko 2023-11-16T16:18:26.485 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/fealnx.ko 2023-11-16T16:18:26.486 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel 2023-11-16T16:18:26.486 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/e100.ko 2023-11-16T16:18:26.486 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/e1000 2023-11-16T16:18:26.486 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-11-16T16:18:26.486 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/e1000e 2023-11-16T16:18:26.486 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-11-16T16:18:26.486 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/fm10k 2023-11-16T16:18:26.486 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-11-16T16:18:26.486 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/i40e 2023-11-16T16:18:26.486 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-11-16T16:18:26.486 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/iavf 2023-11-16T16:18:26.486 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-11-16T16:18:26.486 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/igb 2023-11-16T16:18:26.486 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-11-16T16:18:26.487 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/igbvf 2023-11-16T16:18:26.487 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-11-16T16:18:26.487 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/ixgbe 2023-11-16T16:18:26.487 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-11-16T16:18:26.487 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/ixgbevf 2023-11-16T16:18:26.487 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-11-16T16:18:26.487 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/jme.ko 2023-11-16T16:18:26.487 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/marvell 2023-11-16T16:18:26.487 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-11-16T16:18:26.487 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/marvell/skge.ko 2023-11-16T16:18:26.487 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-11-16T16:18:26.487 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox 2023-11-16T16:18:26.487 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox/mlx4 2023-11-16T16:18:26.487 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-11-16T16:18:26.488 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-11-16T16:18:26.488 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox/mlx5 2023-11-16T16:18:26.488 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-11-16T16:18:26.488 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-11-16T16:18:26.488 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/micrel 2023-11-16T16:18:26.488 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-11-16T16:18:26.488 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/myricom 2023-11-16T16:18:26.488 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/myricom/myri10ge 2023-11-16T16:18:26.488 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-11-16T16:18:26.488 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/natsemi 2023-11-16T16:18:26.488 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-11-16T16:18:26.488 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-11-16T16:18:26.488 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/neterion 2023-11-16T16:18:26.489 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-11-16T16:18:26.489 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/nvidia 2023-11-16T16:18:26.489 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-11-16T16:18:26.489 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/packetengines 2023-11-16T16:18:26.489 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-11-16T16:18:26.489 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-11-16T16:18:26.489 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/qlogic 2023-11-16T16:18:26.489 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/qlogic/netxen 2023-11-16T16:18:26.489 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-11-16T16:18:26.489 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-11-16T16:18:26.489 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-11-16T16:18:26.489 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-11-16T16:18:26.489 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/rdc 2023-11-16T16:18:26.490 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-11-16T16:18:26.490 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/realtek 2023-11-16T16:18:26.490 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-11-16T16:18:26.490 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-11-16T16:18:26.490 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/realtek/atp.ko 2023-11-16T16:18:26.490 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-11-16T16:18:26.490 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/rocker 2023-11-16T16:18:26.490 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-11-16T16:18:26.490 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sfc 2023-11-16T16:18:26.490 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-11-16T16:18:26.490 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/silan 2023-11-16T16:18:26.490 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-11-16T16:18:26.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sis 2023-11-16T16:18:26.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sis/sis190.ko 2023-11-16T16:18:26.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sis/sis900.ko 2023-11-16T16:18:26.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/smsc 2023-11-16T16:18:26.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-11-16T16:18:26.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-11-16T16:18:26.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-11-16T16:18:26.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-11-16T16:18:26.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/stmicro 2023-11-16T16:18:26.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/stmicro/stmmac 2023-11-16T16:18:26.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-11-16T16:18:26.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-11-16T16:18:26.491 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sun 2023-11-16T16:18:26.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sun/cassini.ko 2023-11-16T16:18:26.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sun/niu.ko 2023-11-16T16:18:26.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sun/sungem.ko 2023-11-16T16:18:26.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-11-16T16:18:26.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/tehuti 2023-11-16T16:18:26.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-11-16T16:18:26.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/ti 2023-11-16T16:18:26.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/ti/tlan.ko 2023-11-16T16:18:26.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/via 2023-11-16T16:18:26.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-11-16T16:18:26.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-11-16T16:18:26.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/wiznet 2023-11-16T16:18:26.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-11-16T16:18:26.492 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-11-16T16:18:26.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/xircom 2023-11-16T16:18:26.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-11-16T16:18:26.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/geneve.ko 2023-11-16T16:18:26.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/hyperv 2023-11-16T16:18:26.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-11-16T16:18:26.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ieee802154 2023-11-16T16:18:26.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ieee802154/fakelb.ko 2023-11-16T16:18:26.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ifb.ko 2023-11-16T16:18:26.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ipvlan 2023-11-16T16:18:26.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ipvlan/ipvlan.ko 2023-11-16T16:18:26.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/macvlan.ko 2023-11-16T16:18:26.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/macvtap.ko 2023-11-16T16:18:26.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/mdio 2023-11-16T16:18:26.493 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/mdio.ko 2023-11-16T16:18:26.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-11-16T16:18:26.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-11-16T16:18:26.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/mdio/mdio-regmap.ko 2023-11-16T16:18:26.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/mii.ko 2023-11-16T16:18:26.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/net_failover.ko 2023-11-16T16:18:26.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/netconsole.ko 2023-11-16T16:18:26.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/nlmon.ko 2023-11-16T16:18:26.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/pcs 2023-11-16T16:18:26.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/pcs/pcs-lynx.ko 2023-11-16T16:18:26.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-11-16T16:18:26.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy 2023-11-16T16:18:26.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/amd.ko 2023-11-16T16:18:26.494 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-11-16T16:18:26.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-11-16T16:18:26.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/bcm7xxx.ko 2023-11-16T16:18:26.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/bcm87xx.ko 2023-11-16T16:18:26.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/broadcom.ko 2023-11-16T16:18:26.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/cicada.ko 2023-11-16T16:18:26.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/davicom.ko 2023-11-16T16:18:26.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/dp83640.ko 2023-11-16T16:18:26.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/et1011c.ko 2023-11-16T16:18:26.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/icplus.ko 2023-11-16T16:18:26.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/lxt.ko 2023-11-16T16:18:26.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/marvell.ko 2023-11-16T16:18:26.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/micrel.ko 2023-11-16T16:18:26.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/national.ko 2023-11-16T16:18:26.495 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/phylink.ko 2023-11-16T16:18:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/qsemi.ko 2023-11-16T16:18:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/realtek.ko 2023-11-16T16:18:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/smsc.ko 2023-11-16T16:18:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/ste10Xp.ko 2023-11-16T16:18:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/phy/vitesse.ko 2023-11-16T16:18:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp 2023-11-16T16:18:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/bsd_comp.ko 2023-11-16T16:18:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/ppp_async.ko 2023-11-16T16:18:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/ppp_deflate.ko 2023-11-16T16:18:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/ppp_generic.ko 2023-11-16T16:18:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/ppp_mppe.ko 2023-11-16T16:18:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/ppp_synctty.ko 2023-11-16T16:18:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/pppoe.ko 2023-11-16T16:18:26.496 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/pppox.ko 2023-11-16T16:18:26.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/ppp/pptp.ko 2023-11-16T16:18:26.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/slip 2023-11-16T16:18:26.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/slip/slhc.ko 2023-11-16T16:18:26.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/slip/slip.ko 2023-11-16T16:18:26.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/sungem_phy.ko 2023-11-16T16:18:26.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/tap.ko 2023-11-16T16:18:26.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team 2023-11-16T16:18:26.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team/team.ko 2023-11-16T16:18:26.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team/team_mode_activebackup.ko 2023-11-16T16:18:26.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team/team_mode_broadcast.ko 2023-11-16T16:18:26.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-11-16T16:18:26.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team/team_mode_random.ko 2023-11-16T16:18:26.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-11-16T16:18:26.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/tun.ko 2023-11-16T16:18:26.497 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/veth.ko 2023-11-16T16:18:26.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/virtio_net.ko 2023-11-16T16:18:26.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/vmxnet3 2023-11-16T16:18:26.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-11-16T16:18:26.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/vxlan 2023-11-16T16:18:26.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/vxlan/vxlan.ko 2023-11-16T16:18:26.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/xen-netback 2023-11-16T16:18:26.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/xen-netback/xen-netback.ko 2023-11-16T16:18:26.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/net/xen-netfront.ko 2023-11-16T16:18:26.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ntb 2023-11-16T16:18:26.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ntb/ntb.ko 2023-11-16T16:18:26.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/nvme 2023-11-16T16:18:26.498 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/nvme/host 2023-11-16T16:18:26.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/nvme/host/nvme-core.ko 2023-11-16T16:18:26.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/nvme/host/nvme.ko 2023-11-16T16:18:26.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/parport 2023-11-16T16:18:26.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/parport/parport.ko 2023-11-16T16:18:26.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/parport/parport_cs.ko 2023-11-16T16:18:26.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/parport/parport_pc.ko 2023-11-16T16:18:26.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/parport/parport_serial.ko 2023-11-16T16:18:26.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pci 2023-11-16T16:18:26.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pci/hotplug 2023-11-16T16:18:26.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-11-16T16:18:26.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pci/pcie 2023-11-16T16:18:26.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pci/pcie/aer_inject.ko 2023-11-16T16:18:26.499 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pci/xen-pcifront.ko 2023-11-16T16:18:26.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pcmcia 2023-11-16T16:18:26.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pcmcia/i82092.ko 2023-11-16T16:18:26.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pcmcia/pd6729.ko 2023-11-16T16:18:26.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pcmcia/yenta_socket.ko 2023-11-16T16:18:26.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pinctrl 2023-11-16T16:18:26.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pinctrl/intel 2023-11-16T16:18:26.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-11-16T16:18:26.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform 2023-11-16T16:18:26.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86 2023-11-16T16:18:26.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/acer-wmi.ko 2023-11-16T16:18:26.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/acerhdf.ko 2023-11-16T16:18:26.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/apple-gmux.ko 2023-11-16T16:18:26.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/asus-laptop.ko 2023-11-16T16:18:26.500 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-11-16T16:18:26.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/asus-wmi.ko 2023-11-16T16:18:26.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/classmate-laptop.ko 2023-11-16T16:18:26.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-11-16T16:18:26.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-11-16T16:18:26.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-11-16T16:18:26.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-11-16T16:18:26.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/hdaps.ko 2023-11-16T16:18:26.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/intel 2023-11-16T16:18:26.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-11-16T16:18:26.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/intel_ips.ko 2023-11-16T16:18:26.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/msi-wmi.ko 2023-11-16T16:18:26.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/mxm-wmi.ko 2023-11-16T16:18:26.501 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-11-16T16:18:26.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/samsung-laptop.ko 2023-11-16T16:18:26.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/samsung-q10.ko 2023-11-16T16:18:26.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/topstar-laptop.ko 2023-11-16T16:18:26.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-11-16T16:18:26.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/toshiba_haps.ko 2023-11-16T16:18:26.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/wmi-bmof.ko 2023-11-16T16:18:26.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/platform/x86/wmi.ko 2023-11-16T16:18:26.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/powercap 2023-11-16T16:18:26.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/powercap/intel_rapl_common.ko 2023-11-16T16:18:26.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/powercap/intel_rapl_msr.ko 2023-11-16T16:18:26.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pps 2023-11-16T16:18:26.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pps/clients 2023-11-16T16:18:26.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pps/clients/pps-gpio.ko 2023-11-16T16:18:26.502 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pps/clients/pps-ldisc.ko 2023-11-16T16:18:26.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pps/clients/pps_parport.ko 2023-11-16T16:18:26.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/pps/pps_core.ko 2023-11-16T16:18:26.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ptp 2023-11-16T16:18:26.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ptp/ptp.ko 2023-11-16T16:18:26.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ptp/ptp_kvm.ko 2023-11-16T16:18:26.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc 2023-11-16T16:18:26.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-bq32k.ko 2023-11-16T16:18:26.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1286.ko 2023-11-16T16:18:26.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1307.ko 2023-11-16T16:18:26.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1374.ko 2023-11-16T16:18:26.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1511.ko 2023-11-16T16:18:26.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1553.ko 2023-11-16T16:18:26.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1672.ko 2023-11-16T16:18:26.503 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds1742.ko 2023-11-16T16:18:26.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds2404.ko 2023-11-16T16:18:26.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-ds3232.ko 2023-11-16T16:18:26.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-em3027.ko 2023-11-16T16:18:26.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-fm3130.ko 2023-11-16T16:18:26.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-isl12022.ko 2023-11-16T16:18:26.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-isl1208.ko 2023-11-16T16:18:26.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-m41t80.ko 2023-11-16T16:18:26.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-m48t35.ko 2023-11-16T16:18:26.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-m48t59.ko 2023-11-16T16:18:26.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-max6900.ko 2023-11-16T16:18:26.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-msm6242.ko 2023-11-16T16:18:26.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-pcf2127.ko 2023-11-16T16:18:26.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-pcf85063.ko 2023-11-16T16:18:26.504 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-pcf8523.ko 2023-11-16T16:18:26.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-pcf8563.ko 2023-11-16T16:18:26.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-pcf8583.ko 2023-11-16T16:18:26.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-rp5c01.ko 2023-11-16T16:18:26.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-rs5c372.ko 2023-11-16T16:18:26.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-11-16T16:18:26.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-rx8025.ko 2023-11-16T16:18:26.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-rx8581.ko 2023-11-16T16:18:26.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-11-16T16:18:26.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/rtc/rtc-x1205.ko 2023-11-16T16:18:26.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi 2023-11-16T16:18:26.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/3w-9xxx.ko 2023-11-16T16:18:26.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/3w-sas.ko 2023-11-16T16:18:26.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/3w-xxxx.ko 2023-11-16T16:18:26.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/BusLogic.ko 2023-11-16T16:18:26.505 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/a100u2w.ko 2023-11-16T16:18:26.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/aacraid 2023-11-16T16:18:26.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/aacraid/aacraid.ko 2023-11-16T16:18:26.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/advansys.ko 2023-11-16T16:18:26.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/aic7xxx 2023-11-16T16:18:26.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-11-16T16:18:26.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-11-16T16:18:26.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/am53c974.ko 2023-11-16T16:18:26.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/arcmsr 2023-11-16T16:18:26.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-11-16T16:18:26.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/atp870u.ko 2023-11-16T16:18:26.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/be2iscsi 2023-11-16T16:18:26.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-11-16T16:18:26.506 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/bfa 2023-11-16T16:18:26.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/bfa/bfa.ko 2023-11-16T16:18:26.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/bnx2fc 2023-11-16T16:18:26.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-11-16T16:18:26.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/bnx2i 2023-11-16T16:18:26.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-11-16T16:18:26.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/ch.ko 2023-11-16T16:18:26.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/csiostor 2023-11-16T16:18:26.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/csiostor/csiostor.ko 2023-11-16T16:18:26.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/cxgbi 2023-11-16T16:18:26.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/cxgbi/cxgb3i 2023-11-16T16:18:26.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-11-16T16:18:26.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/cxgbi/cxgb4i 2023-11-16T16:18:26.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-11-16T16:18:26.507 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-11-16T16:18:26.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/dc395x.ko 2023-11-16T16:18:26.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/device_handler 2023-11-16T16:18:26.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-11-16T16:18:26.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-11-16T16:18:26.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-11-16T16:18:26.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-11-16T16:18:26.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/dmx3191d.ko 2023-11-16T16:18:26.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/esas2r 2023-11-16T16:18:26.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/esas2r/esas2r.ko 2023-11-16T16:18:26.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/esp_scsi.ko 2023-11-16T16:18:26.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/fcoe 2023-11-16T16:18:26.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/fcoe/fcoe.ko 2023-11-16T16:18:26.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-11-16T16:18:26.508 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/fnic 2023-11-16T16:18:26.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/fnic/fnic.ko 2023-11-16T16:18:26.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/hpsa.ko 2023-11-16T16:18:26.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/hptiop.ko 2023-11-16T16:18:26.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/hv_storvsc.ko 2023-11-16T16:18:26.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/initio.ko 2023-11-16T16:18:26.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/ipr.ko 2023-11-16T16:18:26.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/ips.ko 2023-11-16T16:18:26.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/isci 2023-11-16T16:18:26.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/isci/isci.ko 2023-11-16T16:18:26.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-11-16T16:18:26.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/iscsi_tcp.ko 2023-11-16T16:18:26.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/libfc 2023-11-16T16:18:26.509 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/libfc/libfc.ko 2023-11-16T16:18:26.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/libiscsi.ko 2023-11-16T16:18:26.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/libiscsi_tcp.ko 2023-11-16T16:18:26.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/libsas 2023-11-16T16:18:26.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/libsas/libsas.ko 2023-11-16T16:18:26.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/lpfc 2023-11-16T16:18:26.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/lpfc/lpfc.ko 2023-11-16T16:18:26.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/megaraid 2023-11-16T16:18:26.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/megaraid.ko 2023-11-16T16:18:26.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-11-16T16:18:26.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-11-16T16:18:26.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-11-16T16:18:26.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/mpt3sas 2023-11-16T16:18:26.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-11-16T16:18:26.510 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/mvsas 2023-11-16T16:18:26.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/mvsas/mvsas.ko 2023-11-16T16:18:26.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/mvumi.ko 2023-11-16T16:18:26.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/pm8001 2023-11-16T16:18:26.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-11-16T16:18:26.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/pmcraid.ko 2023-11-16T16:18:26.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/qla1280.ko 2023-11-16T16:18:26.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/qla2xxx 2023-11-16T16:18:26.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-11-16T16:18:26.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-11-16T16:18:26.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/qla4xxx 2023-11-16T16:18:26.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-11-16T16:18:26.511 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/raid_class.ko 2023-11-16T16:18:26.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/scsi_debug.ko 2023-11-16T16:18:26.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/scsi_transport_fc.ko 2023-11-16T16:18:26.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-11-16T16:18:26.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/scsi_transport_sas.ko 2023-11-16T16:18:26.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/scsi_transport_spi.ko 2023-11-16T16:18:26.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/scsi_transport_srp.ko 2023-11-16T16:18:26.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/ses.ko 2023-11-16T16:18:26.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/st.ko 2023-11-16T16:18:26.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/stex.ko 2023-11-16T16:18:26.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/sym53c8xx_2 2023-11-16T16:18:26.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-11-16T16:18:26.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/virtio_scsi.ko 2023-11-16T16:18:26.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/vmw_pvscsi.ko 2023-11-16T16:18:26.512 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/scsi/wd719x.ko 2023-11-16T16:18:26.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ssb 2023-11-16T16:18:26.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ssb/ssb.ko 2023-11-16T16:18:26.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target 2023-11-16T16:18:26.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/iscsi 2023-11-16T16:18:26.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-11-16T16:18:26.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/loopback 2023-11-16T16:18:26.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/loopback/tcm_loop.ko 2023-11-16T16:18:26.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/target_core_file.ko 2023-11-16T16:18:26.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/target_core_iblock.ko 2023-11-16T16:18:26.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/target_core_mod.ko 2023-11-16T16:18:26.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/target_core_pscsi.ko 2023-11-16T16:18:26.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/target_core_user.ko 2023-11-16T16:18:26.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/tcm_fc 2023-11-16T16:18:26.513 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-11-16T16:18:26.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal 2023-11-16T16:18:26.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel 2023-11-16T16:18:26.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal 2023-11-16T16:18:26.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-11-16T16:18:26.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-11-16T16:18:26.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-11-16T16:18:26.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-11-16T16:18:26.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-11-16T16:18:26.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-11-16T16:18:26.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-11-16T16:18:26.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-11-16T16:18:26.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-11-16T16:18:26.514 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-11-16T16:18:26.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-11-16T16:18:26.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-11-16T16:18:26.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-11-16T16:18:26.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-11-16T16:18:26.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-11-16T16:18:26.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty 2023-11-16T16:18:26.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/ipwireless 2023-11-16T16:18:26.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-11-16T16:18:26.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/n_gsm.ko 2023-11-16T16:18:26.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/n_hdlc.ko 2023-11-16T16:18:26.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/nozomi.ko 2023-11-16T16:18:26.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/serial 2023-11-16T16:18:26.515 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/serial/8250 2023-11-16T16:18:26.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-11-16T16:18:26.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/serial/arc_uart.ko 2023-11-16T16:18:26.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/serial/jsm 2023-11-16T16:18:26.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/tty/serial/jsm/jsm.ko 2023-11-16T16:18:26.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ufs 2023-11-16T16:18:26.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ufs/core 2023-11-16T16:18:26.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ufs/core/ufshcd-core.ko 2023-11-16T16:18:26.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ufs/host 2023-11-16T16:18:26.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-11-16T16:18:26.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/uio 2023-11-16T16:18:26.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/uio/uio.ko 2023-11-16T16:18:26.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/uio/uio_aec.ko 2023-11-16T16:18:26.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/uio/uio_cif.ko 2023-11-16T16:18:26.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/uio/uio_pci_generic.ko 2023-11-16T16:18:26.516 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/uio/uio_sercos3.ko 2023-11-16T16:18:26.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb 2023-11-16T16:18:26.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/class 2023-11-16T16:18:26.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/class/cdc-acm.ko 2023-11-16T16:18:26.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/class/cdc-wdm.ko 2023-11-16T16:18:26.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/class/usblp.ko 2023-11-16T16:18:26.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/class/usbtmc.ko 2023-11-16T16:18:26.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/host 2023-11-16T16:18:26.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/host/sl811-hcd.ko 2023-11-16T16:18:26.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/image 2023-11-16T16:18:26.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/image/mdc800.ko 2023-11-16T16:18:26.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/image/microtek.ko 2023-11-16T16:18:26.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc 2023-11-16T16:18:26.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/adutux.ko 2023-11-16T16:18:26.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/appledisplay.ko 2023-11-16T16:18:26.517 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/emi26.ko 2023-11-16T16:18:26.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/emi62.ko 2023-11-16T16:18:26.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/ezusb.ko 2023-11-16T16:18:26.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/idmouse.ko 2023-11-16T16:18:26.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/iowarrior.ko 2023-11-16T16:18:26.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/isight_firmware.ko 2023-11-16T16:18:26.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/ldusb.ko 2023-11-16T16:18:26.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/legousbtower.ko 2023-11-16T16:18:26.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/sisusbvga 2023-11-16T16:18:26.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-11-16T16:18:26.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/trancevibrator.ko 2023-11-16T16:18:26.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/usb3503.ko 2023-11-16T16:18:26.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/usblcd.ko 2023-11-16T16:18:26.518 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/usbsevseg.ko 2023-11-16T16:18:26.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/uss720.ko 2023-11-16T16:18:26.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/misc/yurex.ko 2023-11-16T16:18:26.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/phy 2023-11-16T16:18:26.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/phy/phy-generic.ko 2023-11-16T16:18:26.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial 2023-11-16T16:18:26.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/aircable.ko 2023-11-16T16:18:26.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ark3116.ko 2023-11-16T16:18:26.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/belkin_sa.ko 2023-11-16T16:18:26.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ch341.ko 2023-11-16T16:18:26.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/cp210x.ko 2023-11-16T16:18:26.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/cyberjack.ko 2023-11-16T16:18:26.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/cypress_m8.ko 2023-11-16T16:18:26.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/digi_acceleport.ko 2023-11-16T16:18:26.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/empeg.ko 2023-11-16T16:18:26.519 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ftdi_sio.ko 2023-11-16T16:18:26.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/garmin_gps.ko 2023-11-16T16:18:26.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/io_edgeport.ko 2023-11-16T16:18:26.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/io_ti.ko 2023-11-16T16:18:26.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ipaq.ko 2023-11-16T16:18:26.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ipw.ko 2023-11-16T16:18:26.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ir-usb.ko 2023-11-16T16:18:26.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-11-16T16:18:26.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/keyspan.ko 2023-11-16T16:18:26.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/keyspan_pda.ko 2023-11-16T16:18:26.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/kl5kusb105.ko 2023-11-16T16:18:26.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/kobil_sct.ko 2023-11-16T16:18:26.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/mct_u232.ko 2023-11-16T16:18:26.520 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/mos7720.ko 2023-11-16T16:18:26.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/mos7840.ko 2023-11-16T16:18:26.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/navman.ko 2023-11-16T16:18:26.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/omninet.ko 2023-11-16T16:18:26.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/opticon.ko 2023-11-16T16:18:26.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/option.ko 2023-11-16T16:18:26.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/oti6858.ko 2023-11-16T16:18:26.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/pl2303.ko 2023-11-16T16:18:26.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/qcaux.ko 2023-11-16T16:18:26.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/qcserial.ko 2023-11-16T16:18:26.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/quatech2.ko 2023-11-16T16:18:26.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/safe_serial.ko 2023-11-16T16:18:26.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/sierra.ko 2023-11-16T16:18:26.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/spcp8x5.ko 2023-11-16T16:18:26.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ssu100.ko 2023-11-16T16:18:26.521 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/symbolserial.ko 2023-11-16T16:18:26.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-11-16T16:18:26.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-11-16T16:18:26.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/usb_debug.ko 2023-11-16T16:18:26.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/usb_wwan.ko 2023-11-16T16:18:26.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/visor.ko 2023-11-16T16:18:26.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/whiteheat.ko 2023-11-16T16:18:26.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/serial/xsens_mt.ko 2023-11-16T16:18:26.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage 2023-11-16T16:18:26.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/uas.ko 2023-11-16T16:18:26.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-alauda.ko 2023-11-16T16:18:26.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-cypress.ko 2023-11-16T16:18:26.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-datafab.ko 2023-11-16T16:18:26.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-11-16T16:18:26.522 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-freecom.ko 2023-11-16T16:18:26.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-isd200.ko 2023-11-16T16:18:26.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-11-16T16:18:26.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-karma.ko 2023-11-16T16:18:26.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-onetouch.ko 2023-11-16T16:18:26.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-realtek.ko 2023-11-16T16:18:26.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-sddr09.ko 2023-11-16T16:18:26.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-sddr55.ko 2023-11-16T16:18:26.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/ums-usbat.ko 2023-11-16T16:18:26.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/storage/usb-storage.ko 2023-11-16T16:18:26.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/usbip 2023-11-16T16:18:26.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/usbip/usbip-core.ko 2023-11-16T16:18:26.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/usbip/usbip-host.ko 2023-11-16T16:18:26.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-11-16T16:18:26.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vfio 2023-11-16T16:18:26.523 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vfio/pci 2023-11-16T16:18:26.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-11-16T16:18:26.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vfio/pci/vfio-pci.ko 2023-11-16T16:18:26.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vfio/vfio.ko 2023-11-16T16:18:26.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-11-16T16:18:26.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vhost 2023-11-16T16:18:26.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vhost/vhost.ko 2023-11-16T16:18:26.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vhost/vhost_iotlb.ko 2023-11-16T16:18:26.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vhost/vhost_net.ko 2023-11-16T16:18:26.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/vhost/vhost_scsi.ko 2023-11-16T16:18:26.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/video 2023-11-16T16:18:26.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/video/backlight 2023-11-16T16:18:26.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/video/backlight/apple_bl.ko 2023-11-16T16:18:26.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/video/backlight/lcd.ko 2023-11-16T16:18:26.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/video/backlight/platform_lcd.ko 2023-11-16T16:18:26.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio 2023-11-16T16:18:26.524 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio.ko 2023-11-16T16:18:26.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio_balloon.ko 2023-11-16T16:18:26.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio_mmio.ko 2023-11-16T16:18:26.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio_pci.ko 2023-11-16T16:18:26.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-11-16T16:18:26.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-11-16T16:18:26.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/virtio/virtio_ring.ko 2023-11-16T16:18:26.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1 2023-11-16T16:18:26.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/masters 2023-11-16T16:18:26.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/masters/ds2482.ko 2023-11-16T16:18:26.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/masters/ds2490.ko 2023-11-16T16:18:26.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves 2023-11-16T16:18:26.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-11-16T16:18:26.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-11-16T16:18:26.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-11-16T16:18:26.525 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-11-16T16:18:26.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-11-16T16:18:26.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-11-16T16:18:26.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-11-16T16:18:26.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-11-16T16:18:26.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-11-16T16:18:26.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_smem.ko 2023-11-16T16:18:26.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/slaves/w1_therm.ko 2023-11-16T16:18:26.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/w1/wire.ko 2023-11-16T16:18:26.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog 2023-11-16T16:18:26.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/alim1535_wdt.ko 2023-11-16T16:18:26.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/alim7101_wdt.ko 2023-11-16T16:18:26.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/f71808e_wdt.ko 2023-11-16T16:18:26.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/hpwdt.ko 2023-11-16T16:18:26.526 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/i6300esb.ko 2023-11-16T16:18:26.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-11-16T16:18:26.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/iTCO_wdt.ko 2023-11-16T16:18:26.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/ib700wdt.ko 2023-11-16T16:18:26.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/ibmasr.ko 2023-11-16T16:18:26.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-11-16T16:18:26.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/it8712f_wdt.ko 2023-11-16T16:18:26.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/it87_wdt.ko 2023-11-16T16:18:26.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/machzwd.ko 2023-11-16T16:18:26.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/nv_tco.ko 2023-11-16T16:18:26.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/pcwd_pci.ko 2023-11-16T16:18:26.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/pcwd_usb.ko 2023-11-16T16:18:26.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-11-16T16:18:26.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/sch311x_wdt.ko 2023-11-16T16:18:26.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/softdog.ko 2023-11-16T16:18:26.527 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/sp5100_tco.ko 2023-11-16T16:18:26.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/via_wdt.ko 2023-11-16T16:18:26.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-11-16T16:18:26.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/w83877f_wdt.ko 2023-11-16T16:18:26.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/w83977f_wdt.ko 2023-11-16T16:18:26.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/wdt_pci.ko 2023-11-16T16:18:26.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/watchdog/xen_wdt.ko 2023-11-16T16:18:26.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen 2023-11-16T16:18:26.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-acpi-processor.ko 2023-11-16T16:18:26.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-evtchn.ko 2023-11-16T16:18:26.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-gntalloc.ko 2023-11-16T16:18:26.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-gntdev.ko 2023-11-16T16:18:26.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-pciback 2023-11-16T16:18:26.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-11-16T16:18:26.528 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xen-privcmd.ko 2023-11-16T16:18:26.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xenfs 2023-11-16T16:18:26.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/drivers/xen/xenfs/xenfs.ko 2023-11-16T16:18:26.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs 2023-11-16T16:18:26.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/9p 2023-11-16T16:18:26.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/9p/9p.ko 2023-11-16T16:18:26.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/affs 2023-11-16T16:18:26.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/affs/affs.ko 2023-11-16T16:18:26.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/befs 2023-11-16T16:18:26.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/befs/befs.ko 2023-11-16T16:18:26.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/binfmt_misc.ko 2023-11-16T16:18:26.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/btrfs 2023-11-16T16:18:26.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/btrfs/btrfs.ko 2023-11-16T16:18:26.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/cachefiles 2023-11-16T16:18:26.529 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/cachefiles/cachefiles.ko 2023-11-16T16:18:26.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ceph 2023-11-16T16:18:26.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ceph/ceph.ko 2023-11-16T16:18:26.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/coda 2023-11-16T16:18:26.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/coda/coda.ko 2023-11-16T16:18:26.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/cramfs 2023-11-16T16:18:26.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/cramfs/cramfs.ko 2023-11-16T16:18:26.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/dlm 2023-11-16T16:18:26.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/dlm/dlm.ko 2023-11-16T16:18:26.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ecryptfs 2023-11-16T16:18:26.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ecryptfs/ecryptfs.ko 2023-11-16T16:18:26.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/f2fs 2023-11-16T16:18:26.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/f2fs/f2fs.ko 2023-11-16T16:18:26.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fat 2023-11-16T16:18:26.530 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fat/fat.ko 2023-11-16T16:18:26.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fat/msdos.ko 2023-11-16T16:18:26.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fat/vfat.ko 2023-11-16T16:18:26.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fscache 2023-11-16T16:18:26.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fscache/fscache.ko 2023-11-16T16:18:26.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fuse 2023-11-16T16:18:26.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fuse/cuse.ko 2023-11-16T16:18:26.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/fuse/fuse.ko 2023-11-16T16:18:26.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/gfs2 2023-11-16T16:18:26.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/gfs2/gfs2.ko 2023-11-16T16:18:26.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/hfs 2023-11-16T16:18:26.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/hfs/hfs.ko 2023-11-16T16:18:26.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/hfsplus 2023-11-16T16:18:26.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/hfsplus/hfsplus.ko 2023-11-16T16:18:26.531 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/isofs 2023-11-16T16:18:26.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/isofs/isofs.ko 2023-11-16T16:18:26.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/jfs 2023-11-16T16:18:26.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/jfs/jfs.ko 2023-11-16T16:18:26.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/lockd 2023-11-16T16:18:26.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/lockd/lockd.ko 2023-11-16T16:18:26.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/minix 2023-11-16T16:18:26.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/minix/minix.ko 2023-11-16T16:18:26.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/netfs 2023-11-16T16:18:26.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/netfs/netfs.ko 2023-11-16T16:18:26.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs 2023-11-16T16:18:26.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/blocklayout 2023-11-16T16:18:26.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-11-16T16:18:26.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/filelayout 2023-11-16T16:18:26.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-11-16T16:18:26.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/flexfilelayout 2023-11-16T16:18:26.532 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-11-16T16:18:26.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/nfs.ko 2023-11-16T16:18:26.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/nfsv3.ko 2023-11-16T16:18:26.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs/nfsv4.ko 2023-11-16T16:18:26.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs_common 2023-11-16T16:18:26.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs_common/grace.ko 2023-11-16T16:18:26.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfs_common/nfs_acl.ko 2023-11-16T16:18:26.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfsd 2023-11-16T16:18:26.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nfsd/nfsd.ko 2023-11-16T16:18:26.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nilfs2 2023-11-16T16:18:26.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nilfs2/nilfs2.ko 2023-11-16T16:18:26.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls 2023-11-16T16:18:26.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-celtic.ko 2023-11-16T16:18:26.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-centeuro.ko 2023-11-16T16:18:26.533 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-croatian.ko 2023-11-16T16:18:26.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-cyrillic.ko 2023-11-16T16:18:26.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-gaelic.ko 2023-11-16T16:18:26.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-greek.ko 2023-11-16T16:18:26.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-iceland.ko 2023-11-16T16:18:26.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-inuit.ko 2023-11-16T16:18:26.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-roman.ko 2023-11-16T16:18:26.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-romanian.ko 2023-11-16T16:18:26.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/mac-turkish.ko 2023-11-16T16:18:26.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp1250.ko 2023-11-16T16:18:26.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp1251.ko 2023-11-16T16:18:26.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp1255.ko 2023-11-16T16:18:26.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp737.ko 2023-11-16T16:18:26.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp775.ko 2023-11-16T16:18:26.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp850.ko 2023-11-16T16:18:26.534 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp852.ko 2023-11-16T16:18:26.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp855.ko 2023-11-16T16:18:26.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp857.ko 2023-11-16T16:18:26.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp860.ko 2023-11-16T16:18:26.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp861.ko 2023-11-16T16:18:26.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp862.ko 2023-11-16T16:18:26.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp863.ko 2023-11-16T16:18:26.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp864.ko 2023-11-16T16:18:26.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp865.ko 2023-11-16T16:18:26.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp866.ko 2023-11-16T16:18:26.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp869.ko 2023-11-16T16:18:26.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp874.ko 2023-11-16T16:18:26.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp932.ko 2023-11-16T16:18:26.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp936.ko 2023-11-16T16:18:26.535 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp949.ko 2023-11-16T16:18:26.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_cp950.ko 2023-11-16T16:18:26.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_euc-jp.ko 2023-11-16T16:18:26.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-1.ko 2023-11-16T16:18:26.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-13.ko 2023-11-16T16:18:26.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-14.ko 2023-11-16T16:18:26.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-15.ko 2023-11-16T16:18:26.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-2.ko 2023-11-16T16:18:26.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-3.ko 2023-11-16T16:18:26.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-4.ko 2023-11-16T16:18:26.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-5.ko 2023-11-16T16:18:26.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-6.ko 2023-11-16T16:18:26.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-7.ko 2023-11-16T16:18:26.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_iso8859-9.ko 2023-11-16T16:18:26.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_koi8-r.ko 2023-11-16T16:18:26.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_koi8-ru.ko 2023-11-16T16:18:26.536 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_koi8-u.ko 2023-11-16T16:18:26.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_ucs2_utils.ko 2023-11-16T16:18:26.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/nls/nls_utf8.ko 2023-11-16T16:18:26.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2 2023-11-16T16:18:26.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/cluster 2023-11-16T16:18:26.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-11-16T16:18:26.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/dlm 2023-11-16T16:18:26.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-11-16T16:18:26.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/dlmfs 2023-11-16T16:18:26.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-11-16T16:18:26.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/ocfs2.ko 2023-11-16T16:18:26.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-11-16T16:18:26.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-11-16T16:18:26.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-11-16T16:18:26.537 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/overlayfs 2023-11-16T16:18:26.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/overlayfs/overlay.ko 2023-11-16T16:18:26.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/pstore 2023-11-16T16:18:26.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/pstore/ramoops.ko 2023-11-16T16:18:26.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/reiserfs 2023-11-16T16:18:26.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/reiserfs/reiserfs.ko 2023-11-16T16:18:26.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/romfs 2023-11-16T16:18:26.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/romfs/romfs.ko 2023-11-16T16:18:26.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/smb 2023-11-16T16:18:26.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/smb/client 2023-11-16T16:18:26.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/smb/client/cifs.ko 2023-11-16T16:18:26.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/smb/common 2023-11-16T16:18:26.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/smb/common/cifs_arc4.ko 2023-11-16T16:18:26.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/smb/common/cifs_md4.ko 2023-11-16T16:18:26.538 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/squashfs 2023-11-16T16:18:26.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/squashfs/squashfs.ko 2023-11-16T16:18:26.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/sysv 2023-11-16T16:18:26.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/sysv/sysv.ko 2023-11-16T16:18:26.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/udf 2023-11-16T16:18:26.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/udf/udf.ko 2023-11-16T16:18:26.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ufs 2023-11-16T16:18:26.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/ufs/ufs.ko 2023-11-16T16:18:26.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/xfs 2023-11-16T16:18:26.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/fs/xfs/xfs.ko 2023-11-16T16:18:26.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/kernel 2023-11-16T16:18:26.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/kernel/rcu 2023-11-16T16:18:26.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/kernel/rcu/rcutorture.ko 2023-11-16T16:18:26.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/kernel/torture.ko 2023-11-16T16:18:26.539 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/kernel/trace 2023-11-16T16:18:26.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-11-16T16:18:26.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib 2023-11-16T16:18:26.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/asn1_encoder.ko 2023-11-16T16:18:26.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/crc-itu-t.ko 2023-11-16T16:18:26.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/crc8.ko 2023-11-16T16:18:26.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/crypto 2023-11-16T16:18:26.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/crypto/libarc4.ko 2023-11-16T16:18:26.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/crypto/libdes.ko 2023-11-16T16:18:26.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/libcrc32c.ko 2023-11-16T16:18:26.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/lru_cache.ko 2023-11-16T16:18:26.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/lz4 2023-11-16T16:18:26.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/lz4/lz4_compress.ko 2023-11-16T16:18:26.540 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/lz4/lz4hc_compress.ko 2023-11-16T16:18:26.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/math 2023-11-16T16:18:26.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/math/cordic.ko 2023-11-16T16:18:26.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/raid6 2023-11-16T16:18:26.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/raid6/raid6_pq.ko 2023-11-16T16:18:26.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/reed_solomon 2023-11-16T16:18:26.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/reed_solomon/reed_solomon.ko 2023-11-16T16:18:26.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/ts_bm.ko 2023-11-16T16:18:26.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/ts_fsm.ko 2023-11-16T16:18:26.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/ts_kmp.ko 2023-11-16T16:18:26.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/zstd 2023-11-16T16:18:26.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/lib/zstd/zstd_compress.ko 2023-11-16T16:18:26.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/mm 2023-11-16T16:18:26.541 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/mm/hwpoison-inject.ko 2023-11-16T16:18:26.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net 2023-11-16T16:18:26.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan 2023-11-16T16:18:26.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/6lowpan.ko 2023-11-16T16:18:26.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_dest.ko 2023-11-16T16:18:26.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_fragment.ko 2023-11-16T16:18:26.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_hop.ko 2023-11-16T16:18:26.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_ipv6.ko 2023-11-16T16:18:26.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_mobility.ko 2023-11-16T16:18:26.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_routing.ko 2023-11-16T16:18:26.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/6lowpan/nhc_udp.ko 2023-11-16T16:18:26.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/802 2023-11-16T16:18:26.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/802/garp.ko 2023-11-16T16:18:26.542 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/802/mrp.ko 2023-11-16T16:18:26.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/802/p8022.ko 2023-11-16T16:18:26.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/802/psnap.ko 2023-11-16T16:18:26.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/802/stp.ko 2023-11-16T16:18:26.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/8021q 2023-11-16T16:18:26.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/8021q/8021q.ko 2023-11-16T16:18:26.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/9p 2023-11-16T16:18:26.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/9p/9pnet.ko 2023-11-16T16:18:26.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/9p/9pnet_fd.ko 2023-11-16T16:18:26.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/9p/9pnet_rdma.ko 2023-11-16T16:18:26.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/9p/9pnet_virtio.ko 2023-11-16T16:18:26.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/batman-adv 2023-11-16T16:18:26.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/batman-adv/batman-adv.ko 2023-11-16T16:18:26.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge 2023-11-16T16:18:26.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/br_netfilter.ko 2023-11-16T16:18:26.543 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/bridge.ko 2023-11-16T16:18:26.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter 2023-11-16T16:18:26.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-11-16T16:18:26.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_among.ko 2023-11-16T16:18:26.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_arp.ko 2023-11-16T16:18:26.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-11-16T16:18:26.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-11-16T16:18:26.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_ip.ko 2023-11-16T16:18:26.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-11-16T16:18:26.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_limit.ko 2023-11-16T16:18:26.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_log.ko 2023-11-16T16:18:26.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_mark.ko 2023-11-16T16:18:26.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-11-16T16:18:26.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-11-16T16:18:26.544 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-11-16T16:18:26.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-11-16T16:18:26.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_snat.ko 2023-11-16T16:18:26.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_stp.ko 2023-11-16T16:18:26.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-11-16T16:18:26.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-11-16T16:18:26.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-11-16T16:18:26.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-11-16T16:18:26.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/ebtables.ko 2023-11-16T16:18:26.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-11-16T16:18:26.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-11-16T16:18:26.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-11-16T16:18:26.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ceph 2023-11-16T16:18:26.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ceph/libceph.ko 2023-11-16T16:18:26.545 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/core 2023-11-16T16:18:26.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/core/failover.ko 2023-11-16T16:18:26.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/core/pktgen.ko 2023-11-16T16:18:26.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dccp 2023-11-16T16:18:26.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dccp/dccp.ko 2023-11-16T16:18:26.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dccp/dccp_diag.ko 2023-11-16T16:18:26.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dccp/dccp_ipv4.ko 2023-11-16T16:18:26.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dccp/dccp_ipv6.ko 2023-11-16T16:18:26.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dns_resolver 2023-11-16T16:18:26.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dns_resolver/dns_resolver.ko 2023-11-16T16:18:26.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dsa 2023-11-16T16:18:26.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dsa/dsa_core.ko 2023-11-16T16:18:26.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dsa/tag_brcm.ko 2023-11-16T16:18:26.546 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dsa/tag_dsa.ko 2023-11-16T16:18:26.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dsa/tag_none.ko 2023-11-16T16:18:26.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/dsa/tag_trailer.ko 2023-11-16T16:18:26.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ieee802154 2023-11-16T16:18:26.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ieee802154/6lowpan 2023-11-16T16:18:26.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-11-16T16:18:26.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ieee802154/ieee802154.ko 2023-11-16T16:18:26.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ieee802154/ieee802154_socket.ko 2023-11-16T16:18:26.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4 2023-11-16T16:18:26.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/ah4.ko 2023-11-16T16:18:26.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/esp4.ko 2023-11-16T16:18:26.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/fou.ko 2023-11-16T16:18:26.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/gre.ko 2023-11-16T16:18:26.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/inet_diag.ko 2023-11-16T16:18:26.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/ip_gre.ko 2023-11-16T16:18:26.547 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/ip_tunnel.ko 2023-11-16T16:18:26.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/ip_vti.ko 2023-11-16T16:18:26.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/ipcomp.ko 2023-11-16T16:18:26.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/ipip.ko 2023-11-16T16:18:26.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter 2023-11-16T16:18:26.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/arp_tables.ko 2023-11-16T16:18:26.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-11-16T16:18:26.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-11-16T16:18:26.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-11-16T16:18:26.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-11-16T16:18:26.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-11-16T16:18:26.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-11-16T16:18:26.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-11-16T16:18:26.548 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-11-16T16:18:26.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-11-16T16:18:26.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/iptable_security.ko 2023-11-16T16:18:26.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-11-16T16:18:26.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-11-16T16:18:26.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-11-16T16:18:26.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-11-16T16:18:26.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-11-16T16:18:26.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-11-16T16:18:26.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-11-16T16:18:26.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-11-16T16:18:26.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-11-16T16:18:26.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-11-16T16:18:26.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_bic.ko 2023-11-16T16:18:26.549 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_dctcp.ko 2023-11-16T16:18:26.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_diag.ko 2023-11-16T16:18:26.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_highspeed.ko 2023-11-16T16:18:26.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_htcp.ko 2023-11-16T16:18:26.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_hybla.ko 2023-11-16T16:18:26.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_illinois.ko 2023-11-16T16:18:26.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_lp.ko 2023-11-16T16:18:26.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_scalable.ko 2023-11-16T16:18:26.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_vegas.ko 2023-11-16T16:18:26.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_veno.ko 2023-11-16T16:18:26.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_westwood.ko 2023-11-16T16:18:26.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tcp_yeah.ko 2023-11-16T16:18:26.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/tunnel4.ko 2023-11-16T16:18:26.550 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/udp_diag.ko 2023-11-16T16:18:26.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/udp_tunnel.ko 2023-11-16T16:18:26.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv4/xfrm4_tunnel.ko 2023-11-16T16:18:26.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6 2023-11-16T16:18:26.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/ah6.ko 2023-11-16T16:18:26.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/esp6.ko 2023-11-16T16:18:26.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/fou6.ko 2023-11-16T16:18:26.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/ip6_tunnel.ko 2023-11-16T16:18:26.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-11-16T16:18:26.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/ip6_vti.ko 2023-11-16T16:18:26.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/ipcomp6.ko 2023-11-16T16:18:26.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter 2023-11-16T16:18:26.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-11-16T16:18:26.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-11-16T16:18:26.551 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-11-16T16:18:26.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-11-16T16:18:26.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-11-16T16:18:26.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-11-16T16:18:26.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-11-16T16:18:26.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-11-16T16:18:26.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-11-16T16:18:26.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-11-16T16:18:26.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-11-16T16:18:26.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-11-16T16:18:26.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-11-16T16:18:26.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-11-16T16:18:26.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-11-16T16:18:26.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-11-16T16:18:26.552 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-11-16T16:18:26.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-11-16T16:18:26.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-11-16T16:18:26.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-11-16T16:18:26.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-11-16T16:18:26.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-11-16T16:18:26.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-11-16T16:18:26.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-11-16T16:18:26.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/sit.ko 2023-11-16T16:18:26.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/tunnel6.ko 2023-11-16T16:18:26.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/ipv6/xfrm6_tunnel.ko 2023-11-16T16:18:26.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/key 2023-11-16T16:18:26.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/key/af_key.ko 2023-11-16T16:18:26.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp 2023-11-16T16:18:26.553 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_core.ko 2023-11-16T16:18:26.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_debugfs.ko 2023-11-16T16:18:26.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_eth.ko 2023-11-16T16:18:26.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_ip.ko 2023-11-16T16:18:26.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_ip6.ko 2023-11-16T16:18:26.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_netlink.ko 2023-11-16T16:18:26.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/l2tp/l2tp_ppp.ko 2023-11-16T16:18:26.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/llc 2023-11-16T16:18:26.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/llc/llc.ko 2023-11-16T16:18:26.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/mac802154 2023-11-16T16:18:26.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/mac802154/mac802154.ko 2023-11-16T16:18:26.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/mpls 2023-11-16T16:18:26.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/mpls/mpls_gso.ko 2023-11-16T16:18:26.554 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter 2023-11-16T16:18:26.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset 2023-11-16T16:18:26.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set.ko 2023-11-16T16:18:26.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-11-16T16:18:26.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-11-16T16:18:26.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-11-16T16:18:26.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-11-16T16:18:26.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-11-16T16:18:26.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-11-16T16:18:26.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-11-16T16:18:26.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-11-16T16:18:26.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-11-16T16:18:26.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-11-16T16:18:26.555 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-11-16T16:18:26.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-11-16T16:18:26.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-11-16T16:18:26.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-11-16T16:18:26.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-11-16T16:18:26.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs 2023-11-16T16:18:26.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs.ko 2023-11-16T16:18:26.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-11-16T16:18:26.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-11-16T16:18:26.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-11-16T16:18:26.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-11-16T16:18:26.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-11-16T16:18:26.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-11-16T16:18:26.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-11-16T16:18:26.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-11-16T16:18:26.556 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-11-16T16:18:26.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-11-16T16:18:26.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-11-16T16:18:26.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-11-16T16:18:26.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-11-16T16:18:26.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conncount.ko 2023-11-16T16:18:26.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack.ko 2023-11-16T16:18:26.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-11-16T16:18:26.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-11-16T16:18:26.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-11-16T16:18:26.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_h323.ko 2023-11-16T16:18:26.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_irc.ko 2023-11-16T16:18:26.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-11-16T16:18:26.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-11-16T16:18:26.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-11-16T16:18:26.557 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_sane.ko 2023-11-16T16:18:26.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_sip.ko 2023-11-16T16:18:26.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-11-16T16:18:26.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-11-16T16:18:26.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_dup_netdev.ko 2023-11-16T16:18:26.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_flow_table.ko 2023-11-16T16:18:26.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_flow_table_inet.ko 2023-11-16T16:18:26.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_log_syslog.ko 2023-11-16T16:18:26.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_nat.ko 2023-11-16T16:18:26.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_nat_amanda.ko 2023-11-16T16:18:26.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_nat_ftp.ko 2023-11-16T16:18:26.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_nat_irc.ko 2023-11-16T16:18:26.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_nat_sip.ko 2023-11-16T16:18:26.558 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_nat_tftp.ko 2023-11-16T16:18:26.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_synproxy_core.ko 2023-11-16T16:18:26.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nf_tables.ko 2023-11-16T16:18:26.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nfnetlink.ko 2023-11-16T16:18:26.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nfnetlink_acct.ko 2023-11-16T16:18:26.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nfnetlink_log.ko 2023-11-16T16:18:26.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nfnetlink_osf.ko 2023-11-16T16:18:26.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nfnetlink_queue.ko 2023-11-16T16:18:26.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_chain_nat.ko 2023-11-16T16:18:26.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_compat.ko 2023-11-16T16:18:26.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_ct.ko 2023-11-16T16:18:26.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_dup_netdev.ko 2023-11-16T16:18:26.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_fib.ko 2023-11-16T16:18:26.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_fwd_netdev.ko 2023-11-16T16:18:26.559 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_hash.ko 2023-11-16T16:18:26.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_limit.ko 2023-11-16T16:18:26.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_log.ko 2023-11-16T16:18:26.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_masq.ko 2023-11-16T16:18:26.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_nat.ko 2023-11-16T16:18:26.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_numgen.ko 2023-11-16T16:18:26.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_queue.ko 2023-11-16T16:18:26.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_quota.ko 2023-11-16T16:18:26.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_redir.ko 2023-11-16T16:18:26.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_reject.ko 2023-11-16T16:18:26.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_reject_inet.ko 2023-11-16T16:18:26.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_socket.ko 2023-11-16T16:18:26.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/nft_tunnel.ko 2023-11-16T16:18:26.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_AUDIT.ko 2023-11-16T16:18:26.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_CHECKSUM.ko 2023-11-16T16:18:26.560 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_CLASSIFY.ko 2023-11-16T16:18:26.561 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-11-16T16:18:26.561 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_CT.ko 2023-11-16T16:18:26.672 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_DSCP.ko 2023-11-16T16:18:26.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_HL.ko 2023-11-16T16:18:26.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_HMARK.ko 2023-11-16T16:18:26.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_IDLETIMER.ko 2023-11-16T16:18:26.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_LED.ko 2023-11-16T16:18:26.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_LOG.ko 2023-11-16T16:18:26.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_MASQUERADE.ko 2023-11-16T16:18:26.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_NETMAP.ko 2023-11-16T16:18:26.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_NFLOG.ko 2023-11-16T16:18:26.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_NFQUEUE.ko 2023-11-16T16:18:26.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_RATEEST.ko 2023-11-16T16:18:26.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_REDIRECT.ko 2023-11-16T16:18:26.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_SECMARK.ko 2023-11-16T16:18:26.673 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_TCPMSS.ko 2023-11-16T16:18:26.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-11-16T16:18:26.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_TEE.ko 2023-11-16T16:18:26.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_TPROXY.ko 2023-11-16T16:18:26.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_TRACE.ko 2023-11-16T16:18:26.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_addrtype.ko 2023-11-16T16:18:26.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_bpf.ko 2023-11-16T16:18:26.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_cgroup.ko 2023-11-16T16:18:26.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_cluster.ko 2023-11-16T16:18:26.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_comment.ko 2023-11-16T16:18:26.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_connbytes.ko 2023-11-16T16:18:26.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_connlabel.ko 2023-11-16T16:18:26.674 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_connlimit.ko 2023-11-16T16:18:26.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_connmark.ko 2023-11-16T16:18:26.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_conntrack.ko 2023-11-16T16:18:26.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_cpu.ko 2023-11-16T16:18:26.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_dccp.ko 2023-11-16T16:18:26.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_devgroup.ko 2023-11-16T16:18:26.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_dscp.ko 2023-11-16T16:18:26.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_ecn.ko 2023-11-16T16:18:26.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_esp.ko 2023-11-16T16:18:26.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_hashlimit.ko 2023-11-16T16:18:26.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_helper.ko 2023-11-16T16:18:26.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_hl.ko 2023-11-16T16:18:26.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_ipcomp.ko 2023-11-16T16:18:26.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_iprange.ko 2023-11-16T16:18:26.675 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_ipvs.ko 2023-11-16T16:18:26.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_l2tp.ko 2023-11-16T16:18:26.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_length.ko 2023-11-16T16:18:26.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_limit.ko 2023-11-16T16:18:26.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_mac.ko 2023-11-16T16:18:26.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_mark.ko 2023-11-16T16:18:26.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_multiport.ko 2023-11-16T16:18:26.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_nat.ko 2023-11-16T16:18:26.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_nfacct.ko 2023-11-16T16:18:26.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_osf.ko 2023-11-16T16:18:26.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_owner.ko 2023-11-16T16:18:26.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_physdev.ko 2023-11-16T16:18:26.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_pkttype.ko 2023-11-16T16:18:26.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_policy.ko 2023-11-16T16:18:26.676 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_quota.ko 2023-11-16T16:18:26.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_rateest.ko 2023-11-16T16:18:26.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_realm.ko 2023-11-16T16:18:26.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_recent.ko 2023-11-16T16:18:26.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_sctp.ko 2023-11-16T16:18:26.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_set.ko 2023-11-16T16:18:26.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_socket.ko 2023-11-16T16:18:26.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_state.ko 2023-11-16T16:18:26.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_statistic.ko 2023-11-16T16:18:26.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_string.ko 2023-11-16T16:18:26.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_tcpmss.ko 2023-11-16T16:18:26.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_time.ko 2023-11-16T16:18:26.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netfilter/xt_u32.ko 2023-11-16T16:18:26.677 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netlink 2023-11-16T16:18:26.678 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/netlink/netlink_diag.ko 2023-11-16T16:18:26.678 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/nsh 2023-11-16T16:18:26.678 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/nsh/nsh.ko 2023-11-16T16:18:26.678 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/openvswitch 2023-11-16T16:18:26.678 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/openvswitch/openvswitch.ko 2023-11-16T16:18:26.678 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/openvswitch/vport-geneve.ko 2023-11-16T16:18:26.678 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/openvswitch/vport-gre.ko 2023-11-16T16:18:26.678 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/openvswitch/vport-vxlan.ko 2023-11-16T16:18:26.678 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/packet 2023-11-16T16:18:26.678 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/packet/af_packet_diag.ko 2023-11-16T16:18:26.678 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/rds 2023-11-16T16:18:26.678 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/rds/rds.ko 2023-11-16T16:18:26.678 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/rds/rds_rdma.ko 2023-11-16T16:18:26.679 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/rds/rds_tcp.ko 2023-11-16T16:18:26.679 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched 2023-11-16T16:18:26.679 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_csum.ko 2023-11-16T16:18:26.679 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_gact.ko 2023-11-16T16:18:26.679 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_ipt.ko 2023-11-16T16:18:26.679 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_mirred.ko 2023-11-16T16:18:26.679 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_nat.ko 2023-11-16T16:18:26.679 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_pedit.ko 2023-11-16T16:18:26.679 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_police.ko 2023-11-16T16:18:26.679 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_simple.ko 2023-11-16T16:18:26.679 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_skbedit.ko 2023-11-16T16:18:26.679 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/act_vlan.ko 2023-11-16T16:18:26.679 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/cls_basic.ko 2023-11-16T16:18:26.679 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/cls_bpf.ko 2023-11-16T16:18:26.680 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/cls_flow.ko 2023-11-16T16:18:26.680 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/cls_fw.ko 2023-11-16T16:18:26.680 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/cls_route.ko 2023-11-16T16:18:26.680 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/cls_u32.ko 2023-11-16T16:18:26.680 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/em_cmp.ko 2023-11-16T16:18:26.680 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/em_ipset.ko 2023-11-16T16:18:26.680 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/em_meta.ko 2023-11-16T16:18:26.680 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/em_nbyte.ko 2023-11-16T16:18:26.680 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/em_text.ko 2023-11-16T16:18:26.680 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/em_u32.ko 2023-11-16T16:18:26.680 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_choke.ko 2023-11-16T16:18:26.680 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_codel.ko 2023-11-16T16:18:26.680 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_drr.ko 2023-11-16T16:18:26.681 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_fq.ko 2023-11-16T16:18:26.681 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_gred.ko 2023-11-16T16:18:26.681 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_hfsc.ko 2023-11-16T16:18:26.681 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_hhf.ko 2023-11-16T16:18:26.681 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_htb.ko 2023-11-16T16:18:26.681 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_ingress.ko 2023-11-16T16:18:26.681 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_mqprio.ko 2023-11-16T16:18:26.681 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_mqprio_lib.ko 2023-11-16T16:18:26.681 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_multiq.ko 2023-11-16T16:18:26.681 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_netem.ko 2023-11-16T16:18:26.681 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_pie.ko 2023-11-16T16:18:26.681 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_plug.ko 2023-11-16T16:18:26.681 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_prio.ko 2023-11-16T16:18:26.682 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_qfq.ko 2023-11-16T16:18:26.682 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_red.ko 2023-11-16T16:18:26.682 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_sfb.ko 2023-11-16T16:18:26.682 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_sfq.ko 2023-11-16T16:18:26.682 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_tbf.ko 2023-11-16T16:18:26.682 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sched/sch_teql.ko 2023-11-16T16:18:26.682 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sctp 2023-11-16T16:18:26.682 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sctp/sctp.ko 2023-11-16T16:18:26.682 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sctp/sctp_diag.ko 2023-11-16T16:18:26.682 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc 2023-11-16T16:18:26.682 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc/auth_gss 2023-11-16T16:18:26.682 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-11-16T16:18:26.683 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-11-16T16:18:26.683 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc/sunrpc.ko 2023-11-16T16:18:26.683 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc/xprtrdma 2023-11-16T16:18:26.683 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-11-16T16:18:26.683 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/tipc 2023-11-16T16:18:26.683 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/tipc/diag.ko 2023-11-16T16:18:26.683 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/tipc/tipc.ko 2023-11-16T16:18:26.683 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/unix 2023-11-16T16:18:26.683 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/unix/unix_diag.ko 2023-11-16T16:18:26.683 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/vmw_vsock 2023-11-16T16:18:26.683 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-11-16T16:18:26.683 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-11-16T16:18:26.683 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/vmw_vsock/vsock.ko 2023-11-16T16:18:26.684 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/vmw_vsock/vsock_diag.ko 2023-11-16T16:18:26.684 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/vmw_vsock/vsock_loopback.ko 2023-11-16T16:18:26.684 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/xfrm 2023-11-16T16:18:26.684 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/net/xfrm/xfrm_ipcomp.ko 2023-11-16T16:18:26.684 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/security 2023-11-16T16:18:26.684 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/security/keys 2023-11-16T16:18:26.684 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/security/keys/encrypted-keys 2023-11-16T16:18:26.684 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-11-16T16:18:26.684 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/security/keys/trusted-keys 2023-11-16T16:18:26.684 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/security/keys/trusted-keys/trusted.ko 2023-11-16T16:18:26.684 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/virt 2023-11-16T16:18:26.684 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/virt/lib 2023-11-16T16:18:26.684 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/kernel/virt/lib/irqbypass.ko 2023-11-16T16:18:26.684 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.alias 2023-11-16T16:18:26.685 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.alias.bin 2023-11-16T16:18:26.685 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.builtin 2023-11-16T16:18:26.685 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.builtin.bin 2023-11-16T16:18:26.685 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.builtin.modinfo 2023-11-16T16:18:26.685 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.dep 2023-11-16T16:18:26.685 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.dep.bin 2023-11-16T16:18:26.685 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.devname 2023-11-16T16:18:26.685 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.order 2023-11-16T16:18:26.685 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.softdep 2023-11-16T16:18:26.685 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.symbols 2023-11-16T16:18:26.685 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.6.0-rc5-g5b73517e0e2f/modules.symbols.bin 2023-11-16T16:18:26.686 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:26.686 DEBUG:teuthology.orchestra.run.smithi002:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-11-16T16:18:26.714 INFO:teuthology.orchestra.run.smithi002.stdout:postinstall scriptlet (using /bin/sh): 2023-11-16T16:18:26.714 INFO:teuthology.orchestra.run.smithi002.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-rc5-g5b73517e0e2f -a -r /boot/System.map-6.6.0-rc5-g5b73517e0e2f ]; then 2023-11-16T16:18:26.714 INFO:teuthology.orchestra.run.smithi002.stdout:cp /boot/vmlinuz-6.6.0-rc5-g5b73517e0e2f /boot/.vmlinuz-6.6.0-rc5-g5b73517e0e2f-rpm 2023-11-16T16:18:26.714 INFO:teuthology.orchestra.run.smithi002.stdout:cp /boot/System.map-6.6.0-rc5-g5b73517e0e2f /boot/.System.map-6.6.0-rc5-g5b73517e0e2f-rpm 2023-11-16T16:18:26.714 INFO:teuthology.orchestra.run.smithi002.stdout:rm -f /boot/vmlinuz-6.6.0-rc5-g5b73517e0e2f /boot/System.map-6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:26.714 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/installkernel 6.6.0-rc5-g5b73517e0e2f /boot/.vmlinuz-6.6.0-rc5-g5b73517e0e2f-rpm /boot/.System.map-6.6.0-rc5-g5b73517e0e2f-rpm 2023-11-16T16:18:26.714 INFO:teuthology.orchestra.run.smithi002.stdout:rm -f /boot/.vmlinuz-6.6.0-rc5-g5b73517e0e2f-rpm /boot/.System.map-6.6.0-rc5-g5b73517e0e2f-rpm 2023-11-16T16:18:26.715 INFO:teuthology.orchestra.run.smithi002.stdout:fi 2023-11-16T16:18:26.715 INFO:teuthology.orchestra.run.smithi002.stdout:preuninstall scriptlet (using /bin/sh): 2023-11-16T16:18:26.715 INFO:teuthology.orchestra.run.smithi002.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-11-16T16:18:26.715 INFO:teuthology.orchestra.run.smithi002.stdout:new-kernel-pkg --remove 6.6.0-rc5-g5b73517e0e2f --rminitrd --initrdfile=/boot/initramfs-6.6.0-rc5-g5b73517e0e2f.img 2023-11-16T16:18:26.715 INFO:teuthology.orchestra.run.smithi002.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-11-16T16:18:26.715 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-install remove 6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:26.715 INFO:teuthology.orchestra.run.smithi002.stdout:fi 2023-11-16T16:18:26.715 INFO:teuthology.orchestra.run.smithi002.stdout:postuninstall scriptlet (using /bin/sh): 2023-11-16T16:18:26.715 INFO:teuthology.orchestra.run.smithi002.stdout:if [ -x /sbin/update-bootloader ]; then 2023-11-16T16:18:26.715 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/update-bootloader --remove 6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:26.715 INFO:teuthology.orchestra.run.smithi002.stdout:fi 2023-11-16T16:18:26.716 DEBUG:teuthology.orchestra.run.smithi002:> sudo rpm -qi grub2-tools 2023-11-16T16:18:26.797 INFO:teuthology.orchestra.run.smithi002.stdout:Name : grub2-tools 2023-11-16T16:18:26.797 INFO:teuthology.orchestra.run.smithi002.stdout:Epoch : 1 2023-11-16T16:18:26.797 INFO:teuthology.orchestra.run.smithi002.stdout:Version : 2.02 2023-11-16T16:18:26.798 INFO:teuthology.orchestra.run.smithi002.stdout:Release : 123.el8 2023-11-16T16:18:26.798 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: x86_64 2023-11-16T16:18:26.798 INFO:teuthology.orchestra.run.smithi002.stdout:Install Date: Thu 14 Jul 2022 06:22:19 PM UTC 2023-11-16T16:18:26.798 INFO:teuthology.orchestra.run.smithi002.stdout:Group : System Environment/Base 2023-11-16T16:18:26.798 INFO:teuthology.orchestra.run.smithi002.stdout:Size : 9265450 2023-11-16T16:18:26.798 INFO:teuthology.orchestra.run.smithi002.stdout:License : GPLv3+ 2023-11-16T16:18:26.798 INFO:teuthology.orchestra.run.smithi002.stdout:Signature : RSA/SHA256, Thu 31 Mar 2022 07:24:35 PM UTC, Key ID 199e2f91fd431d51 2023-11-16T16:18:26.798 INFO:teuthology.orchestra.run.smithi002.stdout:Source RPM : grub2-2.02-123.el8.src.rpm 2023-11-16T16:18:26.798 INFO:teuthology.orchestra.run.smithi002.stdout:Build Date : Mon 28 Mar 2022 02:12:09 PM UTC 2023-11-16T16:18:26.798 INFO:teuthology.orchestra.run.smithi002.stdout:Build Host : x86-vm-09.build.eng.bos.redhat.com 2023-11-16T16:18:26.798 INFO:teuthology.orchestra.run.smithi002.stdout:Relocations : (not relocatable) 2023-11-16T16:18:26.798 INFO:teuthology.orchestra.run.smithi002.stdout:Packager : Red Hat, Inc. 2023-11-16T16:18:26.799 INFO:teuthology.orchestra.run.smithi002.stdout:Vendor : Red Hat, Inc. 2023-11-16T16:18:26.799 INFO:teuthology.orchestra.run.smithi002.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-16T16:18:26.799 INFO:teuthology.orchestra.run.smithi002.stdout:Summary : Support tools for GRUB. 2023-11-16T16:18:26.799 INFO:teuthology.orchestra.run.smithi002.stdout:Description : 2023-11-16T16:18:26.799 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:18:26.799 INFO:teuthology.orchestra.run.smithi002.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-16T16:18:26.799 INFO:teuthology.orchestra.run.smithi002.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-16T16:18:26.799 INFO:teuthology.orchestra.run.smithi002.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-16T16:18:26.799 INFO:teuthology.orchestra.run.smithi002.stdout:hardware devices. 2023-11-16T16:18:26.799 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:18:26.799 INFO:teuthology.orchestra.run.smithi002.stdout:This subpackage provides tools for support of all platforms. 2023-11-16T16:18:26.800 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-16T16:18:26.800 INFO:teuthology.task.kernel:Updating grub on smithi002 to boot 6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:26.800 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-16T16:18:29.113 INFO:teuthology.orchestra.run.smithi002.stderr:Generating grub configuration file ... 2023-11-16T16:18:31.149 INFO:teuthology.orchestra.run.smithi002.stderr:done 2023-11-16T16:18:31.151 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2023-11-16T16:18:31.173 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.DS1rAA9zhH 2023-11-16T16:18:31.210 DEBUG:teuthology.orchestra.run.smithi002:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.DS1rAA9zhH 2023-11-16T16:18:31.245 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.DS1rAA9zhH 2023-11-16T16:18:31.397 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.DS1rAA9zhH is 6KB 2023-11-16T16:18:31.446 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.DS1rAA9zhH 2023-11-16T16:18:31.463 DEBUG:teuthology.orchestra.run.smithi002:> sudo /bin/ls /boot/loader/entries || true 2023-11-16T16:18:31.528 INFO:teuthology.orchestra.run.smithi002.stdout:5a84457b7b52416e95d47d02e2e4a60f-0-rescue.conf 2023-11-16T16:18:31.528 INFO:teuthology.orchestra.run.smithi002.stdout:5a84457b7b52416e95d47d02e2e4a60f-4.18.0-372.9.1.el8.x86_64.conf 2023-11-16T16:18:31.528 INFO:teuthology.orchestra.run.smithi002.stdout:5a84457b7b52416e95d47d02e2e4a60f-6.6.0-rc5-g5b73517e0e2f.conf 2023-11-16T16:18:31.530 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-set-default 5a84457b7b52416e95d47d02e2e4a60f-6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:18:31.686 DEBUG:teuthology.orchestra.run.smithi002:> sudo shutdown -r now 2023-11-16T16:19:01.688 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-11-16T16:19:01.689 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-11-16T16:19:01.690 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T16:19:20.058 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.2 2023-11-16T16:19:29.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-11-16T16:19:29.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T16:20:29.064 DEBUG:teuthology.orchestra.remote:timed out 2023-11-16T16:20:41.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-11-16T16:20:41.067 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T16:20:41.419 DEBUG:teuthology.orchestra.run.smithi002:> true 2023-11-16T16:20:41.851 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-11-16T16:20:41.851 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.6.0-rc5-g5b73517e0e2f"... 2023-11-16T16:20:41.851 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2023-11-16T16:20:41.895 INFO:teuthology.orchestra.run.smithi002.stdout:6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:20:41.895 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-rc5-g5b73517e0e2f vs 6.6.0-rc5-g5b73517e0e2f 2023-11-16T16:20:41.895 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-16T16:20:41.895 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-11-16T16:20:42.896 DEBUG:teuthology.parallel:result is None 2023-11-16T16:20:42.896 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-16T16:20:42.906 INFO:teuthology.task.internal:Creating test directory... 2023-11-16T16:20:42.907 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-16T16:20:42.912 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-16T16:20:42.915 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-16T16:20:42.955 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-16T16:20:43.027 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-16T16:20:43.037 INFO:teuthology.task.internal:Creating archive directory... 2023-11-16T16:20:43.038 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-16T16:20:43.043 DEBUG:teuthology.orchestra.run.smithi141:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-16T16:20:43.045 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-16T16:20:43.111 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-16T16:20:43.117 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-16T16:20:43.118 DEBUG:teuthology.orchestra.run.smithi141:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-16T16:20:43.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T16:20:43.138 DEBUG:teuthology.orchestra.run.smithi155:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-16T16:20:43.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T16:20:43.156 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-16T16:20:43.161 DEBUG:teuthology.orchestra.run.smithi141:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-16T16:20:43.180 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-16T16:20:43.240 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T16:20:43.257 INFO:teuthology.orchestra.run.smithi141.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T16:20:43.258 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T16:20:43.275 INFO:teuthology.orchestra.run.smithi141.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T16:20:43.291 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T16:20:43.325 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-16T16:20:43.329 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-16T16:20:43.336 INFO:teuthology.task.internal:Configuring sudo... 2023-11-16T16:20:43.336 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-16T16:20:43.340 DEBUG:teuthology.orchestra.run.smithi141:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-16T16:20:43.343 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-16T16:20:43.406 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-16T16:20:43.415 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-16T16:20:43.415 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-16T16:20:43.420 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-16T16:20:43.422 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-16T16:20:43.458 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T16:20:43.569 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T16:20:43.666 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T16:20:43.739 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T16:20:43.821 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T16:20:43.822 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-16T16:20:43.891 DEBUG:teuthology.orchestra.run.smithi141:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T16:20:43.938 DEBUG:teuthology.orchestra.run.smithi141:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T16:20:43.992 DEBUG:teuthology.orchestra.run.smithi141:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T16:20:44.078 DEBUG:teuthology.orchestra.run.smithi141:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T16:20:44.123 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-11-16T16:20:44.123 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-16T16:20:44.193 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T16:20:44.240 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-16T16:20:44.301 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T16:20:44.386 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-16T16:20:44.432 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-11-16T16:20:44.432 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-16T16:20:44.512 DEBUG:teuthology.orchestra.run.smithi002:> sudo service rsyslog restart 2023-11-16T16:20:44.517 DEBUG:teuthology.orchestra.run.smithi141:> sudo service rsyslog restart 2023-11-16T16:20:44.519 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2023-11-16T16:20:44.595 INFO:teuthology.orchestra.run.smithi141.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T16:20:44.619 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T16:20:44.677 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T16:20:45.029 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-16T16:20:45.037 INFO:teuthology.task.internal:Starting timer... 2023-11-16T16:20:45.037 INFO:teuthology.run_tasks:Running task pcp... 2023-11-16T16:20:45.048 INFO:teuthology.run_tasks:Running task selinux... 2023-11-16T16:20:45.063 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-16T16:20:45.063 DEBUG:teuthology.orchestra.run.smithi002:> sudo service auditd rotate 2023-11-16T16:20:45.167 INFO:teuthology.orchestra.run.smithi002.stdout:Rotating logs: 2023-11-16T16:20:45.171 DEBUG:teuthology.orchestra.run.smithi141:> sudo service auditd rotate 2023-11-16T16:20:45.229 INFO:teuthology.orchestra.run.smithi141.stdout:Rotating logs: 2023-11-16T16:20:45.230 DEBUG:teuthology.orchestra.run.smithi155:> sudo service auditd rotate 2023-11-16T16:20:45.298 INFO:teuthology.orchestra.run.smithi155.stdout:Rotating logs: 2023-11-16T16:20:45.299 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-16T16:20:45.300 DEBUG:teuthology.orchestra.run.smithi002:> /usr/sbin/getenforce 2023-11-16T16:20:45.357 INFO:teuthology.orchestra.run.smithi002.stdout:Permissive 2023-11-16T16:20:45.357 DEBUG:teuthology.orchestra.run.smithi141:> /usr/sbin/getenforce 2023-11-16T16:20:45.383 INFO:teuthology.orchestra.run.smithi141.stdout:Permissive 2023-11-16T16:20:45.383 DEBUG:teuthology.orchestra.run.smithi155:> /usr/sbin/getenforce 2023-11-16T16:20:45.402 INFO:teuthology.orchestra.run.smithi155.stdout:Permissive 2023-11-16T16:20:45.402 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi002.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi141.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi155.front.sepia.ceph.com': 'permissive'} 2023-11-16T16:20:45.403 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T16:20:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T16:20:45.475 DEBUG:teuthology.orchestra.run.smithi141:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T16:20:45.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T16:20:45.504 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T16:20:45.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T16:20:45.532 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-16T16:20:45.532 DEBUG:teuthology.orchestra.run.smithi002:> sudo /usr/sbin/setenforce permissive 2023-11-16T16:20:45.611 DEBUG:teuthology.orchestra.run.smithi141:> sudo /usr/sbin/setenforce permissive 2023-11-16T16:20:45.638 DEBUG:teuthology.orchestra.run.smithi155:> sudo /usr/sbin/setenforce permissive 2023-11-16T16:20:45.665 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-16T16:20:45.675 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-16T16:20:45.675 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-16T16:20:45.695 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-16T16:20:45.698 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi002.front.sepia.ceph.com,smithi141.front.sepia.ceph.com,smithi155.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-16T16:34:47.490 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi002.front.sepia.ceph.com'), Remote(name='ubuntu@smithi141.front.sepia.ceph.com'), Remote(name='ubuntu@smithi155.front.sepia.ceph.com')] 2023-11-16T16:34:47.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-11-16T16:34:47.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T16:34:47.583 DEBUG:teuthology.orchestra.run.smithi002:> true 2023-11-16T16:34:47.697 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2023-11-16T16:34:47.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-11-16T16:34:47.697 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T16:34:47.765 DEBUG:teuthology.orchestra.run.smithi141:> true 2023-11-16T16:34:47.832 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-11-16T16:34:47.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2023-11-16T16:34:47.833 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-16T16:34:47.900 DEBUG:teuthology.orchestra.run.smithi155:> true 2023-11-16T16:34:47.969 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2023-11-16T16:34:47.969 INFO:teuthology.run_tasks:Running task clock... 2023-11-16T16:34:47.979 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-16T16:34:47.980 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-16T16:34:47.980 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T16:34:47.985 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-16T16:34:47.985 DEBUG:teuthology.orchestra.run.smithi141:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T16:34:47.987 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-16T16:34:47.987 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T16:34:48.018 INFO:teuthology.orchestra.run.smithi141.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-16T16:34:48.034 INFO:teuthology.orchestra.run.smithi141.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-16T16:34:48.044 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-16T16:34:48.056 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-16T16:34:48.059 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-16T16:34:48.092 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-16T16:34:48.100 INFO:teuthology.orchestra.run.smithi141.stderr:sudo: ntpd: command not found 2023-11-16T16:34:48.112 INFO:teuthology.orchestra.run.smithi141.stdout:506 Cannot talk to daemon 2023-11-16T16:34:48.121 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: ntpd: command not found 2023-11-16T16:34:48.127 INFO:teuthology.orchestra.run.smithi141.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-16T16:34:48.133 INFO:teuthology.orchestra.run.smithi155.stdout:506 Cannot talk to daemon 2023-11-16T16:34:48.140 INFO:teuthology.orchestra.run.smithi141.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-16T16:34:48.147 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-16T16:34:48.162 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-16T16:34:48.188 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: ntpd: command not found 2023-11-16T16:34:48.218 INFO:teuthology.orchestra.run.smithi002.stdout:506 Cannot talk to daemon 2023-11-16T16:34:48.219 INFO:teuthology.orchestra.run.smithi141.stderr:bash: ntpq: command not found 2023-11-16T16:34:48.223 INFO:teuthology.orchestra.run.smithi141.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T16:34:48.223 INFO:teuthology.orchestra.run.smithi141.stdout:=============================================================================== 2023-11-16T16:34:48.223 INFO:teuthology.orchestra.run.smithi141.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:48.223 INFO:teuthology.orchestra.run.smithi141.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:48.223 INFO:teuthology.orchestra.run.smithi141.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:48.223 INFO:teuthology.orchestra.run.smithi141.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:48.227 INFO:teuthology.orchestra.run.smithi155.stderr:bash: ntpq: command not found 2023-11-16T16:34:48.231 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T16:34:48.231 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2023-11-16T16:34:48.231 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:48.231 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:48.231 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:48.231 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:48.254 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-16T16:34:48.289 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-16T16:34:48.381 INFO:teuthology.orchestra.run.smithi002.stderr:bash: ntpq: command not found 2023-11-16T16:34:48.389 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T16:34:48.389 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2023-11-16T16:34:48.389 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:48.389 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:48.389 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:48.389 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T16:34:48.391 INFO:teuthology.run_tasks:Running task pexec... 2023-11-16T16:34:48.399 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-16T16:34:48.399 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T16:34:48.400 DEBUG:teuthology.orchestra.run.smithi002:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-16T16:34:48.400 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi141.front.sepia.ceph.com 2023-11-16T16:34:48.400 DEBUG:teuthology.orchestra.run.smithi141:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-16T16:34:48.400 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi155.front.sepia.ceph.com 2023-11-16T16:34:48.401 DEBUG:teuthology.orchestra.run.smithi155:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-16T16:34:48.771 INFO:teuthology.orchestra.run.smithi141.stdout:Updating Subscription Management repositories. 2023-11-16T16:34:48.773 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-11-16T16:34:48.918 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2023-11-16T16:34:50.268 INFO:teuthology.orchestra.run.smithi141.stdout:Last metadata expiration check: 0:05:16 ago on Thu 16 Nov 2023 04:29:33 PM UTC. 2023-11-16T16:34:50.320 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:05:11 ago on Thu 16 Nov 2023 04:29:38 PM UTC. 2023-11-16T16:34:50.683 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:02:07 ago on Thu 16 Nov 2023 04:32:43 PM UTC. 2023-11-16T16:34:51.983 INFO:teuthology.orchestra.run.smithi141.stdout:Dependencies resolved. 2023-11-16T16:34:51.984 INFO:teuthology.orchestra.run.smithi141.stdout:================================================================================ 2023-11-16T16:34:51.984 INFO:teuthology.orchestra.run.smithi141.stdout: Package Architecture Version Repository Size 2023-11-16T16:34:51.984 INFO:teuthology.orchestra.run.smithi141.stdout:================================================================================ 2023-11-16T16:34:51.984 INFO:teuthology.orchestra.run.smithi141.stdout:Resetting modules: 2023-11-16T16:34:51.984 INFO:teuthology.orchestra.run.smithi141.stdout: container-tools 2023-11-16T16:34:51.984 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-11-16T16:34:51.985 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction Summary 2023-11-16T16:34:51.985 INFO:teuthology.orchestra.run.smithi141.stdout:================================================================================ 2023-11-16T16:34:51.985 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-11-16T16:34:52.205 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-11-16T16:34:52.206 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-11-16T16:34:52.206 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2023-11-16T16:34:52.206 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-11-16T16:34:52.207 INFO:teuthology.orchestra.run.smithi155.stdout:Resetting modules: 2023-11-16T16:34:52.207 INFO:teuthology.orchestra.run.smithi155.stdout: container-tools 2023-11-16T16:34:52.207 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-11-16T16:34:52.207 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-11-16T16:34:52.207 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2023-11-16T16:34:52.207 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-11-16T16:34:52.621 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-11-16T16:34:52.623 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-11-16T16:34:52.623 INFO:teuthology.orchestra.run.smithi002.stdout: Package Architecture Version Repository Size 2023-11-16T16:34:52.623 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-11-16T16:34:52.623 INFO:teuthology.orchestra.run.smithi002.stdout:Resetting modules: 2023-11-16T16:34:52.623 INFO:teuthology.orchestra.run.smithi002.stdout: container-tools 2023-11-16T16:34:52.623 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:34:52.623 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-11-16T16:34:52.623 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2023-11-16T16:34:52.624 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:34:52.947 INFO:teuthology.orchestra.run.smithi141.stdout:Last metadata expiration check: 0:05:19 ago on Thu 16 Nov 2023 04:29:33 PM UTC. 2023-11-16T16:34:53.165 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:05:14 ago on Thu 16 Nov 2023 04:29:38 PM UTC. 2023-11-16T16:34:53.912 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:02:10 ago on Thu 16 Nov 2023 04:32:43 PM UTC. 2023-11-16T16:35:03.221 INFO:teuthology.orchestra.run.smithi141.stdout:Complete! 2023-11-16T16:35:03.685 INFO:teuthology.orchestra.run.smithi141.stdout:Updating Subscription Management repositories. 2023-11-16T16:35:03.807 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-11-16T16:35:04.301 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-11-16T16:35:04.441 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-11-16T16:35:05.056 INFO:teuthology.orchestra.run.smithi141.stdout:Last metadata expiration check: 0:05:31 ago on Thu 16 Nov 2023 04:29:33 PM UTC. 2023-11-16T16:35:05.061 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2023-11-16T16:35:05.829 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:05:27 ago on Thu 16 Nov 2023 04:29:38 PM UTC. 2023-11-16T16:35:06.764 INFO:teuthology.orchestra.run.smithi141.stdout:Dependencies resolved. 2023-11-16T16:35:06.858 INFO:teuthology.orchestra.run.smithi141.stdout:============================================================================================================ 2023-11-16T16:35:06.858 INFO:teuthology.orchestra.run.smithi141.stdout: Package Arch Version Repository Size 2023-11-16T16:35:06.858 INFO:teuthology.orchestra.run.smithi141.stdout:============================================================================================================ 2023-11-16T16:35:06.858 INFO:teuthology.orchestra.run.smithi141.stdout:Installing group/module packages: 2023-11-16T16:35:06.858 INFO:teuthology.orchestra.run.smithi141.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-11-16T16:35:06.858 INFO:teuthology.orchestra.run.smithi141.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-11-16T16:35:06.858 INFO:teuthology.orchestra.run.smithi141.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-11-16T16:35:06.859 INFO:teuthology.orchestra.run.smithi141.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-11-16T16:35:06.859 INFO:teuthology.orchestra.run.smithi141.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-11-16T16:35:06.859 INFO:teuthology.orchestra.run.smithi141.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-11-16T16:35:06.859 INFO:teuthology.orchestra.run.smithi141.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-11-16T16:35:06.859 INFO:teuthology.orchestra.run.smithi141.stdout:Installing dependencies: 2023-11-16T16:35:06.859 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-11-16T16:35:06.859 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-11-16T16:35:06.859 INFO:teuthology.orchestra.run.smithi141.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-11-16T16:35:06.859 INFO:teuthology.orchestra.run.smithi141.stdout:Installing module profiles: 2023-11-16T16:35:06.859 INFO:teuthology.orchestra.run.smithi141.stdout: container-tools/common 2023-11-16T16:35:06.859 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-11-16T16:35:06.859 INFO:teuthology.orchestra.run.smithi141.stdout:Enabling module streams: 2023-11-16T16:35:06.860 INFO:teuthology.orchestra.run.smithi141.stdout: container-tools rhel8 2023-11-16T16:35:06.860 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-11-16T16:35:06.860 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction Summary 2023-11-16T16:35:06.860 INFO:teuthology.orchestra.run.smithi141.stdout:============================================================================================================ 2023-11-16T16:35:06.860 INFO:teuthology.orchestra.run.smithi141.stdout:Install 10 Packages 2023-11-16T16:35:06.860 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-11-16T16:35:06.860 INFO:teuthology.orchestra.run.smithi141.stdout:Total download size: 18 M 2023-11-16T16:35:06.861 INFO:teuthology.orchestra.run.smithi141.stdout:Installed size: 65 M 2023-11-16T16:35:06.861 INFO:teuthology.orchestra.run.smithi141.stdout:Downloading Packages: 2023-11-16T16:35:06.912 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:02:23 ago on Thu 16 Nov 2023 04:32:43 PM UTC. 2023-11-16T16:35:06.942 INFO:teuthology.orchestra.run.smithi141.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 1.1 MB/s | 94 kB 00:00 2023-11-16T16:35:07.051 INFO:teuthology.orchestra.run.smithi141.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 130 kB/s | 25 kB 00:00 2023-11-16T16:35:07.101 INFO:teuthology.orchestra.run.smithi141.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 169 kB/s | 41 kB 00:00 2023-11-16T16:35:07.135 INFO:teuthology.orchestra.run.smithi141.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 257 kB/s | 49 kB 00:00 2023-11-16T16:35:07.176 INFO:teuthology.orchestra.run.smithi141.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 4.1 MB/s | 524 kB 00:00 2023-11-16T16:35:07.252 INFO:teuthology.orchestra.run.smithi141.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.3 MB/s | 149 kB 00:00 2023-11-16T16:35:07.302 INFO:teuthology.orchestra.run.smithi141.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.7 MB/s | 213 kB 00:00 2023-11-16T16:35:07.411 INFO:teuthology.orchestra.run.smithi141.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 26 MB/s | 8.0 MB 00:00 2023-11-16T16:35:07.469 INFO:teuthology.orchestra.run.smithi141.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 13 MB/s | 2.2 MB 00:00 2023-11-16T16:35:07.568 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-11-16T16:35:07.569 INFO:teuthology.orchestra.run.smithi141.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 22 MB/s | 7.0 MB 00:00 2023-11-16T16:35:07.570 INFO:teuthology.orchestra.run.smithi141.stdout:-------------------------------------------------------------------------------- 2023-11-16T16:35:07.570 INFO:teuthology.orchestra.run.smithi141.stdout:Total 26 MB/s | 18 MB 00:00 2023-11-16T16:35:07.660 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================================ 2023-11-16T16:35:07.660 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2023-11-16T16:35:07.660 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================================ 2023-11-16T16:35:07.660 INFO:teuthology.orchestra.run.smithi155.stdout:Installing group/module packages: 2023-11-16T16:35:07.660 INFO:teuthology.orchestra.run.smithi155.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-11-16T16:35:07.660 INFO:teuthology.orchestra.run.smithi155.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-11-16T16:35:07.660 INFO:teuthology.orchestra.run.smithi155.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-11-16T16:35:07.660 INFO:teuthology.orchestra.run.smithi155.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-11-16T16:35:07.661 INFO:teuthology.orchestra.run.smithi155.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-11-16T16:35:07.661 INFO:teuthology.orchestra.run.smithi155.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-11-16T16:35:07.661 INFO:teuthology.orchestra.run.smithi155.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-11-16T16:35:07.661 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2023-11-16T16:35:07.661 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-11-16T16:35:07.661 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-11-16T16:35:07.661 INFO:teuthology.orchestra.run.smithi155.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-11-16T16:35:07.661 INFO:teuthology.orchestra.run.smithi155.stdout:Installing module profiles: 2023-11-16T16:35:07.661 INFO:teuthology.orchestra.run.smithi155.stdout: container-tools/common 2023-11-16T16:35:07.661 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-11-16T16:35:07.661 INFO:teuthology.orchestra.run.smithi155.stdout:Enabling module streams: 2023-11-16T16:35:07.661 INFO:teuthology.orchestra.run.smithi155.stdout: container-tools rhel8 2023-11-16T16:35:07.661 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-11-16T16:35:07.662 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-11-16T16:35:07.662 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================================ 2023-11-16T16:35:07.662 INFO:teuthology.orchestra.run.smithi155.stdout:Install 10 Packages 2023-11-16T16:35:07.662 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-11-16T16:35:07.662 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 18 M 2023-11-16T16:35:07.662 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 65 M 2023-11-16T16:35:07.662 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2023-11-16T16:35:07.770 INFO:teuthology.orchestra.run.smithi141.stdout:Running transaction check 2023-11-16T16:35:07.789 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction check succeeded. 2023-11-16T16:35:07.789 INFO:teuthology.orchestra.run.smithi141.stdout:Running transaction test 2023-11-16T16:35:07.796 INFO:teuthology.orchestra.run.smithi155.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 701 kB/s | 94 kB 00:00 2023-11-16T16:35:07.813 INFO:teuthology.orchestra.run.smithi155.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 270 kB/s | 41 kB 00:00 2023-11-16T16:35:07.839 INFO:teuthology.orchestra.run.smithi155.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 141 kB/s | 25 kB 00:00 2023-11-16T16:35:07.964 INFO:teuthology.orchestra.run.smithi155.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 295 kB/s | 49 kB 00:00 2023-11-16T16:35:07.972 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction test succeeded. 2023-11-16T16:35:07.976 INFO:teuthology.orchestra.run.smithi141.stdout:Running transaction 2023-11-16T16:35:07.997 INFO:teuthology.orchestra.run.smithi155.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 2.8 MB/s | 524 kB 00:00 2023-11-16T16:35:08.098 INFO:teuthology.orchestra.run.smithi155.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.1 MB/s | 149 kB 00:00 2023-11-16T16:35:08.238 INFO:teuthology.orchestra.run.smithi155.stdout:(7/10): buildah-1.26.2-1.module+el8.6.0+16771+2 20 MB/s | 8.0 MB 00:00 2023-11-16T16:35:08.273 INFO:teuthology.orchestra.run.smithi155.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 772 kB/s | 213 kB 00:00 2023-11-16T16:35:08.407 INFO:teuthology.orchestra.run.smithi155.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 23 MB/s | 7.0 MB 00:00 2023-11-16T16:35:08.457 INFO:teuthology.orchestra.run.smithi155.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 10 MB/s | 2.2 MB 00:00 2023-11-16T16:35:08.458 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2023-11-16T16:35:08.458 INFO:teuthology.orchestra.run.smithi155.stdout:Total 23 MB/s | 18 MB 00:00 2023-11-16T16:35:08.505 INFO:teuthology.orchestra.run.smithi141.stdout: Preparing : 1/1 2023-11-16T16:35:08.657 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2023-11-16T16:35:08.676 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2023-11-16T16:35:08.676 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2023-11-16T16:35:08.690 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-11-16T16:35:08.858 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2023-11-16T16:35:08.862 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2023-11-16T16:35:08.880 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T16:35:08.946 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-11-16T16:35:09.052 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================================================ 2023-11-16T16:35:09.052 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2023-11-16T16:35:09.052 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================================================ 2023-11-16T16:35:09.052 INFO:teuthology.orchestra.run.smithi002.stdout:Installing group/module packages: 2023-11-16T16:35:09.052 INFO:teuthology.orchestra.run.smithi002.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-11-16T16:35:09.052 INFO:teuthology.orchestra.run.smithi002.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-11-16T16:35:09.052 INFO:teuthology.orchestra.run.smithi002.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-11-16T16:35:09.052 INFO:teuthology.orchestra.run.smithi002.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-11-16T16:35:09.053 INFO:teuthology.orchestra.run.smithi002.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-11-16T16:35:09.053 INFO:teuthology.orchestra.run.smithi002.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-11-16T16:35:09.053 INFO:teuthology.orchestra.run.smithi002.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-11-16T16:35:09.053 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2023-11-16T16:35:09.053 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-11-16T16:35:09.053 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-11-16T16:35:09.053 INFO:teuthology.orchestra.run.smithi002.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-11-16T16:35:09.053 INFO:teuthology.orchestra.run.smithi002.stdout:Installing module profiles: 2023-11-16T16:35:09.053 INFO:teuthology.orchestra.run.smithi002.stdout: container-tools/common 2023-11-16T16:35:09.053 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:35:09.053 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling module streams: 2023-11-16T16:35:09.053 INFO:teuthology.orchestra.run.smithi002.stdout: container-tools rhel8 2023-11-16T16:35:09.053 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:35:09.054 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-11-16T16:35:09.054 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================================================ 2023-11-16T16:35:09.054 INFO:teuthology.orchestra.run.smithi002.stdout:Install 10 Packages 2023-11-16T16:35:09.054 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:35:09.054 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 18 M 2023-11-16T16:35:09.055 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 65 M 2023-11-16T16:35:09.055 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-11-16T16:35:09.082 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-11-16T16:35:09.226 INFO:teuthology.orchestra.run.smithi002.stdout:(1/10): python3-pytoml-0.1.14-5.git7dea353.el8. 146 kB/s | 25 kB 00:00 2023-11-16T16:35:09.243 INFO:teuthology.orchestra.run.smithi002.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 218 kB/s | 41 kB 00:00 2023-11-16T16:35:09.268 INFO:teuthology.orchestra.run.smithi002.stdout:(3/10): python3-pyxdg-0.25-16.el8.noarch.rpm 441 kB/s | 94 kB 00:00 2023-11-16T16:35:09.287 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-11-16T16:35:09.359 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2023-11-16T16:35:09.360 INFO:teuthology.orchestra.run.smithi002.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 369 kB/s | 49 kB 00:00 2023-11-16T16:35:09.385 INFO:teuthology.orchestra.run.smithi002.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.6 MB/s | 524 kB 00:00 2023-11-16T16:35:09.510 INFO:teuthology.orchestra.run.smithi002.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 994 kB/s | 149 kB 00:00 2023-11-16T16:35:09.536 INFO:teuthology.orchestra.run.smithi002.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.4 MB/s | 213 kB 00:00 2023-11-16T16:35:09.540 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-11-16T16:35:09.633 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-11-16T16:35:09.693 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T16:35:09.778 INFO:teuthology.orchestra.run.smithi002.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 16 MB/s | 8.0 MB 00:00 2023-11-16T16:35:09.853 INFO:teuthology.orchestra.run.smithi002.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 6.9 MB/s | 2.2 MB 00:00 2023-11-16T16:35:09.861 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-11-16T16:35:10.020 INFO:teuthology.orchestra.run.smithi002.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 14 MB/s | 7.0 MB 00:00 2023-11-16T16:35:10.021 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-11-16T16:35:10.021 INFO:teuthology.orchestra.run.smithi002.stdout:Total 19 MB/s | 18 MB 00:00 2023-11-16T16:35:10.028 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-11-16T16:35:10.363 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-11-16T16:35:10.368 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-11-16T16:35:10.393 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-11-16T16:35:10.393 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-11-16T16:35:10.395 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-11-16T16:35:10.922 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-11-16T16:35:10.927 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-11-16T16:35:11.175 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-11-16T16:35:11.252 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-11-16T16:35:11.482 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-11-16T16:35:11.540 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-11-16T16:35:11.636 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-11-16T16:35:11.753 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-11-16T16:35:11.830 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T16:35:11.932 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T16:35:12.042 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-11-16T16:35:12.119 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T16:35:12.119 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-11-16T16:35:12.119 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T16:35:12.119 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-11-16T16:35:12.120 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-11-16T16:35:12.120 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-11-16T16:35:12.120 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-11-16T16:35:12.120 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-11-16T16:35:12.120 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-11-16T16:35:12.120 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-11-16T16:35:12.162 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-11-16T16:35:12.265 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-11-16T16:35:12.408 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-11-16T16:35:12.433 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-11-16T16:35:12.627 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T16:35:12.786 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-11-16T16:35:12.893 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T16:35:12.894 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-11-16T16:35:12.894 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T16:35:12.894 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-11-16T16:35:12.894 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-11-16T16:35:12.894 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-11-16T16:35:12.894 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-11-16T16:35:12.894 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-11-16T16:35:12.895 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-11-16T16:35:12.895 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-11-16T16:35:13.252 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-11-16T16:35:13.253 INFO:teuthology.orchestra.run.smithi141.stdout:Last metadata expiration check: 0:05:39 ago on Thu 16 Nov 2023 04:29:33 PM UTC. 2023-11-16T16:35:13.629 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-11-16T16:35:14.146 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-11-16T16:35:14.146 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:05:35 ago on Thu 16 Nov 2023 04:29:38 PM UTC. 2023-11-16T16:35:14.582 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-11-16T16:35:14.844 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-11-16T16:35:15.046 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-11-16T16:35:15.244 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T16:35:15.661 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-11-16T16:35:15.661 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-11-16T16:35:15.661 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-11-16T16:35:15.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-11-16T16:35:15.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-11-16T16:35:15.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-11-16T16:35:15.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-11-16T16:35:15.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-11-16T16:35:15.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-11-16T16:35:15.662 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-11-16T16:35:17.195 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-11-16T16:35:17.195 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:02:33 ago on Thu 16 Nov 2023 04:32:43 PM UTC. 2023-11-16T16:35:20.970 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2023-11-16T16:35:21.190 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-11-16T16:35:21.190 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2023-11-16T16:35:21.190 INFO:teuthology.orchestra.run.smithi155.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:21.190 INFO:teuthology.orchestra.run.smithi155.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T16:35:21.190 INFO:teuthology.orchestra.run.smithi155.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:21.190 INFO:teuthology.orchestra.run.smithi155.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-11-16T16:35:21.190 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-16T16:35:21.190 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-16T16:35:21.190 INFO:teuthology.orchestra.run.smithi155.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:21.191 INFO:teuthology.orchestra.run.smithi155.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:21.191 INFO:teuthology.orchestra.run.smithi155.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T16:35:21.191 INFO:teuthology.orchestra.run.smithi155.stdout: yajl-2.1.0-10.el8.x86_64 2023-11-16T16:35:21.191 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-11-16T16:35:21.191 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-11-16T16:35:21.343 DEBUG:teuthology.parallel:result is None 2023-11-16T16:35:21.496 INFO:teuthology.orchestra.run.smithi141.stdout:Installed products updated. 2023-11-16T16:35:21.691 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-11-16T16:35:21.691 INFO:teuthology.orchestra.run.smithi141.stdout:Installed: 2023-11-16T16:35:21.691 INFO:teuthology.orchestra.run.smithi141.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:21.691 INFO:teuthology.orchestra.run.smithi141.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T16:35:21.691 INFO:teuthology.orchestra.run.smithi141.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:21.691 INFO:teuthology.orchestra.run.smithi141.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-11-16T16:35:21.691 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-16T16:35:21.691 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-16T16:35:21.692 INFO:teuthology.orchestra.run.smithi141.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:21.692 INFO:teuthology.orchestra.run.smithi141.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:21.692 INFO:teuthology.orchestra.run.smithi141.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T16:35:21.692 INFO:teuthology.orchestra.run.smithi141.stdout: yajl-2.1.0-10.el8.x86_64 2023-11-16T16:35:21.692 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-11-16T16:35:21.692 INFO:teuthology.orchestra.run.smithi141.stdout:Complete! 2023-11-16T16:35:21.843 DEBUG:teuthology.parallel:result is None 2023-11-16T16:35:26.447 INFO:teuthology.orchestra.run.smithi002.stdout:Installed products updated. 2023-11-16T16:35:26.665 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:35:26.665 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-11-16T16:35:26.666 INFO:teuthology.orchestra.run.smithi002.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:26.666 INFO:teuthology.orchestra.run.smithi002.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T16:35:26.666 INFO:teuthology.orchestra.run.smithi002.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:26.666 INFO:teuthology.orchestra.run.smithi002.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-11-16T16:35:26.666 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-16T16:35:26.666 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-16T16:35:26.666 INFO:teuthology.orchestra.run.smithi002.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:26.666 INFO:teuthology.orchestra.run.smithi002.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-11-16T16:35:26.666 INFO:teuthology.orchestra.run.smithi002.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-11-16T16:35:26.666 INFO:teuthology.orchestra.run.smithi002.stdout: yajl-2.1.0-10.el8.x86_64 2023-11-16T16:35:26.666 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:35:26.667 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-11-16T16:35:26.920 DEBUG:teuthology.parallel:result is None 2023-11-16T16:35:26.920 INFO:teuthology.run_tasks:Running task install... 2023-11-16T16:35:26.928 DEBUG:teuthology.task.install:project ceph 2023-11-16T16:35:26.928 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207'}} 2023-11-16T16:35:26.928 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207'} 2023-11-16T16:35:26.929 INFO:teuthology.task.install:Using flavor: default 2023-11-16T16:35:26.934 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-16T16:35:26.934 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-11-16T16:35:26.934 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207', 'tag': None, 'wait_for_package': False} 2023-11-16T16:35:26.935 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:35:26.936 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207', 'tag': None, 'wait_for_package': False} 2023-11-16T16:35:26.936 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:35:26.937 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207', 'tag': None, 'wait_for_package': False} 2023-11-16T16:35:26.937 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:35:27.084 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/ 2023-11-16T16:35:27.084 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3905.g5e4a743a 2023-11-16T16:35:27.088 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/ 2023-11-16T16:35:27.088 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3905.g5e4a743a 2023-11-16T16:35:27.195 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/ 2023-11-16T16:35:27.196 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3905.g5e4a743a 2023-11-16T16:35:27.468 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-16T16:35:27.468 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T16:35:27.468 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-16T16:35:27.479 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-16T16:35:27.479 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-11-16T16:35:27.480 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-16T16:35:27.508 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-16T16:35:27.508 DEBUG:teuthology.orchestra.run.smithi141:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e4a743ad7fd1422ebd6c4dc950a21a303279207/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-16T16:35:27.539 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-16T16:35:27.539 DEBUG:teuthology.orchestra.run.smithi002:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e4a743ad7fd1422ebd6c4dc950a21a303279207/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-16T16:35:27.577 DEBUG:teuthology.orchestra.run.smithi141:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-16T16:35:27.598 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-16T16:35:27.598 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-11-16T16:35:27.598 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-16T16:35:27.626 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-16T16:35:27.626 DEBUG:teuthology.orchestra.run.smithi155:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e4a743ad7fd1422ebd6c4dc950a21a303279207/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-16T16:35:27.652 DEBUG:teuthology.orchestra.run.smithi002:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-16T16:35:27.653 DEBUG:teuthology.orchestra.run.smithi141:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-16T16:35:27.693 DEBUG:teuthology.orchestra.run.smithi155:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-16T16:35:27.719 INFO:teuthology.orchestra.run.smithi141.stdout:check_obsoletes = 1 2023-11-16T16:35:27.720 DEBUG:teuthology.orchestra.run.smithi141:> sudo yum clean all 2023-11-16T16:35:27.754 DEBUG:teuthology.orchestra.run.smithi002:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-16T16:35:27.773 DEBUG:teuthology.orchestra.run.smithi155:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-16T16:35:27.825 INFO:teuthology.orchestra.run.smithi002.stdout:check_obsoletes = 1 2023-11-16T16:35:27.828 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum clean all 2023-11-16T16:35:27.839 INFO:teuthology.orchestra.run.smithi155.stdout:check_obsoletes = 1 2023-11-16T16:35:27.840 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum clean all 2023-11-16T16:35:28.091 INFO:teuthology.orchestra.run.smithi141.stdout:Updating Subscription Management repositories. 2023-11-16T16:35:28.213 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-11-16T16:35:28.317 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2023-11-16T16:35:29.072 INFO:teuthology.orchestra.run.smithi155.stdout:44 files removed 2023-11-16T16:35:29.121 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-16T16:35:29.180 INFO:teuthology.orchestra.run.smithi141.stdout:44 files removed 2023-11-16T16:35:29.235 DEBUG:teuthology.orchestra.run.smithi141:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-16T16:35:29.347 INFO:teuthology.orchestra.run.smithi002.stdout:44 files removed 2023-11-16T16:35:29.404 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-16T16:35:29.454 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2023-11-16T16:35:29.569 INFO:teuthology.orchestra.run.smithi141.stdout:Updating Subscription Management repositories. 2023-11-16T16:35:29.891 INFO:teuthology.orchestra.run.smithi002.stdout:Updating Subscription Management repositories. 2023-11-16T16:35:31.162 INFO:teuthology.orchestra.run.smithi155.stdout:ceph packages for x86_64 119 kB/s | 80 kB 00:00 2023-11-16T16:35:31.358 INFO:teuthology.orchestra.run.smithi141.stdout:ceph packages for x86_64 117 kB/s | 80 kB 00:00 2023-11-16T16:35:31.815 INFO:teuthology.orchestra.run.smithi002.stdout:ceph packages for x86_64 101 kB/s | 80 kB 00:00 2023-11-16T16:35:31.962 INFO:teuthology.orchestra.run.smithi155.stdout:ceph noarch packages 12 kB/s | 9.4 kB 00:00 2023-11-16T16:35:32.084 INFO:teuthology.orchestra.run.smithi141.stdout:ceph noarch packages 14 kB/s | 9.4 kB 00:00 2023-11-16T16:35:32.457 INFO:teuthology.orchestra.run.smithi002.stdout:ceph noarch packages 15 kB/s | 9.4 kB 00:00 2023-11-16T16:35:32.518 INFO:teuthology.orchestra.run.smithi141.stdout:ceph source packages 3.6 kB/s | 1.5 kB 00:00 2023-11-16T16:35:32.571 INFO:teuthology.orchestra.run.smithi155.stdout:ceph source packages 2.5 kB/s | 1.5 kB 00:00 2023-11-16T16:35:32.685 INFO:teuthology.orchestra.run.smithi141.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-11-16T16:35:32.704 INFO:teuthology.orchestra.run.smithi155.stdout:Copr repo for python3-asyncssh owned by ceph 31 kB/s | 3.5 kB 00:00 2023-11-16T16:35:32.917 INFO:teuthology.orchestra.run.smithi002.stdout:ceph source packages 3.4 kB/s | 1.5 kB 00:00 2023-11-16T16:35:33.051 INFO:teuthology.orchestra.run.smithi002.stdout:Copr repo for python3-asyncssh owned by ceph 31 kB/s | 3.5 kB 00:00 2023-11-16T16:35:34.353 INFO:teuthology.orchestra.run.smithi141.stdout:Extra Packages for Enterprise Linux 9.7 MB/s | 16 MB 00:01 2023-11-16T16:35:34.923 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 8.6 MB/s | 16 MB 00:01 2023-11-16T16:35:35.075 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 16 MB 00:02 2023-11-16T16:35:37.919 INFO:teuthology.orchestra.run.smithi141.stdout:lab-extras 457 kB/s | 24 kB 00:00 2023-11-16T16:35:38.622 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 450 kB/s | 24 kB 00:00 2023-11-16T16:35:39.096 INFO:teuthology.orchestra.run.smithi141.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 MB/s | 47 MB 00:01 2023-11-16T16:35:39.422 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 421 kB/s | 24 kB 00:00 2023-11-16T16:35:39.775 INFO:teuthology.orchestra.run.smithi155.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 44 MB/s | 47 MB 00:01 2023-11-16T16:35:40.892 INFO:teuthology.orchestra.run.smithi002.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 MB/s | 47 MB 00:01 2023-11-16T16:35:47.233 INFO:teuthology.orchestra.run.smithi141.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 47 MB/s | 53 MB 00:01 2023-11-16T16:35:48.068 INFO:teuthology.orchestra.run.smithi155.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 45 MB/s | 53 MB 00:01 2023-11-16T16:35:50.707 INFO:teuthology.orchestra.run.smithi002.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 35 MB/s | 53 MB 00:01 2023-11-16T16:35:58.429 INFO:teuthology.orchestra.run.smithi141.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T16:35:58.430 INFO:teuthology.orchestra.run.smithi141.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T16:35:58.444 INFO:teuthology.orchestra.run.smithi141.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-11-16T16:35:58.445 INFO:teuthology.orchestra.run.smithi141.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-11-16T16:35:58.446 INFO:teuthology.orchestra.run.smithi141.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-11-16T16:35:58.448 INFO:teuthology.orchestra.run.smithi141.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-16T16:35:58.450 INFO:teuthology.orchestra.run.smithi141.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-16T16:35:58.453 INFO:teuthology.orchestra.run.smithi141.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-16T16:35:58.454 INFO:teuthology.orchestra.run.smithi141.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-16T16:35:58.587 INFO:teuthology.orchestra.run.smithi141.stdout:Dependencies resolved. 2023-11-16T16:35:58.599 INFO:teuthology.orchestra.run.smithi141.stdout:=========================================================================================================================== 2023-11-16T16:35:58.599 INFO:teuthology.orchestra.run.smithi141.stdout: Package Arch Version Repository Size 2023-11-16T16:35:58.599 INFO:teuthology.orchestra.run.smithi141.stdout:=========================================================================================================================== 2023-11-16T16:35:58.599 INFO:teuthology.orchestra.run.smithi141.stdout:Installing: 2023-11-16T16:35:58.599 INFO:teuthology.orchestra.run.smithi141.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-11-16T16:35:58.599 INFO:teuthology.orchestra.run.smithi141.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-11-16T16:35:58.600 INFO:teuthology.orchestra.run.smithi141.stdout: ceph x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 6.4 k 2023-11-16T16:35:58.600 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-base x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 5.3 M 2023-11-16T16:35:58.600 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-fuse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 847 k 2023-11-16T16:35:58.600 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 165 k 2023-11-16T16:35:58.600 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 1.5 M 2023-11-16T16:35:58.600 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 127 k 2023-11-16T16:35:58.600 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 1.7 M 2023-11-16T16:35:58.600 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 7.4 M 2023-11-16T16:35:58.600 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-rook noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 47 k 2023-11-16T16:35:58.600 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-radosgw x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 13 M 2023-11-16T16:35:58.600 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-test x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 41 M 2023-11-16T16:35:58.601 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-volume noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 265 k 2023-11-16T16:35:58.601 INFO:teuthology.orchestra.run.smithi141.stdout: cephadm noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 83 k 2023-11-16T16:35:58.601 INFO:teuthology.orchestra.run.smithi141.stdout: cephfs-mirror x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 258 k 2023-11-16T16:35:58.601 INFO:teuthology.orchestra.run.smithi141.stdout: cephfs-top noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 18 k 2023-11-16T16:35:58.601 INFO:teuthology.orchestra.run.smithi141.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-11-16T16:35:58.601 INFO:teuthology.orchestra.run.smithi141.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-11-16T16:35:58.601 INFO:teuthology.orchestra.run.smithi141.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-11-16T16:35:58.601 INFO:teuthology.orchestra.run.smithi141.stdout: libcephfs-devel x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 31 k 2023-11-16T16:35:58.601 INFO:teuthology.orchestra.run.smithi141.stdout: libcephfs2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 771 k 2023-11-16T16:35:58.601 INFO:teuthology.orchestra.run.smithi141.stdout: librados-devel x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 139 k 2023-11-16T16:35:58.601 INFO:teuthology.orchestra.run.smithi141.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-11-16T16:35:58.601 INFO:teuthology.orchestra.run.smithi141.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-11-16T16:35:58.602 INFO:teuthology.orchestra.run.smithi141.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-11-16T16:35:58.602 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-11-16T16:35:58.602 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-11-16T16:35:58.602 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-11-16T16:35:58.602 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cephfs x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 197 k 2023-11-16T16:35:58.602 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rados x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 371 k 2023-11-16T16:35:58.602 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rbd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 365 k 2023-11-16T16:35:58.602 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rgw x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 107 k 2023-11-16T16:35:58.602 INFO:teuthology.orchestra.run.smithi141.stdout: rbd-fuse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 94 k 2023-11-16T16:35:58.602 INFO:teuthology.orchestra.run.smithi141.stdout: rbd-mirror x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.4 M 2023-11-16T16:35:58.602 INFO:teuthology.orchestra.run.smithi141.stdout: rbd-nbd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 180 k 2023-11-16T16:35:58.603 INFO:teuthology.orchestra.run.smithi141.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-11-16T16:35:58.603 INFO:teuthology.orchestra.run.smithi141.stdout:Upgrading: 2023-11-16T16:35:58.603 INFO:teuthology.orchestra.run.smithi141.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-11-16T16:35:58.603 INFO:teuthology.orchestra.run.smithi141.stdout: librados2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.7 M 2023-11-16T16:35:58.603 INFO:teuthology.orchestra.run.smithi141.stdout: librbd1 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.6 M 2023-11-16T16:35:58.603 INFO:teuthology.orchestra.run.smithi141.stdout:Installing dependencies: 2023-11-16T16:35:58.603 INFO:teuthology.orchestra.run.smithi141.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-11-16T16:35:58.603 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-common x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 22 M 2023-11-16T16:35:58.603 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 23 k 2023-11-16T16:35:58.603 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mds x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 2.3 M 2023-11-16T16:35:58.603 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 240 k 2023-11-16T16:35:58.603 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mon x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 4.6 M 2023-11-16T16:35:58.604 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-osd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 18 M 2023-11-16T16:35:58.604 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 15 k 2023-11-16T16:35:58.604 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-selinux x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 24 k 2023-11-16T16:35:58.604 INFO:teuthology.orchestra.run.smithi141.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-16T16:35:58.604 INFO:teuthology.orchestra.run.smithi141.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-16T16:35:58.604 INFO:teuthology.orchestra.run.smithi141.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-16T16:35:58.604 INFO:teuthology.orchestra.run.smithi141.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-11-16T16:35:58.604 INFO:teuthology.orchestra.run.smithi141.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-11-16T16:35:58.604 INFO:teuthology.orchestra.run.smithi141.stdout: libcephsqlite x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 137 k 2023-11-16T16:35:58.604 INFO:teuthology.orchestra.run.smithi141.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-11-16T16:35:58.604 INFO:teuthology.orchestra.run.smithi141.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-16T16:35:58.605 INFO:teuthology.orchestra.run.smithi141.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-16T16:35:58.605 INFO:teuthology.orchestra.run.smithi141.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-11-16T16:35:58.605 INFO:teuthology.orchestra.run.smithi141.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-16T16:35:58.605 INFO:teuthology.orchestra.run.smithi141.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-16T16:35:58.605 INFO:teuthology.orchestra.run.smithi141.stdout: libradosstriper1 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 473 k 2023-11-16T16:35:58.605 INFO:teuthology.orchestra.run.smithi141.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-16T16:35:58.605 INFO:teuthology.orchestra.run.smithi141.stdout: librgw2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 6.5 M 2023-11-16T16:35:58.605 INFO:teuthology.orchestra.run.smithi141.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-11-16T16:35:58.605 INFO:teuthology.orchestra.run.smithi141.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-11-16T16:35:58.605 INFO:teuthology.orchestra.run.smithi141.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-16T16:35:58.605 INFO:teuthology.orchestra.run.smithi141.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-11-16T16:35:58.606 INFO:teuthology.orchestra.run.smithi141.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-16T16:35:58.606 INFO:teuthology.orchestra.run.smithi141.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-16T16:35:58.606 INFO:teuthology.orchestra.run.smithi141.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-16T16:35:58.606 INFO:teuthology.orchestra.run.smithi141.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-16T16:35:58.606 INFO:teuthology.orchestra.run.smithi141.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-11-16T16:35:58.606 INFO:teuthology.orchestra.run.smithi141.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-11-16T16:35:58.606 INFO:teuthology.orchestra.run.smithi141.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-11-16T16:35:58.606 INFO:teuthology.orchestra.run.smithi141.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-16T16:35:58.606 INFO:teuthology.orchestra.run.smithi141.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-16T16:35:58.606 INFO:teuthology.orchestra.run.smithi141.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-16T16:35:58.606 INFO:teuthology.orchestra.run.smithi141.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-16T16:35:58.606 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-16T16:35:58.607 INFO:teuthology.orchestra.run.smithi141.stdout: python3-ceph-argparse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 45 k 2023-11-16T16:35:58.607 INFO:teuthology.orchestra.run.smithi141.stdout: python3-ceph-common x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 119 k 2023-11-16T16:35:58.607 INFO:teuthology.orchestra.run.smithi141.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-16T16:35:58.607 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-16T16:35:58.607 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-16T16:35:58.607 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-16T16:35:58.607 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-16T16:35:58.607 INFO:teuthology.orchestra.run.smithi141.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-16T16:35:58.607 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-16T16:35:58.607 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-16T16:35:58.607 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-16T16:35:58.607 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-16T16:35:58.607 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-16T16:35:58.608 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T16:35:58.608 INFO:teuthology.orchestra.run.smithi141.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-16T16:35:58.608 INFO:teuthology.orchestra.run.smithi141.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-16T16:35:58.608 INFO:teuthology.orchestra.run.smithi141.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-16T16:35:58.608 INFO:teuthology.orchestra.run.smithi141.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-16T16:35:58.608 INFO:teuthology.orchestra.run.smithi141.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-16T16:35:58.608 INFO:teuthology.orchestra.run.smithi141.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-16T16:35:58.608 INFO:teuthology.orchestra.run.smithi141.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-16T16:35:58.608 INFO:teuthology.orchestra.run.smithi141.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-16T16:35:58.608 INFO:teuthology.orchestra.run.smithi141.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-16T16:35:58.608 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-16T16:35:58.608 INFO:teuthology.orchestra.run.smithi141.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-16T16:35:58.609 INFO:teuthology.orchestra.run.smithi141.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-16T16:35:58.609 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-16T16:35:58.609 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-16T16:35:58.609 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-16T16:35:58.609 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-16T16:35:58.609 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-16T16:35:58.609 INFO:teuthology.orchestra.run.smithi141.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-16T16:35:58.609 INFO:teuthology.orchestra.run.smithi141.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T16:35:58.609 INFO:teuthology.orchestra.run.smithi141.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-16T16:35:58.609 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-16T16:35:58.609 INFO:teuthology.orchestra.run.smithi141.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-16T16:35:58.610 INFO:teuthology.orchestra.run.smithi141.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-16T16:35:58.610 INFO:teuthology.orchestra.run.smithi141.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-16T16:35:58.610 INFO:teuthology.orchestra.run.smithi141.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-16T16:35:58.610 INFO:teuthology.orchestra.run.smithi141.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-16T16:35:58.610 INFO:teuthology.orchestra.run.smithi141.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-16T16:35:58.610 INFO:teuthology.orchestra.run.smithi141.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-16T16:35:58.610 INFO:teuthology.orchestra.run.smithi141.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-16T16:35:58.610 INFO:teuthology.orchestra.run.smithi141.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-16T16:35:58.610 INFO:teuthology.orchestra.run.smithi141.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-16T16:35:58.610 INFO:teuthology.orchestra.run.smithi141.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-16T16:35:58.610 INFO:teuthology.orchestra.run.smithi141.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-16T16:35:58.610 INFO:teuthology.orchestra.run.smithi141.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-16T16:35:58.611 INFO:teuthology.orchestra.run.smithi141.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-11-16T16:35:58.611 INFO:teuthology.orchestra.run.smithi141.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-16T16:35:58.611 INFO:teuthology.orchestra.run.smithi141.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-11-16T16:35:58.611 INFO:teuthology.orchestra.run.smithi141.stdout:Enabling module streams: 2023-11-16T16:35:58.611 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql 10 2023-11-16T16:35:58.611 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-11-16T16:35:58.611 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction Summary 2023-11-16T16:35:58.611 INFO:teuthology.orchestra.run.smithi141.stdout:=========================================================================================================================== 2023-11-16T16:35:58.611 INFO:teuthology.orchestra.run.smithi141.stdout:Install 128 Packages 2023-11-16T16:35:58.611 INFO:teuthology.orchestra.run.smithi141.stdout:Upgrade 3 Packages 2023-11-16T16:35:58.611 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-11-16T16:35:58.612 INFO:teuthology.orchestra.run.smithi141.stdout:Total download size: 197 M 2023-11-16T16:35:58.612 INFO:teuthology.orchestra.run.smithi141.stdout:Downloading Packages: 2023-11-16T16:35:58.856 INFO:teuthology.orchestra.run.smithi141.stdout:(1/131): ceph-18.0.0-3905.g5e4a743a.el8.x86_64. 36 kB/s | 6.4 kB 00:00 2023-11-16T16:35:59.189 INFO:teuthology.orchestra.run.smithi141.stdout:(2/131): ceph-fuse-18.0.0-3905.g5e4a743a.el8.x8 2.5 MB/s | 847 kB 00:00 2023-11-16T16:35:59.257 INFO:teuthology.orchestra.run.smithi141.stdout:(3/131): ceph-immutable-object-cache-18.0.0-390 2.4 MB/s | 165 kB 00:00 2023-11-16T16:35:59.382 INFO:teuthology.orchestra.run.smithi155.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T16:35:59.384 INFO:teuthology.orchestra.run.smithi155.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T16:35:59.398 INFO:teuthology.orchestra.run.smithi155.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-11-16T16:35:59.399 INFO:teuthology.orchestra.run.smithi155.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-11-16T16:35:59.400 INFO:teuthology.orchestra.run.smithi155.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-11-16T16:35:59.401 INFO:teuthology.orchestra.run.smithi155.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-16T16:35:59.403 INFO:teuthology.orchestra.run.smithi155.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-16T16:35:59.406 INFO:teuthology.orchestra.run.smithi155.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-16T16:35:59.407 INFO:teuthology.orchestra.run.smithi155.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-16T16:35:59.541 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2023-11-16T16:35:59.553 INFO:teuthology.orchestra.run.smithi155.stdout:=========================================================================================================================== 2023-11-16T16:35:59.553 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2023-11-16T16:35:59.554 INFO:teuthology.orchestra.run.smithi155.stdout:=========================================================================================================================== 2023-11-16T16:35:59.554 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2023-11-16T16:35:59.554 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-11-16T16:35:59.554 INFO:teuthology.orchestra.run.smithi155.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-11-16T16:35:59.554 INFO:teuthology.orchestra.run.smithi155.stdout: ceph x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 6.4 k 2023-11-16T16:35:59.554 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 5.3 M 2023-11-16T16:35:59.554 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 847 k 2023-11-16T16:35:59.554 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 165 k 2023-11-16T16:35:59.554 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 1.5 M 2023-11-16T16:35:59.554 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 127 k 2023-11-16T16:35:59.555 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 1.7 M 2023-11-16T16:35:59.555 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 7.4 M 2023-11-16T16:35:59.555 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 47 k 2023-11-16T16:35:59.555 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 13 M 2023-11-16T16:35:59.555 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 41 M 2023-11-16T16:35:59.555 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 265 k 2023-11-16T16:35:59.555 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 83 k 2023-11-16T16:35:59.555 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-mirror x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 258 k 2023-11-16T16:35:59.555 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-top noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 18 k 2023-11-16T16:35:59.555 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-11-16T16:35:59.556 INFO:teuthology.orchestra.run.smithi155.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-11-16T16:35:59.556 INFO:teuthology.orchestra.run.smithi155.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-11-16T16:35:59.556 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 31 k 2023-11-16T16:35:59.556 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 771 k 2023-11-16T16:35:59.556 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 139 k 2023-11-16T16:35:59.556 INFO:teuthology.orchestra.run.smithi155.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-11-16T16:35:59.556 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-11-16T16:35:59.556 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-11-16T16:35:59.556 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-11-16T16:35:59.556 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-11-16T16:35:59.556 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-11-16T16:35:59.557 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 197 k 2023-11-16T16:35:59.557 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 371 k 2023-11-16T16:35:59.557 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 365 k 2023-11-16T16:35:59.557 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 107 k 2023-11-16T16:35:59.557 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 94 k 2023-11-16T16:35:59.557 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.4 M 2023-11-16T16:35:59.557 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 180 k 2023-11-16T16:35:59.557 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-11-16T16:35:59.557 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrading: 2023-11-16T16:35:59.557 INFO:teuthology.orchestra.run.smithi155.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-11-16T16:35:59.558 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.7 M 2023-11-16T16:35:59.558 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.6 M 2023-11-16T16:35:59.558 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2023-11-16T16:35:59.558 INFO:teuthology.orchestra.run.smithi155.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-11-16T16:35:59.558 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 22 M 2023-11-16T16:35:59.558 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 23 k 2023-11-16T16:35:59.558 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 2.3 M 2023-11-16T16:35:59.558 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 240 k 2023-11-16T16:35:59.558 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 4.6 M 2023-11-16T16:35:59.559 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 18 M 2023-11-16T16:35:59.559 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 15 k 2023-11-16T16:35:59.559 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 24 k 2023-11-16T16:35:59.559 INFO:teuthology.orchestra.run.smithi155.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-16T16:35:59.559 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-16T16:35:59.559 INFO:teuthology.orchestra.run.smithi155.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-16T16:35:59.559 INFO:teuthology.orchestra.run.smithi155.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-11-16T16:35:59.559 INFO:teuthology.orchestra.run.smithi155.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-11-16T16:35:59.559 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 137 k 2023-11-16T16:35:59.559 INFO:teuthology.orchestra.run.smithi155.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-11-16T16:35:59.559 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-16T16:35:59.560 INFO:teuthology.orchestra.run.smithi155.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-16T16:35:59.560 INFO:teuthology.orchestra.run.smithi155.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-11-16T16:35:59.560 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-16T16:35:59.560 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-16T16:35:59.560 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 473 k 2023-11-16T16:35:59.560 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-16T16:35:59.560 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 6.5 M 2023-11-16T16:35:59.560 INFO:teuthology.orchestra.run.smithi155.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-11-16T16:35:59.560 INFO:teuthology.orchestra.run.smithi155.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-11-16T16:35:59.561 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-16T16:35:59.561 INFO:teuthology.orchestra.run.smithi155.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-11-16T16:35:59.561 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-16T16:35:59.561 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-16T16:35:59.561 INFO:teuthology.orchestra.run.smithi155.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-16T16:35:59.561 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-16T16:35:59.561 INFO:teuthology.orchestra.run.smithi155.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-11-16T16:35:59.561 INFO:teuthology.orchestra.run.smithi155.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-11-16T16:35:59.561 INFO:teuthology.orchestra.run.smithi155.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-11-16T16:35:59.561 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-16T16:35:59.561 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-16T16:35:59.562 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-16T16:35:59.562 INFO:teuthology.orchestra.run.smithi155.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-16T16:35:59.562 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-16T16:35:59.562 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 45 k 2023-11-16T16:35:59.562 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 119 k 2023-11-16T16:35:59.562 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-16T16:35:59.562 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-16T16:35:59.562 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-16T16:35:59.562 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-16T16:35:59.562 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-16T16:35:59.563 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-16T16:35:59.563 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-16T16:35:59.563 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-16T16:35:59.563 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-16T16:35:59.563 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-16T16:35:59.563 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-16T16:35:59.563 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T16:35:59.563 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-16T16:35:59.563 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-16T16:35:59.563 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-16T16:35:59.564 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-16T16:35:59.564 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-16T16:35:59.564 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-16T16:35:59.564 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-16T16:35:59.564 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-16T16:35:59.564 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-16T16:35:59.564 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-16T16:35:59.564 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-16T16:35:59.564 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-16T16:35:59.564 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-16T16:35:59.565 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-16T16:35:59.565 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-16T16:35:59.565 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-16T16:35:59.565 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-16T16:35:59.565 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-16T16:35:59.565 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T16:35:59.565 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-16T16:35:59.565 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-16T16:35:59.565 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-16T16:35:59.565 INFO:teuthology.orchestra.run.smithi155.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-16T16:35:59.566 INFO:teuthology.orchestra.run.smithi155.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-16T16:35:59.566 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-16T16:35:59.566 INFO:teuthology.orchestra.run.smithi155.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-16T16:35:59.566 INFO:teuthology.orchestra.run.smithi155.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-16T16:35:59.566 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-16T16:35:59.566 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-16T16:35:59.566 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-16T16:35:59.566 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-16T16:35:59.566 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-16T16:35:59.566 INFO:teuthology.orchestra.run.smithi155.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-16T16:35:59.567 INFO:teuthology.orchestra.run.smithi155.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-16T16:35:59.567 INFO:teuthology.orchestra.run.smithi155.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-11-16T16:35:59.567 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-16T16:35:59.567 INFO:teuthology.orchestra.run.smithi155.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-11-16T16:35:59.567 INFO:teuthology.orchestra.run.smithi155.stdout:Enabling module streams: 2023-11-16T16:35:59.567 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql 10 2023-11-16T16:35:59.567 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-11-16T16:35:59.567 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2023-11-16T16:35:59.567 INFO:teuthology.orchestra.run.smithi155.stdout:=========================================================================================================================== 2023-11-16T16:35:59.567 INFO:teuthology.orchestra.run.smithi155.stdout:Install 128 Packages 2023-11-16T16:35:59.567 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 3 Packages 2023-11-16T16:35:59.568 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-11-16T16:35:59.568 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 197 M 2023-11-16T16:35:59.568 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2023-11-16T16:35:59.799 INFO:teuthology.orchestra.run.smithi141.stdout:(4/131): ceph-mds-18.0.0-3905.g5e4a743a.el8.x86 4.2 MB/s | 2.3 MB 00:00 2023-11-16T16:35:59.935 INFO:teuthology.orchestra.run.smithi155.stdout:(1/131): ceph-18.0.0-3905.g5e4a743a.el8.x86_64. 30 kB/s | 6.4 kB 00:00 2023-11-16T16:36:00.190 INFO:teuthology.orchestra.run.smithi141.stdout:(5/131): ceph-mgr-18.0.0-3905.g5e4a743a.el8.x86 4.0 MB/s | 1.5 MB 00:00 2023-11-16T16:36:00.307 INFO:teuthology.orchestra.run.smithi141.stdout:(6/131): ceph-base-18.0.0-3905.g5e4a743a.el8.x8 3.3 MB/s | 5.3 MB 00:01 2023-11-16T16:36:00.860 INFO:teuthology.orchestra.run.smithi155.stdout:(2/131): ceph-fuse-18.0.0-3905.g5e4a743a.el8.x8 916 kB/s | 847 kB 00:00 2023-11-16T16:36:01.027 INFO:teuthology.orchestra.run.smithi155.stdout:(3/131): ceph-immutable-object-cache-18.0.0-390 994 kB/s | 165 kB 00:00 2023-11-16T16:36:01.224 INFO:teuthology.orchestra.run.smithi141.stdout:(7/131): ceph-mon-18.0.0-3905.g5e4a743a.el8.x86 4.5 MB/s | 4.6 MB 00:01 2023-11-16T16:36:01.708 INFO:teuthology.orchestra.run.smithi141.stdout:(8/131): ceph-common-18.0.0-3905.g5e4a743a.el8. 7.1 MB/s | 22 MB 00:03 2023-11-16T16:36:01.791 INFO:teuthology.orchestra.run.smithi141.stdout:(9/131): ceph-selinux-18.0.0-3905.g5e4a743a.el8 293 kB/s | 24 kB 00:00 2023-11-16T16:36:02.661 INFO:teuthology.orchestra.run.smithi155.stdout:(4/131): ceph-mds-18.0.0-3905.g5e4a743a.el8.x86 1.4 MB/s | 2.3 MB 00:01 2023-11-16T16:36:03.219 INFO:teuthology.orchestra.run.smithi155.stdout:(5/131): ceph-base-18.0.0-3905.g5e4a743a.el8.x8 1.5 MB/s | 5.3 MB 00:03 2023-11-16T16:36:03.424 INFO:teuthology.orchestra.run.smithi002.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T16:36:03.425 INFO:teuthology.orchestra.run.smithi002.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-16T16:36:03.440 INFO:teuthology.orchestra.run.smithi002.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-11-16T16:36:03.442 INFO:teuthology.orchestra.run.smithi002.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-11-16T16:36:03.443 INFO:teuthology.orchestra.run.smithi002.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-11-16T16:36:03.444 INFO:teuthology.orchestra.run.smithi002.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-16T16:36:03.447 INFO:teuthology.orchestra.run.smithi002.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-16T16:36:03.450 INFO:teuthology.orchestra.run.smithi002.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-16T16:36:03.451 INFO:teuthology.orchestra.run.smithi002.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-16T16:36:03.602 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2023-11-16T16:36:03.616 INFO:teuthology.orchestra.run.smithi002.stdout:=========================================================================================================================== 2023-11-16T16:36:03.616 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2023-11-16T16:36:03.616 INFO:teuthology.orchestra.run.smithi002.stdout:=========================================================================================================================== 2023-11-16T16:36:03.616 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2023-11-16T16:36:03.616 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-11-16T16:36:03.617 INFO:teuthology.orchestra.run.smithi002.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-11-16T16:36:03.617 INFO:teuthology.orchestra.run.smithi002.stdout: ceph x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 6.4 k 2023-11-16T16:36:03.617 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 5.3 M 2023-11-16T16:36:03.617 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-fuse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 847 k 2023-11-16T16:36:03.617 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 165 k 2023-11-16T16:36:03.617 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 1.5 M 2023-11-16T16:36:03.617 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 127 k 2023-11-16T16:36:03.617 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 1.7 M 2023-11-16T16:36:03.617 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 7.4 M 2023-11-16T16:36:03.617 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-rook noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 47 k 2023-11-16T16:36:03.617 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-radosgw x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 13 M 2023-11-16T16:36:03.618 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-test x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 41 M 2023-11-16T16:36:03.618 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-volume noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 265 k 2023-11-16T16:36:03.618 INFO:teuthology.orchestra.run.smithi002.stdout: cephadm noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 83 k 2023-11-16T16:36:03.618 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-mirror x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 258 k 2023-11-16T16:36:03.618 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-top noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 18 k 2023-11-16T16:36:03.618 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-11-16T16:36:03.618 INFO:teuthology.orchestra.run.smithi002.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-11-16T16:36:03.618 INFO:teuthology.orchestra.run.smithi002.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-11-16T16:36:03.618 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-devel x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 31 k 2023-11-16T16:36:03.618 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 771 k 2023-11-16T16:36:03.618 INFO:teuthology.orchestra.run.smithi002.stdout: librados-devel x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 139 k 2023-11-16T16:36:03.619 INFO:teuthology.orchestra.run.smithi002.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-11-16T16:36:03.619 INFO:teuthology.orchestra.run.smithi002.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-11-16T16:36:03.619 INFO:teuthology.orchestra.run.smithi002.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-11-16T16:36:03.619 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-11-16T16:36:03.619 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-11-16T16:36:03.619 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-11-16T16:36:03.619 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cephfs x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 197 k 2023-11-16T16:36:03.619 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rados x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 371 k 2023-11-16T16:36:03.620 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rbd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 365 k 2023-11-16T16:36:03.620 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rgw x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 107 k 2023-11-16T16:36:03.620 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-fuse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 94 k 2023-11-16T16:36:03.620 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-mirror x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.4 M 2023-11-16T16:36:03.620 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-nbd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 180 k 2023-11-16T16:36:03.620 INFO:teuthology.orchestra.run.smithi002.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-11-16T16:36:03.620 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrading: 2023-11-16T16:36:03.620 INFO:teuthology.orchestra.run.smithi002.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-11-16T16:36:03.620 INFO:teuthology.orchestra.run.smithi002.stdout: librados2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.7 M 2023-11-16T16:36:03.620 INFO:teuthology.orchestra.run.smithi002.stdout: librbd1 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 3.6 M 2023-11-16T16:36:03.620 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2023-11-16T16:36:03.621 INFO:teuthology.orchestra.run.smithi002.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-11-16T16:36:03.621 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-common x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 22 M 2023-11-16T16:36:03.621 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 23 k 2023-11-16T16:36:03.621 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mds x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 2.3 M 2023-11-16T16:36:03.621 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 240 k 2023-11-16T16:36:03.621 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mon x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 4.6 M 2023-11-16T16:36:03.621 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-osd x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 18 M 2023-11-16T16:36:03.621 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3905.g5e4a743a.el8 ceph-noarch 15 k 2023-11-16T16:36:03.621 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-selinux x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 24 k 2023-11-16T16:36:03.621 INFO:teuthology.orchestra.run.smithi002.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-16T16:36:03.621 INFO:teuthology.orchestra.run.smithi002.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-16T16:36:03.622 INFO:teuthology.orchestra.run.smithi002.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-16T16:36:03.622 INFO:teuthology.orchestra.run.smithi002.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-11-16T16:36:03.622 INFO:teuthology.orchestra.run.smithi002.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-11-16T16:36:03.622 INFO:teuthology.orchestra.run.smithi002.stdout: libcephsqlite x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 137 k 2023-11-16T16:36:03.622 INFO:teuthology.orchestra.run.smithi002.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-11-16T16:36:03.622 INFO:teuthology.orchestra.run.smithi002.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-16T16:36:03.622 INFO:teuthology.orchestra.run.smithi002.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-16T16:36:03.622 INFO:teuthology.orchestra.run.smithi002.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-11-16T16:36:03.622 INFO:teuthology.orchestra.run.smithi002.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-16T16:36:03.622 INFO:teuthology.orchestra.run.smithi002.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-16T16:36:03.622 INFO:teuthology.orchestra.run.smithi002.stdout: libradosstriper1 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 473 k 2023-11-16T16:36:03.623 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-16T16:36:03.623 INFO:teuthology.orchestra.run.smithi002.stdout: librgw2 x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 6.5 M 2023-11-16T16:36:03.623 INFO:teuthology.orchestra.run.smithi002.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-11-16T16:36:03.623 INFO:teuthology.orchestra.run.smithi002.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-11-16T16:36:03.623 INFO:teuthology.orchestra.run.smithi002.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-16T16:36:03.623 INFO:teuthology.orchestra.run.smithi002.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-11-16T16:36:03.623 INFO:teuthology.orchestra.run.smithi002.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-16T16:36:03.623 INFO:teuthology.orchestra.run.smithi002.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-16T16:36:03.623 INFO:teuthology.orchestra.run.smithi002.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-16T16:36:03.623 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-16T16:36:03.624 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-11-16T16:36:03.624 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-11-16T16:36:03.624 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-11-16T16:36:03.624 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-16T16:36:03.624 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-16T16:36:03.624 INFO:teuthology.orchestra.run.smithi002.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-16T16:36:03.624 INFO:teuthology.orchestra.run.smithi002.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-16T16:36:03.624 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-16T16:36:03.624 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 45 k 2023-11-16T16:36:03.624 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common x86_64 2:18.0.0-3905.g5e4a743a.el8 ceph 119 k 2023-11-16T16:36:03.625 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-16T16:36:03.625 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-16T16:36:03.625 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-16T16:36:03.625 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-16T16:36:03.625 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-16T16:36:03.625 INFO:teuthology.orchestra.run.smithi002.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-16T16:36:03.625 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-16T16:36:03.625 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-16T16:36:03.625 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-16T16:36:03.625 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-16T16:36:03.626 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-16T16:36:03.626 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T16:36:03.626 INFO:teuthology.orchestra.run.smithi002.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-16T16:36:03.626 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-16T16:36:03.626 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-16T16:36:03.626 INFO:teuthology.orchestra.run.smithi002.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-16T16:36:03.626 INFO:teuthology.orchestra.run.smithi002.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-16T16:36:03.626 INFO:teuthology.orchestra.run.smithi002.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-16T16:36:03.626 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-16T16:36:03.626 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-16T16:36:03.627 INFO:teuthology.orchestra.run.smithi002.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-16T16:36:03.627 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-16T16:36:03.627 INFO:teuthology.orchestra.run.smithi002.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-16T16:36:03.627 INFO:teuthology.orchestra.run.smithi002.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-16T16:36:03.627 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-16T16:36:03.627 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-16T16:36:03.627 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-16T16:36:03.627 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-16T16:36:03.627 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-16T16:36:03.627 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-16T16:36:03.627 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-16T16:36:03.627 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-16T16:36:03.628 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-16T16:36:03.628 INFO:teuthology.orchestra.run.smithi002.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-16T16:36:03.628 INFO:teuthology.orchestra.run.smithi002.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-16T16:36:03.628 INFO:teuthology.orchestra.run.smithi002.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-16T16:36:03.628 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-16T16:36:03.628 INFO:teuthology.orchestra.run.smithi002.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-16T16:36:03.628 INFO:teuthology.orchestra.run.smithi002.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-16T16:36:03.628 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-16T16:36:03.628 INFO:teuthology.orchestra.run.smithi002.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-16T16:36:03.628 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-16T16:36:03.629 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-16T16:36:03.629 INFO:teuthology.orchestra.run.smithi002.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-16T16:36:03.629 INFO:teuthology.orchestra.run.smithi002.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-16T16:36:03.629 INFO:teuthology.orchestra.run.smithi002.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-16T16:36:03.629 INFO:teuthology.orchestra.run.smithi002.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-11-16T16:36:03.629 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-16T16:36:03.629 INFO:teuthology.orchestra.run.smithi002.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-11-16T16:36:03.629 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling module streams: 2023-11-16T16:36:03.629 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql 10 2023-11-16T16:36:03.629 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:36:03.630 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2023-11-16T16:36:03.630 INFO:teuthology.orchestra.run.smithi002.stdout:=========================================================================================================================== 2023-11-16T16:36:03.630 INFO:teuthology.orchestra.run.smithi002.stdout:Install 128 Packages 2023-11-16T16:36:03.630 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrade 3 Packages 2023-11-16T16:36:03.630 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:36:03.630 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 197 M 2023-11-16T16:36:03.631 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2023-11-16T16:36:03.669 INFO:teuthology.orchestra.run.smithi155.stdout:(6/131): ceph-mgr-18.0.0-3905.g5e4a743a.el8.x86 1.5 MB/s | 1.5 MB 00:01 2023-11-16T16:36:03.935 INFO:teuthology.orchestra.run.smithi141.stdout:(10/131): ceph-radosgw-18.0.0-3905.g5e4a743a.el 4.8 MB/s | 13 MB 00:02 2023-11-16T16:36:04.042 INFO:teuthology.orchestra.run.smithi141.stdout:(11/131): cephfs-mirror-18.0.0-3905.g5e4a743a.e 2.4 MB/s | 258 kB 00:00 2023-11-16T16:36:04.142 INFO:teuthology.orchestra.run.smithi141.stdout:(12/131): libcephfs-devel-18.0.0-3905.g5e4a743a 311 kB/s | 31 kB 00:00 2023-11-16T16:36:04.272 INFO:teuthology.orchestra.run.smithi002.stdout:(1/131): ceph-18.0.0-3905.g5e4a743a.el8.x86_64. 14 kB/s | 6.4 kB 00:00 2023-11-16T16:36:04.334 INFO:teuthology.orchestra.run.smithi141.stdout:(13/131): ceph-osd-18.0.0-3905.g5e4a743a.el8.x8 4.4 MB/s | 18 MB 00:04 2023-11-16T16:36:04.368 INFO:teuthology.orchestra.run.smithi141.stdout:(14/131): libcephfs2-18.0.0-3905.g5e4a743a.el8. 3.3 MB/s | 771 kB 00:00 2023-11-16T16:36:04.418 INFO:teuthology.orchestra.run.smithi141.stdout:(15/131): libcephsqlite-18.0.0-3905.g5e4a743a.e 1.6 MB/s | 137 kB 00:00 2023-11-16T16:36:04.468 INFO:teuthology.orchestra.run.smithi141.stdout:(16/131): librados-devel-18.0.0-3905.g5e4a743a. 1.4 MB/s | 139 kB 00:00 2023-11-16T16:36:04.585 INFO:teuthology.orchestra.run.smithi141.stdout:(17/131): libradosstriper1-18.0.0-3905.g5e4a743 2.8 MB/s | 473 kB 00:00 2023-11-16T16:36:04.677 INFO:teuthology.orchestra.run.smithi141.stdout:(18/131): python3-ceph-argparse-18.0.0-3905.g5e 488 kB/s | 45 kB 00:00 2023-11-16T16:36:04.777 INFO:teuthology.orchestra.run.smithi141.stdout:(19/131): python3-ceph-common-18.0.0-3905.g5e4a 1.2 MB/s | 119 kB 00:00 2023-11-16T16:36:04.870 INFO:teuthology.orchestra.run.smithi141.stdout:(20/131): python3-cephfs-18.0.0-3905.g5e4a743a. 2.1 MB/s | 197 kB 00:00 2023-11-16T16:36:05.045 INFO:teuthology.orchestra.run.smithi141.stdout:(21/131): python3-rados-18.0.0-3905.g5e4a743a.e 2.1 MB/s | 371 kB 00:00 2023-11-16T16:36:05.195 INFO:teuthology.orchestra.run.smithi141.stdout:(22/131): python3-rbd-18.0.0-3905.g5e4a743a.el8 2.4 MB/s | 365 kB 00:00 2023-11-16T16:36:05.270 INFO:teuthology.orchestra.run.smithi141.stdout:(23/131): python3-rgw-18.0.0-3905.g5e4a743a.el8 1.4 MB/s | 107 kB 00:00 2023-11-16T16:36:05.336 INFO:teuthology.orchestra.run.smithi155.stdout:(7/131): ceph-common-18.0.0-3905.g5e4a743a.el8. 3.8 MB/s | 22 MB 00:05 2023-11-16T16:36:05.354 INFO:teuthology.orchestra.run.smithi141.stdout:(24/131): rbd-fuse-18.0.0-3905.g5e4a743a.el8.x8 1.1 MB/s | 94 kB 00:00 2023-11-16T16:36:05.403 INFO:teuthology.orchestra.run.smithi155.stdout:(8/131): ceph-mon-18.0.0-3905.g5e4a743a.el8.x86 2.1 MB/s | 4.6 MB 00:02 2023-11-16T16:36:05.456 INFO:teuthology.orchestra.run.smithi002.stdout:(2/131): ceph-fuse-18.0.0-3905.g5e4a743a.el8.x8 716 kB/s | 847 kB 00:01 2023-11-16T16:36:05.503 INFO:teuthology.orchestra.run.smithi155.stdout:(9/131): ceph-selinux-18.0.0-3905.g5e4a743a.el8 244 kB/s | 24 kB 00:00 2023-11-16T16:36:05.787 INFO:teuthology.orchestra.run.smithi141.stdout:(25/131): librgw2-18.0.0-3905.g5e4a743a.el8.x86 4.9 MB/s | 6.5 MB 00:01 2023-11-16T16:36:05.880 INFO:teuthology.orchestra.run.smithi141.stdout:(26/131): rbd-nbd-18.0.0-3905.g5e4a743a.el8.x86 1.9 MB/s | 180 kB 00:00 2023-11-16T16:36:05.897 INFO:teuthology.orchestra.run.smithi002.stdout:(3/131): ceph-immutable-object-cache-18.0.0-390 375 kB/s | 165 kB 00:00 2023-11-16T16:36:05.963 INFO:teuthology.orchestra.run.smithi141.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3905.g 278 kB/s | 23 kB 00:00 2023-11-16T16:36:06.055 INFO:teuthology.orchestra.run.smithi141.stdout:(28/131): ceph-mgr-cephadm-18.0.0-3905.g5e4a743 1.4 MB/s | 127 kB 00:00 2023-11-16T16:36:06.431 INFO:teuthology.orchestra.run.smithi141.stdout:(29/131): rbd-mirror-18.0.0-3905.g5e4a743a.el8. 3.2 MB/s | 3.4 MB 00:01 2023-11-16T16:36:06.531 INFO:teuthology.orchestra.run.smithi141.stdout:(30/131): ceph-mgr-dashboard-18.0.0-3905.g5e4a7 3.6 MB/s | 1.7 MB 00:00 2023-11-16T16:36:06.647 INFO:teuthology.orchestra.run.smithi141.stdout:(31/131): ceph-mgr-modules-core-18.0.0-3905.g5e 2.0 MB/s | 240 kB 00:00 2023-11-16T16:36:06.765 INFO:teuthology.orchestra.run.smithi141.stdout:(32/131): ceph-mgr-rook-18.0.0-3905.g5e4a743a.e 406 kB/s | 47 kB 00:00 2023-11-16T16:36:06.873 INFO:teuthology.orchestra.run.smithi141.stdout:(33/131): ceph-prometheus-alerts-18.0.0-3905.g5 134 kB/s | 15 kB 00:00 2023-11-16T16:36:06.982 INFO:teuthology.orchestra.run.smithi141.stdout:(34/131): ceph-volume-18.0.0-3905.g5e4a743a.el8 2.4 MB/s | 265 kB 00:00 2023-11-16T16:36:07.366 INFO:teuthology.orchestra.run.smithi141.stdout:(35/131): ceph-test-18.0.0-3905.g5e4a743a.el8.x 7.3 MB/s | 41 MB 00:05 2023-11-16T16:36:07.382 INFO:teuthology.orchestra.run.smithi141.stdout:(36/131): cephadm-18.0.0-3905.g5e4a743a.el8.noa 209 kB/s | 83 kB 00:00 2023-11-16T16:36:07.414 INFO:teuthology.orchestra.run.smithi002.stdout:(4/131): ceph-base-18.0.0-3905.g5e4a743a.el8.x8 1.5 MB/s | 5.3 MB 00:03 2023-11-16T16:36:07.449 INFO:teuthology.orchestra.run.smithi141.stdout:(37/131): cephfs-top-18.0.0-3905.g5e4a743a.el8. 216 kB/s | 18 kB 00:00 2023-11-16T16:36:07.900 INFO:teuthology.orchestra.run.smithi141.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 131 kB/s | 59 kB 00:00 2023-11-16T16:36:08.025 INFO:teuthology.orchestra.run.smithi141.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 477 kB/s | 306 kB 00:00 2023-11-16T16:36:08.058 INFO:teuthology.orchestra.run.smithi141.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 474 kB/s | 75 kB 00:00 2023-11-16T16:36:08.164 INFO:teuthology.orchestra.run.smithi002.stdout:(5/131): ceph-common-18.0.0-3905.g5e4a743a.el8. 5.0 MB/s | 22 MB 00:04 2023-11-16T16:36:08.200 INFO:teuthology.orchestra.run.smithi141.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 308 kB/s | 44 kB 00:00 2023-11-16T16:36:08.223 INFO:teuthology.orchestra.run.smithi002.stdout:(6/131): ceph-mds-18.0.0-3905.g5e4a743a.el8.x86 1.0 MB/s | 2.3 MB 00:02 2023-11-16T16:36:08.242 INFO:teuthology.orchestra.run.smithi141.stdout:(42/131): python3-asyncssh-2.7.0-2.el8.noarch.r 2.1 MB/s | 476 kB 00:00 2023-11-16T16:36:08.351 INFO:teuthology.orchestra.run.smithi141.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 304 kB/s | 33 kB 00:00 2023-11-16T16:36:08.415 INFO:teuthology.orchestra.run.smithi002.stdout:(7/131): ceph-mgr-18.0.0-3905.g5e4a743a.el8.x86 1.5 MB/s | 1.5 MB 00:01 2023-11-16T16:36:08.418 INFO:teuthology.orchestra.run.smithi141.stdout:(44/131): python3-beautifulsoup4-4.6.3-2.el8.1. 853 kB/s | 185 kB 00:00 2023-11-16T16:36:08.443 INFO:teuthology.orchestra.run.smithi141.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 178 kB/s | 16 kB 00:00 2023-11-16T16:36:08.560 INFO:teuthology.orchestra.run.smithi141.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.2 MB/s | 173 kB 00:00 2023-11-16T16:36:08.585 INFO:teuthology.orchestra.run.smithi141.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 2.6 MB/s | 384 kB 00:00 2023-11-16T16:36:08.690 INFO:teuthology.orchestra.run.smithi002.stdout:(8/131): ceph-mon-18.0.0-3905.g5e4a743a.el8.x86 8.8 MB/s | 4.6 MB 00:00 2023-11-16T16:36:08.710 INFO:teuthology.orchestra.run.smithi141.stdout:(48/131): ceph-mgr-diskprediction-local-18.0.0- 3.2 MB/s | 7.4 MB 00:02 2023-11-16T16:36:08.728 INFO:teuthology.orchestra.run.smithi141.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 78 kB/s | 11 kB 00:00 2023-11-16T16:36:08.749 INFO:teuthology.orchestra.run.smithi002.stdout:(9/131): ceph-selinux-18.0.0-3905.g5e4a743a.el8 417 kB/s | 24 kB 00:00 2023-11-16T16:36:08.753 INFO:teuthology.orchestra.run.smithi141.stdout:(50/131): python3-google-auth-1.1.1-10.el8.noar 422 kB/s | 81 kB 00:00 2023-11-16T16:36:08.811 INFO:teuthology.orchestra.run.smithi141.stdout:(51/131): python3-jaraco-functools-2.0-4.el8.no 175 kB/s | 18 kB 00:00 2023-11-16T16:36:08.887 INFO:teuthology.orchestra.run.smithi141.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 363 kB/s | 49 kB 00:00 2023-11-16T16:36:08.921 INFO:teuthology.orchestra.run.smithi141.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 561 kB/s | 59 kB 00:00 2023-11-16T16:36:09.078 INFO:teuthology.orchestra.run.smithi155.stdout:(10/131): ceph-radosgw-18.0.0-3905.g5e4a743a.el 3.5 MB/s | 13 MB 00:03 2023-11-16T16:36:09.194 INFO:teuthology.orchestra.run.smithi141.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 196 kB/s | 60 kB 00:00 2023-11-16T16:36:09.203 INFO:teuthology.orchestra.run.smithi155.stdout:(11/131): cephfs-mirror-18.0.0-3905.g5e4a743a.e 2.0 MB/s | 258 kB 00:00 2023-11-16T16:36:09.221 INFO:teuthology.orchestra.run.smithi141.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 941 kB/s | 283 kB 00:00 2023-11-16T16:36:09.305 INFO:teuthology.orchestra.run.smithi141.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 142 kB/s | 16 kB 00:00 2023-11-16T16:36:09.330 INFO:teuthology.orchestra.run.smithi141.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 310 kB/s | 34 kB 00:00 2023-11-16T16:36:09.430 INFO:teuthology.orchestra.run.smithi141.stdout:(58/131): python3-rsa-4.9-2.el8.noarch.rpm 643 kB/s | 64 kB 00:00 2023-11-16T16:36:09.455 INFO:teuthology.orchestra.run.smithi141.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 1.3 MB/s | 196 kB 00:00 2023-11-16T16:36:09.524 INFO:teuthology.orchestra.run.smithi155.stdout:(12/131): libcephfs-devel-18.0.0-3905.g5e4a743a 96 kB/s | 31 kB 00:00 2023-11-16T16:36:09.531 INFO:teuthology.orchestra.run.smithi141.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 187 kB/s | 19 kB 00:00 2023-11-16T16:36:09.573 INFO:teuthology.orchestra.run.smithi141.stdout:(61/131): python3-kubernetes-11.0.0-6.el8.noarc 1.4 MB/s | 1.2 MB 00:00 2023-11-16T16:36:09.598 INFO:teuthology.orchestra.run.smithi141.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 168 kB/s | 24 kB 00:00 2023-11-16T16:36:09.657 INFO:teuthology.orchestra.run.smithi141.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 233 kB/s | 29 kB 00:00 2023-11-16T16:36:09.682 INFO:teuthology.orchestra.run.smithi141.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 253 kB/s | 27 kB 00:00 2023-11-16T16:36:09.757 INFO:teuthology.orchestra.run.smithi141.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 1.6 MB/s | 255 kB 00:00 2023-11-16T16:36:09.783 INFO:teuthology.orchestra.run.smithi141.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.0 MB/s | 251 kB 00:00 2023-11-16T16:36:09.808 INFO:teuthology.orchestra.run.smithi141.stdout:(67/131): python3-websocket-client-0.56.0-5.el8 488 kB/s | 61 kB 00:00 2023-11-16T16:36:09.812 INFO:teuthology.orchestra.run.smithi155.stdout:(13/131): libcephfs2-18.0.0-3905.g5e4a743a.el8. 2.6 MB/s | 771 kB 00:00 2023-11-16T16:36:09.875 INFO:teuthology.orchestra.run.smithi141.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 730 kB/s | 85 kB 00:00 2023-11-16T16:36:09.896 INFO:teuthology.orchestra.run.smithi155.stdout:(14/131): libcephsqlite-18.0.0-3905.g5e4a743a.e 1.6 MB/s | 137 kB 00:00 2023-11-16T16:36:09.900 INFO:teuthology.orchestra.run.smithi141.stdout:(69/131): python3-zc-lockfile-2.0-2.el8.noarch. 196 kB/s | 23 kB 00:00 2023-11-16T16:36:09.926 INFO:teuthology.orchestra.run.smithi141.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2023-11-16T16:36:09.971 INFO:teuthology.orchestra.run.smithi155.stdout:(15/131): librados-devel-18.0.0-3905.g5e4a743a. 1.8 MB/s | 139 kB 00:00 2023-11-16T16:36:10.034 INFO:teuthology.orchestra.run.smithi141.stdout:(71/131): thrift-0.13.0-2.el8.x86_64.rpm 7.7 MB/s | 1.7 MB 00:00 2023-11-16T16:36:10.109 INFO:teuthology.orchestra.run.smithi141.stdout:(72/131): python3-pyasn1-modules-0.3.7-6.el8.no 620 kB/s | 110 kB 00:00 2023-11-16T16:36:10.155 INFO:teuthology.orchestra.run.smithi155.stdout:(16/131): libradosstriper1-18.0.0-3905.g5e4a743 2.5 MB/s | 473 kB 00:00 2023-11-16T16:36:10.193 INFO:teuthology.orchestra.run.smithi141.stdout:(73/131): python3-pytz-2017.2-9.el8.noarch.rpm 342 kB/s | 54 kB 00:00 2023-11-16T16:36:10.251 INFO:teuthology.orchestra.run.smithi141.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.8 MB/s | 259 kB 00:00 2023-11-16T16:36:10.327 INFO:teuthology.orchestra.run.smithi141.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 2.4 MB/s | 323 kB 00:00 2023-11-16T16:36:10.502 INFO:teuthology.orchestra.run.smithi141.stdout:(76/131): python3-scipy-1.0.0-21.module+el8.5.0 23 MB/s | 14 MB 00:00 2023-11-16T16:36:10.527 INFO:teuthology.orchestra.run.smithi141.stdout:(77/131): python3-markupsafe-0.23-19.el8.x86_64 141 kB/s | 39 kB 00:00 2023-11-16T16:36:10.569 INFO:teuthology.orchestra.run.smithi141.stdout:(78/131): bison-3.0.4-10.el8.x86_64.rpm 2.8 MB/s | 688 kB 00:00 2023-11-16T16:36:10.603 INFO:teuthology.orchestra.run.smithi141.stdout:(79/131): python3-jinja2-2.10.1-3.el8.noarch.rp 5.2 MB/s | 538 kB 00:00 2023-11-16T16:36:10.653 INFO:teuthology.orchestra.run.smithi141.stdout:(80/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.2 MB/s | 157 kB 00:00 2023-11-16T16:36:10.687 INFO:teuthology.orchestra.run.smithi141.stdout:(81/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2023-11-16T16:36:10.688 INFO:teuthology.orchestra.run.smithi155.stdout:(17/131): ceph-osd-18.0.0-3905.g5e4a743a.el8.x8 2.5 MB/s | 18 MB 00:07 2023-11-16T16:36:10.754 INFO:teuthology.orchestra.run.smithi141.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 4.6 MB/s | 713 kB 00:00 2023-11-16T16:36:10.780 INFO:teuthology.orchestra.run.smithi155.stdout:(18/131): python3-ceph-argparse-18.0.0-3905.g5e 490 kB/s | 45 kB 00:00 2023-11-16T16:36:10.812 INFO:teuthology.orchestra.run.smithi141.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 212 kB/s | 27 kB 00:00 2023-11-16T16:36:10.864 INFO:teuthology.orchestra.run.smithi155.stdout:(19/131): python3-ceph-common-18.0.0-3905.g5e4a 1.4 MB/s | 119 kB 00:00 2023-11-16T16:36:10.879 INFO:teuthology.orchestra.run.smithi141.stdout:(84/131): python3-babel-2.5.1-7.el8.noarch.rpm 21 MB/s | 4.8 MB 00:00 2023-11-16T16:36:10.905 INFO:teuthology.orchestra.run.smithi141.stdout:(85/131): flex-2.6.1-9.el8.x86_64.rpm 2.1 MB/s | 320 kB 00:00 2023-11-16T16:36:10.922 INFO:teuthology.orchestra.run.smithi141.stdout:(86/131): python3-prettytable-0.7.2-14.el8.noar 403 kB/s | 44 kB 00:00 2023-11-16T16:36:11.006 INFO:teuthology.orchestra.run.smithi155.stdout:(20/131): python3-cephfs-18.0.0-3905.g5e4a743a. 1.4 MB/s | 197 kB 00:00 2023-11-16T16:36:11.055 INFO:teuthology.orchestra.run.smithi141.stdout:(87/131): python3-numpy-1.14.3-10.el8.x86_64.rp 21 MB/s | 3.7 MB 00:00 2023-11-16T16:36:11.080 INFO:teuthology.orchestra.run.smithi141.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.4 MB/s | 225 kB 00:00 2023-11-16T16:36:11.106 INFO:teuthology.orchestra.run.smithi141.stdout:(89/131): python3-werkzeug-0.12.2-4.el8.noarch. 2.2 MB/s | 457 kB 00:00 2023-11-16T16:36:11.147 INFO:teuthology.orchestra.run.smithi155.stdout:(21/131): python3-rados-18.0.0-3905.g5e4a743a.e 2.6 MB/s | 371 kB 00:00 2023-11-16T16:36:11.197 INFO:teuthology.orchestra.run.smithi141.stdout:(90/131): uuid-1.6.2-43.el8.x86_64.rpm 451 kB/s | 64 kB 00:00 2023-11-16T16:36:11.256 INFO:teuthology.orchestra.run.smithi141.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 133 kB/s | 20 kB 00:00 2023-11-16T16:36:11.273 INFO:teuthology.orchestra.run.smithi155.stdout:(22/131): python3-rbd-18.0.0-3905.g5e4a743a.el8 2.9 MB/s | 365 kB 00:00 2023-11-16T16:36:11.281 INFO:teuthology.orchestra.run.smithi141.stdout:(92/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 629 kB/s | 126 kB 00:00 2023-11-16T16:36:11.323 INFO:teuthology.orchestra.run.smithi155.stdout:(23/131): python3-rgw-18.0.0-3905.g5e4a743a.el8 2.1 MB/s | 107 kB 00:00 2023-11-16T16:36:11.357 INFO:teuthology.orchestra.run.smithi141.stdout:(93/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 651 kB/s | 103 kB 00:00 2023-11-16T16:36:11.373 INFO:teuthology.orchestra.run.smithi155.stdout:(24/131): rbd-fuse-18.0.0-3905.g5e4a743a.el8.x8 1.8 MB/s | 94 kB 00:00 2023-11-16T16:36:11.432 INFO:teuthology.orchestra.run.smithi141.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 3.9 MB/s | 709 kB 00:00 2023-11-16T16:36:11.474 INFO:teuthology.orchestra.run.smithi141.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 1.7 MB/s | 198 kB 00:00 2023-11-16T16:36:11.549 INFO:teuthology.orchestra.run.smithi141.stdout:(96/131): openblas-threads-0.3.15-3.el8.x86_64. 18 MB/s | 4.9 MB 00:00 2023-11-16T16:36:11.616 INFO:teuthology.orchestra.run.smithi002.stdout:(10/131): ceph-test-18.0.0-3905.g5e4a743a.el8.x 14 MB/s | 41 MB 00:02 2023-11-16T16:36:11.674 INFO:teuthology.orchestra.run.smithi002.stdout:(11/131): cephfs-mirror-18.0.0-3905.g5e4a743a.e 4.4 MB/s | 258 kB 00:00 2023-11-16T16:36:11.708 INFO:teuthology.orchestra.run.smithi002.stdout:(12/131): libcephfs-devel-18.0.0-3905.g5e4a743a 935 kB/s | 31 kB 00:00 2023-11-16T16:36:11.716 INFO:teuthology.orchestra.run.smithi141.stdout:(97/131): openblas-0.3.15-3.el8.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2023-11-16T16:36:11.741 INFO:teuthology.orchestra.run.smithi141.stdout:(98/131): jq-1.6-3.el8.x86_64.rpm 756 kB/s | 202 kB 00:00 2023-11-16T16:36:11.757 INFO:teuthology.orchestra.run.smithi155.stdout:(25/131): librgw2-18.0.0-3905.g5e4a743a.el8.x86 4.1 MB/s | 6.5 MB 00:01 2023-11-16T16:36:11.766 INFO:teuthology.orchestra.run.smithi141.stdout:(99/131): librdkafka-0.11.4-3.el8.x86_64.rpm 1.6 MB/s | 354 kB 00:00 2023-11-16T16:36:11.800 INFO:teuthology.orchestra.run.smithi002.stdout:(13/131): libcephfs2-18.0.0-3905.g5e4a743a.el8. 8.2 MB/s | 771 kB 00:00 2023-11-16T16:36:11.816 INFO:teuthology.orchestra.run.smithi155.stdout:(26/131): rbd-nbd-18.0.0-3905.g5e4a743a.el8.x86 3.0 MB/s | 180 kB 00:00 2023-11-16T16:36:11.850 INFO:teuthology.orchestra.run.smithi002.stdout:(14/131): libcephsqlite-18.0.0-3905.g5e4a743a.e 2.7 MB/s | 137 kB 00:00 2023-11-16T16:36:11.866 INFO:teuthology.orchestra.run.smithi155.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3905.g 461 kB/s | 23 kB 00:00 2023-11-16T16:36:11.900 INFO:teuthology.orchestra.run.smithi002.stdout:(15/131): librados-devel-18.0.0-3905.g5e4a743a. 2.7 MB/s | 139 kB 00:00 2023-11-16T16:36:11.950 INFO:teuthology.orchestra.run.smithi155.stdout:(28/131): rbd-mirror-18.0.0-3905.g5e4a743a.el8. 6.0 MB/s | 3.4 MB 00:00 2023-11-16T16:36:11.959 INFO:teuthology.orchestra.run.smithi002.stdout:(16/131): libradosstriper1-18.0.0-3905.g5e4a743 8.0 MB/s | 473 kB 00:00 2023-11-16T16:36:11.975 INFO:teuthology.orchestra.run.smithi155.stdout:(29/131): ceph-mgr-cephadm-18.0.0-3905.g5e4a743 1.1 MB/s | 127 kB 00:00 2023-11-16T16:36:11.975 INFO:teuthology.orchestra.run.smithi141.stdout:(100/131): postgresql-server-10.21-2.module+el8 20 MB/s | 5.1 MB 00:00 2023-11-16T16:36:12.009 INFO:teuthology.orchestra.run.smithi141.stdout:(101/131): postgresql-contrib-10.21-2.module+el 3.0 MB/s | 808 kB 00:00 2023-11-16T16:36:12.059 INFO:teuthology.orchestra.run.smithi141.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 5.1 MB/s | 1.5 MB 00:00 2023-11-16T16:36:12.109 INFO:teuthology.orchestra.run.smithi141.stdout:(103/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 350 kB/s | 47 kB 00:00 2023-11-16T16:36:12.135 INFO:teuthology.orchestra.run.smithi141.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 564 kB/s | 43 kB 00:00 2023-11-16T16:36:12.160 INFO:teuthology.orchestra.run.smithi141.stdout:(105/131): libselinux-devel-2.9-5.el8.x86_64.rp 1.3 MB/s | 200 kB 00:00 2023-11-16T16:36:12.268 INFO:teuthology.orchestra.run.smithi141.stdout:(106/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.4 MB/s | 220 kB 00:00 2023-11-16T16:36:12.294 INFO:teuthology.orchestra.run.smithi141.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 652 kB/s | 87 kB 00:00 2023-11-16T16:36:12.319 INFO:teuthology.orchestra.run.smithi141.stdout:(108/131): keyutils-libs-devel-1.5.10-9.el8.x86 261 kB/s | 48 kB 00:00 2023-11-16T16:36:12.444 INFO:teuthology.orchestra.run.smithi141.stdout:(109/131): python3-requests-oauthlib-1.0.0-1.el 243 kB/s | 43 kB 00:00 2023-11-16T16:36:12.458 INFO:teuthology.orchestra.run.smithi155.stdout:(30/131): ceph-mgr-dashboard-18.0.0-3905.g5e4a7 3.4 MB/s | 1.7 MB 00:00 2023-11-16T16:36:12.470 INFO:teuthology.orchestra.run.smithi141.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.5 MB/s | 229 kB 00:00 2023-11-16T16:36:12.495 INFO:teuthology.orchestra.run.smithi141.stdout:(111/131): python3-pycparser-2.14-14.el8.noarch 542 kB/s | 109 kB 00:00 2023-11-16T16:36:12.501 INFO:teuthology.orchestra.run.smithi002.stdout:(17/131): librgw2-18.0.0-3905.g5e4a743a.el8.x86 12 MB/s | 6.5 MB 00:00 2023-11-16T16:36:12.534 INFO:teuthology.orchestra.run.smithi155.stdout:(31/131): ceph-mgr-modules-core-18.0.0-3905.g5e 3.1 MB/s | 240 kB 00:00 2023-11-16T16:36:12.546 INFO:teuthology.orchestra.run.smithi141.stdout:(112/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 5.4 MB/s | 560 kB 00:00 2023-11-16T16:36:12.584 INFO:teuthology.orchestra.run.smithi155.stdout:(32/131): ceph-mgr-rook-18.0.0-3905.g5e4a743a.e 951 kB/s | 47 kB 00:00 2023-11-16T16:36:12.588 INFO:teuthology.orchestra.run.smithi141.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 1.6 MB/s | 155 kB 00:00 2023-11-16T16:36:12.621 INFO:teuthology.orchestra.run.smithi141.stdout:(114/131): libaio-devel-0.3.112-1.el8.x86_64.rp 124 kB/s | 19 kB 00:00 2023-11-16T16:36:12.634 INFO:teuthology.orchestra.run.smithi155.stdout:(33/131): ceph-prometheus-alerts-18.0.0-3905.g5 291 kB/s | 15 kB 00:00 2023-11-16T16:36:12.643 INFO:teuthology.orchestra.run.smithi002.stdout:(18/131): ceph-radosgw-18.0.0-3905.g5e4a743a.el 3.1 MB/s | 13 MB 00:04 2023-11-16T16:36:12.655 INFO:teuthology.orchestra.run.smithi141.stdout:(115/131): python3-cryptography-3.2.1-5.el8.x86 5.0 MB/s | 559 kB 00:00 2023-11-16T16:36:12.694 INFO:teuthology.orchestra.run.smithi002.stdout:(19/131): python3-ceph-argparse-18.0.0-3905.g5e 232 kB/s | 45 kB 00:00 2023-11-16T16:36:12.709 INFO:teuthology.orchestra.run.smithi155.stdout:(34/131): ceph-volume-18.0.0-3905.g5e4a743a.el8 3.4 MB/s | 265 kB 00:00 2023-11-16T16:36:12.727 INFO:teuthology.orchestra.run.smithi002.stdout:(20/131): python3-ceph-common-18.0.0-3905.g5e4a 1.4 MB/s | 119 kB 00:00 2023-11-16T16:36:12.730 INFO:teuthology.orchestra.run.smithi141.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 128 kB/s | 18 kB 00:00 2023-11-16T16:36:12.752 INFO:teuthology.orchestra.run.smithi002.stdout:(21/131): python3-cephfs-18.0.0-3905.g5e4a743a. 3.4 MB/s | 197 kB 00:00 2023-11-16T16:36:12.760 INFO:teuthology.orchestra.run.smithi155.stdout:(35/131): cephadm-18.0.0-3905.g5e4a743a.el8.noa 1.6 MB/s | 83 kB 00:00 2023-11-16T16:36:12.811 INFO:teuthology.orchestra.run.smithi002.stdout:(22/131): python3-rbd-18.0.0-3905.g5e4a743a.el8 6.1 MB/s | 365 kB 00:00 2023-11-16T16:36:12.814 INFO:teuthology.orchestra.run.smithi141.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.5 MB/s | 238 kB 00:00 2023-11-16T16:36:12.818 INFO:teuthology.orchestra.run.smithi155.stdout:(36/131): cephfs-top-18.0.0-3905.g5e4a743a.el8. 309 kB/s | 18 kB 00:00 2023-11-16T16:36:12.836 INFO:teuthology.orchestra.run.smithi002.stdout:(23/131): python3-rados-18.0.0-3905.g5e4a743a.e 3.3 MB/s | 371 kB 00:00 2023-11-16T16:36:12.847 INFO:teuthology.orchestra.run.smithi141.stdout:(118/131): pcre2-devel-10.32-2.el8.x86_64.rpm 2.6 MB/s | 605 kB 00:00 2023-11-16T16:36:12.861 INFO:teuthology.orchestra.run.smithi002.stdout:(24/131): python3-rgw-18.0.0-3905.g5e4a743a.el8 2.1 MB/s | 107 kB 00:00 2023-11-16T16:36:12.887 INFO:teuthology.orchestra.run.smithi002.stdout:(25/131): rbd-fuse-18.0.0-3905.g5e4a743a.el8.x8 1.8 MB/s | 94 kB 00:00 2023-11-16T16:36:12.937 INFO:teuthology.orchestra.run.smithi002.stdout:(26/131): rbd-nbd-18.0.0-3905.g5e4a743a.el8.x86 3.5 MB/s | 180 kB 00:00 2023-11-16T16:36:12.939 INFO:teuthology.orchestra.run.smithi141.stdout:(119/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 11 MB/s | 2.3 MB 00:00 2023-11-16T16:36:12.971 INFO:teuthology.orchestra.run.smithi002.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3905.g 694 kB/s | 23 kB 00:00 2023-11-16T16:36:12.981 INFO:teuthology.orchestra.run.smithi141.stdout:(120/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 433 kB/s | 72 kB 00:00 2023-11-16T16:36:13.006 INFO:teuthology.orchestra.run.smithi141.stdout:(121/131): fuse-2.9.7-15.el8.x86_64.rpm 521 kB/s | 83 kB 00:00 2023-11-16T16:36:13.021 INFO:teuthology.orchestra.run.smithi002.stdout:(28/131): ceph-mgr-cephadm-18.0.0-3905.g5e4a743 2.5 MB/s | 127 kB 00:00 2023-11-16T16:36:13.048 INFO:teuthology.orchestra.run.smithi141.stdout:(122/131): elfutils-libelf-devel-0.186-1.el8.x8 550 kB/s | 60 kB 00:00 2023-11-16T16:36:13.107 INFO:teuthology.orchestra.run.smithi141.stdout:(123/131): libuuid-devel-2.32.1-35.el8.x86_64.r 782 kB/s | 98 kB 00:00 2023-11-16T16:36:13.149 INFO:teuthology.orchestra.run.smithi141.stdout:(124/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 406 kB/s | 58 kB 00:00 2023-11-16T16:36:13.180 INFO:teuthology.orchestra.run.smithi002.stdout:(29/131): ceph-mgr-dashboard-18.0.0-3905.g5e4a7 11 MB/s | 1.7 MB 00:00 2023-11-16T16:36:13.216 INFO:teuthology.orchestra.run.smithi141.stdout:(125/131): libcom_err-devel-1.45.6-4.el8.x86_64 231 kB/s | 39 kB 00:00 2023-11-16T16:36:13.274 INFO:teuthology.orchestra.run.smithi141.stdout:(126/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 3.8 MB/s | 644 kB 00:00 2023-11-16T16:36:13.277 INFO:teuthology.orchestra.run.smithi155.stdout:(37/131): ceph-mgr-diskprediction-local-18.0.0- 5.7 MB/s | 7.4 MB 00:01 2023-11-16T16:36:13.300 INFO:teuthology.orchestra.run.smithi141.stdout:(127/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.1 MB/s | 171 kB 00:00 2023-11-16T16:36:13.358 INFO:teuthology.orchestra.run.smithi141.stdout:(128/131): NetworkManager-initscripts-updown-1. 968 kB/s | 138 kB 00:00 2023-11-16T16:36:13.444 INFO:teuthology.orchestra.run.smithi155.stdout:(38/131): gperftools-libs-2.7-9.el8.x86_64.rpm 490 kB/s | 306 kB 00:00 2023-11-16T16:36:13.509 INFO:teuthology.orchestra.run.smithi141.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 5.2 MB/s | 798 kB 00:00 2023-11-16T16:36:13.569 INFO:teuthology.orchestra.run.smithi155.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 608 kB/s | 75 kB 00:00 2023-11-16T16:36:13.680 INFO:teuthology.orchestra.run.smithi002.stdout:(30/131): ceph-mgr-diskprediction-local-18.0.0- 15 MB/s | 7.4 MB 00:00 2023-11-16T16:36:13.711 INFO:teuthology.orchestra.run.smithi155.stdout:(40/131): liboath-2.6.2-3.el8.x86_64.rpm 135 kB/s | 59 kB 00:00 2023-11-16T16:36:13.738 INFO:teuthology.orchestra.run.smithi002.stdout:(31/131): rbd-mirror-18.0.0-3905.g5e4a743a.el8. 3.9 MB/s | 3.4 MB 00:00 2023-11-16T16:36:13.761 INFO:teuthology.orchestra.run.smithi155.stdout:(41/131): python3-asyncssh-2.7.0-2.el8.noarch.r 2.4 MB/s | 476 kB 00:00 2023-11-16T16:36:13.763 INFO:teuthology.orchestra.run.smithi002.stdout:(32/131): ceph-mgr-modules-core-18.0.0-3905.g5e 2.8 MB/s | 240 kB 00:00 2023-11-16T16:36:13.789 INFO:teuthology.orchestra.run.smithi002.stdout:(33/131): ceph-mgr-rook-18.0.0-3905.g5e4a743a.e 944 kB/s | 47 kB 00:00 2023-11-16T16:36:13.814 INFO:teuthology.orchestra.run.smithi002.stdout:(34/131): ceph-prometheus-alerts-18.0.0-3905.g5 288 kB/s | 15 kB 00:00 2023-11-16T16:36:13.837 INFO:teuthology.orchestra.run.smithi155.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 348 kB/s | 44 kB 00:00 2023-11-16T16:36:13.873 INFO:teuthology.orchestra.run.smithi002.stdout:(35/131): cephadm-18.0.0-3905.g5e4a743a.el8.noa 1.4 MB/s | 83 kB 00:00 2023-11-16T16:36:13.879 INFO:teuthology.orchestra.run.smithi155.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.5 MB/s | 185 kB 00:00 2023-11-16T16:36:13.898 INFO:teuthology.orchestra.run.smithi002.stdout:(36/131): ceph-volume-18.0.0-3905.g5e4a743a.el8 2.4 MB/s | 265 kB 00:00 2023-11-16T16:36:13.924 INFO:teuthology.orchestra.run.smithi002.stdout:(37/131): cephfs-top-18.0.0-3905.g5e4a743a.el8. 356 kB/s | 18 kB 00:00 2023-11-16T16:36:13.962 INFO:teuthology.orchestra.run.smithi155.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 263 kB/s | 33 kB 00:00 2023-11-16T16:36:13.979 INFO:teuthology.orchestra.run.smithi155.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 163 kB/s | 16 kB 00:00 2023-11-16T16:36:14.059 INFO:teuthology.orchestra.run.smithi141.stdout:(130/131): librados2-18.0.0-3905.g5e4a743a.el8. 4.7 MB/s | 3.7 MB 00:00 2023-11-16T16:36:14.091 INFO:teuthology.orchestra.run.smithi002.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 353 kB/s | 59 kB 00:00 2023-11-16T16:36:14.113 INFO:teuthology.orchestra.run.smithi155.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 2.8 MB/s | 384 kB 00:00 2023-11-16T16:36:14.125 INFO:teuthology.orchestra.run.smithi141.stdout:(131/131): librbd1-18.0.0-3905.g5e4a743a.el8.x8 4.3 MB/s | 3.6 MB 00:00 2023-11-16T16:36:14.129 INFO:teuthology.orchestra.run.smithi141.stdout:-------------------------------------------------------------------------------- 2023-11-16T16:36:14.129 INFO:teuthology.orchestra.run.smithi141.stdout:Total 13 MB/s | 197 MB 00:15 2023-11-16T16:36:14.166 INFO:teuthology.orchestra.run.smithi002.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2023-11-16T16:36:14.188 INFO:teuthology.orchestra.run.smithi155.stdout:(47/131): python3-cheroot-8.5.2-1.el8.noarch.rp 766 kB/s | 173 kB 00:00 2023-11-16T16:36:14.191 INFO:teuthology.orchestra.run.smithi002.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 751 kB/s | 75 kB 00:00 2023-11-16T16:36:14.222 INFO:teuthology.orchestra.run.smithi155.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 749 kB/s | 81 kB 00:00 2023-11-16T16:36:14.258 INFO:teuthology.orchestra.run.smithi002.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 654 kB/s | 44 kB 00:00 2023-11-16T16:36:14.280 INFO:teuthology.orchestra.run.smithi155.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2023-11-16T16:36:14.283 INFO:teuthology.orchestra.run.smithi002.stdout:(42/131): python3-asyncssh-2.7.0-2.el8.noarch.r 4.0 MB/s | 476 kB 00:00 2023-11-16T16:36:14.314 INFO:teuthology.orchestra.run.smithi155.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 191 kB/s | 18 kB 00:00 2023-11-16T16:36:14.342 INFO:teuthology.orchestra.run.smithi002.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 567 kB/s | 33 kB 00:00 2023-11-16T16:36:14.368 INFO:teuthology.orchestra.run.smithi002.stdout:(44/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.7 MB/s | 185 kB 00:00 2023-11-16T16:36:14.401 INFO:teuthology.orchestra.run.smithi002.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 278 kB/s | 16 kB 00:00 2023-11-16T16:36:14.414 INFO:teuthology.orchestra.run.smithi155.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 488 kB/s | 49 kB 00:00 2023-11-16T16:36:14.443 INFO:teuthology.orchestra.run.smithi002.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 2.2 MB/s | 173 kB 00:00 2023-11-16T16:36:14.477 INFO:teuthology.orchestra.run.smithi002.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 5.0 MB/s | 384 kB 00:00 2023-11-16T16:36:14.514 INFO:teuthology.orchestra.run.smithi155.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 588 kB/s | 59 kB 00:00 2023-11-16T16:36:14.535 INFO:teuthology.orchestra.run.smithi002.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 883 kB/s | 81 kB 00:00 2023-11-16T16:36:14.560 INFO:teuthology.orchestra.run.smithi002.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 133 kB/s | 11 kB 00:00 2023-11-16T16:36:14.594 INFO:teuthology.orchestra.run.smithi002.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 301 kB/s | 18 kB 00:00 2023-11-16T16:36:14.623 INFO:teuthology.orchestra.run.smithi155.stdout:(53/131): python3-kubernetes-11.0.0-6.el8.noarc 3.4 MB/s | 1.2 MB 00:00 2023-11-16T16:36:14.640 INFO:teuthology.orchestra.run.smithi155.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 482 kB/s | 60 kB 00:00 2023-11-16T16:36:14.653 INFO:teuthology.orchestra.run.smithi002.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 840 kB/s | 49 kB 00:00 2023-11-16T16:36:14.711 INFO:teuthology.orchestra.run.smithi002.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2023-11-16T16:36:14.740 INFO:teuthology.orchestra.run.smithi155.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 155 kB/s | 16 kB 00:00 2023-11-16T16:36:14.745 INFO:teuthology.orchestra.run.smithi002.stdout:(53/131): python3-kubernetes-11.0.0-6.el8.noarc 6.3 MB/s | 1.2 MB 00:00 2023-11-16T16:36:14.766 INFO:teuthology.orchestra.run.smithi155.stdout:(56/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.9 MB/s | 283 kB 00:00 2023-11-16T16:36:14.781 INFO:teuthology.orchestra.run.smithi002.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 898 kB/s | 60 kB 00:00 2023-11-16T16:36:14.812 INFO:teuthology.orchestra.run.smithi002.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 4.1 MB/s | 283 kB 00:00 2023-11-16T16:36:14.832 INFO:teuthology.orchestra.run.smithi155.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 366 kB/s | 34 kB 00:00 2023-11-16T16:36:14.837 INFO:teuthology.orchestra.run.smithi002.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 268 kB/s | 16 kB 00:00 2023-11-16T16:36:14.871 INFO:teuthology.orchestra.run.smithi002.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 576 kB/s | 34 kB 00:00 2023-11-16T16:36:14.883 INFO:teuthology.orchestra.run.smithi155.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 1.6 MB/s | 196 kB 00:00 2023-11-16T16:36:14.896 INFO:teuthology.orchestra.run.smithi002.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 3.3 MB/s | 196 kB 00:00 2023-11-16T16:36:14.922 INFO:teuthology.orchestra.run.smithi002.stdout:(59/131): python3-rsa-4.9-2.el8.noarch.rpm 1.3 MB/s | 64 kB 00:00 2023-11-16T16:36:14.963 INFO:teuthology.orchestra.run.smithi141.stdout:Running transaction check 2023-11-16T16:36:14.978 INFO:teuthology.orchestra.run.smithi002.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 231 kB/s | 19 kB 00:00 2023-11-16T16:36:14.983 INFO:teuthology.orchestra.run.smithi155.stdout:(59/131): python3-rsa-4.9-2.el8.noarch.rpm 427 kB/s | 64 kB 00:00 2023-11-16T16:36:15.000 INFO:teuthology.orchestra.run.smithi155.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 160 kB/s | 19 kB 00:00 2023-11-16T16:36:15.006 INFO:teuthology.orchestra.run.smithi002.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 285 kB/s | 24 kB 00:00 2023-11-16T16:36:15.031 INFO:teuthology.orchestra.run.smithi002.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 553 kB/s | 29 kB 00:00 2023-11-16T16:36:15.065 INFO:teuthology.orchestra.run.smithi002.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 470 kB/s | 27 kB 00:00 2023-11-16T16:36:15.083 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction check succeeded. 2023-11-16T16:36:15.083 INFO:teuthology.orchestra.run.smithi141.stdout:Running transaction test 2023-11-16T16:36:15.098 INFO:teuthology.orchestra.run.smithi002.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 3.7 MB/s | 255 kB 00:00 2023-11-16T16:36:15.140 INFO:teuthology.orchestra.run.smithi002.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 3.3 MB/s | 251 kB 00:00 2023-11-16T16:36:15.157 INFO:teuthology.orchestra.run.smithi002.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 1.0 MB/s | 61 kB 00:00 2023-11-16T16:36:15.199 INFO:teuthology.orchestra.run.smithi002.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.4 MB/s | 85 kB 00:00 2023-11-16T16:36:15.225 INFO:teuthology.orchestra.run.smithi002.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 343 kB/s | 23 kB 00:00 2023-11-16T16:36:15.250 INFO:teuthology.orchestra.run.smithi002.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-11-16T16:36:15.342 INFO:teuthology.orchestra.run.smithi002.stdout:(70/131): thrift-0.13.0-2.el8.x86_64.rpm 12 MB/s | 1.7 MB 00:00 2023-11-16T16:36:15.384 INFO:teuthology.orchestra.run.smithi155.stdout:(61/131): ceph-test-18.0.0-3905.g5e4a743a.el8.x 4.1 MB/s | 41 MB 00:09 2023-11-16T16:36:15.401 INFO:teuthology.orchestra.run.smithi155.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 57 kB/s | 24 kB 00:00 2023-11-16T16:36:15.417 INFO:teuthology.orchestra.run.smithi155.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 70 kB/s | 29 kB 00:00 2023-11-16T16:36:15.467 INFO:teuthology.orchestra.run.smithi002.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 884 kB/s | 110 kB 00:00 2023-11-16T16:36:15.534 INFO:teuthology.orchestra.run.smithi155.stdout:(64/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.1 MB/s | 251 kB 00:00 2023-11-16T16:36:15.559 INFO:teuthology.orchestra.run.smithi155.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 1.8 MB/s | 255 kB 00:00 2023-11-16T16:36:15.635 INFO:teuthology.orchestra.run.smithi155.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 612 kB/s | 61 kB 00:00 2023-11-16T16:36:15.659 INFO:teuthology.orchestra.run.smithi002.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 283 kB/s | 54 kB 00:00 2023-11-16T16:36:15.668 INFO:teuthology.orchestra.run.smithi155.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 796 kB/s | 85 kB 00:00 2023-11-16T16:36:15.735 INFO:teuthology.orchestra.run.smithi155.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 229 kB/s | 23 kB 00:00 2023-11-16T16:36:15.760 INFO:teuthology.orchestra.run.smithi155.stdout:(69/131): python3-trustme-0.6.0-4.el8.noarch.rp 73 kB/s | 27 kB 00:00 2023-11-16T16:36:15.786 INFO:teuthology.orchestra.run.smithi155.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2023-11-16T16:36:15.893 INFO:teuthology.orchestra.run.smithi002.stdout:(73/131): python3-scipy-1.0.0-21.module+el8.5.0 21 MB/s | 14 MB 00:00 2023-11-16T16:36:15.920 INFO:teuthology.orchestra.run.smithi155.stdout:(71/131): thrift-0.13.0-2.el8.x86_64.rpm 6.9 MB/s | 1.7 MB 00:00 2023-11-16T16:36:15.926 INFO:teuthology.orchestra.run.smithi002.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 971 kB/s | 259 kB 00:00 2023-11-16T16:36:15.962 INFO:teuthology.orchestra.run.smithi155.stdout:(72/131): python3-pyasn1-modules-0.3.7-6.el8.no 650 kB/s | 110 kB 00:00 2023-11-16T16:36:16.044 INFO:teuthology.orchestra.run.smithi002.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 2.1 MB/s | 323 kB 00:00 2023-11-16T16:36:16.143 INFO:teuthology.orchestra.run.smithi002.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 179 kB/s | 39 kB 00:00 2023-11-16T16:36:16.169 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction test succeeded. 2023-11-16T16:36:16.173 INFO:teuthology.orchestra.run.smithi141.stdout:Running transaction 2023-11-16T16:36:16.194 INFO:teuthology.orchestra.run.smithi002.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 4.5 MB/s | 688 kB 00:00 2023-11-16T16:36:16.262 INFO:teuthology.orchestra.run.smithi155.stdout:(73/131): python3-scipy-1.0.0-21.module+el8.5.0 27 MB/s | 14 MB 00:00 2023-11-16T16:36:16.287 INFO:teuthology.orchestra.run.smithi155.stdout:(74/131): python3-pytz-2017.2-9.el8.noarch.rpm 148 kB/s | 54 kB 00:00 2023-11-16T16:36:16.312 INFO:teuthology.orchestra.run.smithi155.stdout:(75/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 751 kB/s | 259 kB 00:00 2023-11-16T16:36:16.379 INFO:teuthology.orchestra.run.smithi155.stdout:(76/131): socat-1.7.4.1-1.el8.x86_64.rpm 2.7 MB/s | 323 kB 00:00 2023-11-16T16:36:16.402 INFO:teuthology.orchestra.run.smithi002.stdout:(78/131): ceph-osd-18.0.0-3905.g5e4a743a.el8.x8 2.2 MB/s | 18 MB 00:08 2023-11-16T16:36:16.421 INFO:teuthology.orchestra.run.smithi155.stdout:(77/131): python3-markupsafe-0.23-19.el8.x86_64 290 kB/s | 39 kB 00:00 2023-11-16T16:36:16.428 INFO:teuthology.orchestra.run.smithi002.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 672 kB/s | 157 kB 00:00 2023-11-16T16:36:16.463 INFO:teuthology.orchestra.run.smithi155.stdout:(78/131): bison-3.0.4-10.el8.x86_64.rpm 4.5 MB/s | 688 kB 00:00 2023-11-16T16:36:16.470 INFO:teuthology.orchestra.run.smithi002.stdout:(80/131): python3-jinja2-2.10.1-3.el8.noarch.rp 1.6 MB/s | 538 kB 00:00 2023-11-16T16:36:16.513 INFO:teuthology.orchestra.run.smithi155.stdout:(79/131): python3-jinja2-2.10.1-3.el8.noarch.rp 3.9 MB/s | 538 kB 00:00 2023-11-16T16:36:16.547 INFO:teuthology.orchestra.run.smithi155.stdout:(80/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.2 MB/s | 157 kB 00:00 2023-11-16T16:36:16.570 INFO:teuthology.orchestra.run.smithi002.stdout:(81/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2023-11-16T16:36:16.629 INFO:teuthology.orchestra.run.smithi002.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 3.6 MB/s | 713 kB 00:00 2023-11-16T16:36:16.672 INFO:teuthology.orchestra.run.smithi155.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 4.4 MB/s | 713 kB 00:00 2023-11-16T16:36:16.789 INFO:teuthology.orchestra.run.smithi155.stdout:(82/131): oniguruma-6.8.2-2.el8.x86_64.rpm 575 kB/s | 187 kB 00:00 2023-11-16T16:36:16.814 INFO:teuthology.orchestra.run.smithi155.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 187 kB/s | 27 kB 00:00 2023-11-16T16:36:16.862 INFO:teuthology.orchestra.run.smithi002.stdout:(83/131): python3-babel-2.5.1-7.el8.noarch.rpm 12 MB/s | 4.8 MB 00:00 2023-11-16T16:36:16.888 INFO:teuthology.orchestra.run.smithi002.stdout:(84/131): python3-jsonpatch-1.21-2.el8.noarch.r 84 kB/s | 27 kB 00:00 2023-11-16T16:36:16.914 INFO:teuthology.orchestra.run.smithi155.stdout:(84/131): flex-2.6.1-9.el8.x86_64.rpm 2.5 MB/s | 320 kB 00:00 2023-11-16T16:36:16.921 INFO:teuthology.orchestra.run.smithi002.stdout:(85/131): flex-2.6.1-9.el8.x86_64.rpm 1.1 MB/s | 320 kB 00:00 2023-11-16T16:36:16.996 INFO:teuthology.orchestra.run.smithi002.stdout:(86/131): python3-prettytable-0.7.2-14.el8.noar 329 kB/s | 44 kB 00:00 2023-11-16T16:36:16.998 INFO:teuthology.orchestra.run.smithi155.stdout:(85/131): python3-prettytable-0.7.2-14.el8.noar 240 kB/s | 44 kB 00:00 2023-11-16T16:36:17.065 INFO:teuthology.orchestra.run.smithi155.stdout:(86/131): python3-babel-2.5.1-7.el8.noarch.rpm 9.2 MB/s | 4.8 MB 00:00 2023-11-16T16:36:17.105 INFO:teuthology.orchestra.run.smithi002.stdout:(87/131): python3-numpy-1.14.3-10.el8.x86_64.rp 17 MB/s | 3.7 MB 00:00 2023-11-16T16:36:17.139 INFO:teuthology.orchestra.run.smithi002.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.5 MB/s | 225 kB 00:00 2023-11-16T16:36:17.157 INFO:teuthology.orchestra.run.smithi155.stdout:(87/131): python3-numpy-1.14.3-10.el8.x86_64.rp 15 MB/s | 3.7 MB 00:00 2023-11-16T16:36:17.181 INFO:teuthology.orchestra.run.smithi002.stdout:(89/131): python3-werkzeug-0.12.2-4.el8.noarch. 1.7 MB/s | 457 kB 00:00 2023-11-16T16:36:17.191 INFO:teuthology.orchestra.run.smithi155.stdout:(88/131): python3-werkzeug-0.12.2-4.el8.noarch. 2.3 MB/s | 457 kB 00:00 2023-11-16T16:36:17.231 INFO:teuthology.orchestra.run.smithi002.stdout:(90/131): uuid-1.6.2-43.el8.x86_64.rpm 509 kB/s | 64 kB 00:00 2023-11-16T16:36:17.241 INFO:teuthology.orchestra.run.smithi155.stdout:(89/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.3 MB/s | 225 kB 00:00 2023-11-16T16:36:17.265 INFO:teuthology.orchestra.run.smithi002.stdout:(91/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.0 MB/s | 126 kB 00:00 2023-11-16T16:36:17.308 INFO:teuthology.orchestra.run.smithi155.stdout:(90/131): uuid-1.6.2-43.el8.x86_64.rpm 426 kB/s | 64 kB 00:00 2023-11-16T16:36:17.340 INFO:teuthology.orchestra.run.smithi002.stdout:(92/131): python3-jsonpointer-1.10-11.el8.noarc 126 kB/s | 20 kB 00:00 2023-11-16T16:36:17.341 INFO:teuthology.orchestra.run.smithi155.stdout:(91/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 839 kB/s | 126 kB 00:00 2023-11-16T16:36:17.367 INFO:teuthology.orchestra.run.smithi155.stdout:(92/131): python3-jsonpointer-1.10-11.el8.noarc 159 kB/s | 20 kB 00:00 2023-11-16T16:36:17.373 INFO:teuthology.orchestra.run.smithi002.stdout:(93/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 729 kB/s | 103 kB 00:00 2023-11-16T16:36:17.384 INFO:teuthology.orchestra.run.smithi155.stdout:(93/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 1.3 MB/s | 103 kB 00:00 2023-11-16T16:36:17.417 INFO:teuthology.orchestra.run.smithi155.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 9.2 MB/s | 709 kB 00:00 2023-11-16T16:36:17.449 INFO:teuthology.orchestra.run.smithi002.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 3.8 MB/s | 709 kB 00:00 2023-11-16T16:36:17.483 INFO:teuthology.orchestra.run.smithi002.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 1.8 MB/s | 198 kB 00:00 2023-11-16T16:36:17.551 INFO:teuthology.orchestra.run.smithi155.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 1.2 MB/s | 198 kB 00:00 2023-11-16T16:36:17.625 INFO:teuthology.orchestra.run.smithi002.stdout:(96/131): openblas-threads-0.3.15-3.el8.x86_64. 17 MB/s | 4.9 MB 00:00 2023-11-16T16:36:17.658 INFO:teuthology.orchestra.run.smithi002.stdout:(97/131): jq-1.6-3.el8.x86_64.rpm 1.1 MB/s | 202 kB 00:00 2023-11-16T16:36:17.682 INFO:teuthology.orchestra.run.smithi155.stdout:(96/131): openblas-threads-0.3.15-3.el8.x86_64. 15 MB/s | 4.9 MB 00:00 2023-11-16T16:36:17.735 INFO:teuthology.orchestra.run.smithi155.stdout:(97/131): jq-1.6-3.el8.x86_64.rpm 1.1 MB/s | 202 kB 00:00 2023-11-16T16:36:17.800 INFO:teuthology.orchestra.run.smithi002.stdout:(98/131): openblas-0.3.15-3.el8.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2023-11-16T16:36:17.827 INFO:teuthology.orchestra.run.smithi155.stdout:(98/131): openblas-0.3.15-3.el8.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2023-11-16T16:36:17.852 INFO:teuthology.orchestra.run.smithi155.stdout:(99/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.0 MB/s | 354 kB 00:00 2023-11-16T16:36:17.942 INFO:teuthology.orchestra.run.smithi002.stdout:(99/131): postgresql-contrib-10.21-2.module+el8 5.6 MB/s | 808 kB 00:00 2023-11-16T16:36:17.960 INFO:teuthology.orchestra.run.smithi155.stdout:(100/131): postgresql-server-10.21-2.module+el8 22 MB/s | 5.1 MB 00:00 2023-11-16T16:36:17.967 INFO:teuthology.orchestra.run.smithi002.stdout:(100/131): librdkafka-0.11.4-3.el8.x86_64.rpm 1.0 MB/s | 354 kB 00:00 2023-11-16T16:36:17.994 INFO:teuthology.orchestra.run.smithi155.stdout:(101/131): postgresql-contrib-10.21-2.module+el 4.7 MB/s | 808 kB 00:00 2023-11-16T16:36:18.045 INFO:teuthology.orchestra.run.smithi155.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 7.8 MB/s | 1.5 MB 00:00 2023-11-16T16:36:18.086 INFO:teuthology.orchestra.run.smithi155.stdout:(103/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 373 kB/s | 47 kB 00:00 2023-11-16T16:36:18.118 INFO:teuthology.orchestra.run.smithi002.stdout:(101/131): postgresql-server-10.21-2.module+el8 11 MB/s | 5.1 MB 00:00 2023-11-16T16:36:18.134 INFO:teuthology.orchestra.run.smithi002.stdout:(102/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 280 kB/s | 47 kB 00:00 2023-11-16T16:36:18.145 INFO:teuthology.orchestra.run.smithi155.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 425 kB/s | 43 kB 00:00 2023-11-16T16:36:18.170 INFO:teuthology.orchestra.run.smithi155.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.6 MB/s | 220 kB 00:00 2023-11-16T16:36:18.176 INFO:teuthology.orchestra.run.smithi002.stdout:(103/131): postgresql-10.21-2.module+el8.6.0+15 6.4 MB/s | 1.5 MB 00:00 2023-11-16T16:36:18.220 INFO:teuthology.orchestra.run.smithi155.stdout:(106/131): libselinux-devel-2.9-5.el8.x86_64.rp 884 kB/s | 200 kB 00:00 2023-11-16T16:36:18.279 INFO:teuthology.orchestra.run.smithi155.stdout:(107/131): keyutils-libs-devel-1.5.10-9.el8.x86 359 kB/s | 48 kB 00:00 2023-11-16T16:36:18.285 INFO:teuthology.orchestra.run.smithi002.stdout:(104/131): libselinux-devel-2.9-5.el8.x86_64.rp 1.2 MB/s | 200 kB 00:00 2023-11-16T16:36:18.310 INFO:teuthology.orchestra.run.smithi002.stdout:(105/131): python3-jwt-1.6.1-2.el8.noarch.rpm 243 kB/s | 43 kB 00:00 2023-11-16T16:36:18.313 INFO:teuthology.orchestra.run.smithi155.stdout:(108/131): libsepol-devel-2.9-3.el8.x86_64.rpm 611 kB/s | 87 kB 00:00 2023-11-16T16:36:18.346 INFO:teuthology.orchestra.run.smithi155.stdout:(109/131): python3-requests-oauthlib-1.0.0-1.el 340 kB/s | 43 kB 00:00 2023-11-16T16:36:18.369 INFO:teuthology.orchestra.run.smithi002.stdout:(106/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.1 MB/s | 220 kB 00:00 2023-11-16T16:36:18.405 INFO:teuthology.orchestra.run.smithi155.stdout:(110/131): python3-pycparser-2.14-14.el8.noarch 869 kB/s | 109 kB 00:00 2023-11-16T16:36:18.422 INFO:teuthology.orchestra.run.smithi155.stdout:(111/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.1 MB/s | 229 kB 00:00 2023-11-16T16:36:18.428 INFO:teuthology.orchestra.run.smithi002.stdout:(107/131): keyutils-libs-devel-1.5.10-9.el8.x86 338 kB/s | 48 kB 00:00 2023-11-16T16:36:18.453 INFO:teuthology.orchestra.run.smithi002.stdout:(108/131): libsepol-devel-2.9-3.el8.x86_64.rpm 612 kB/s | 87 kB 00:00 2023-11-16T16:36:18.478 INFO:teuthology.orchestra.run.smithi002.stdout:(109/131): python3-requests-oauthlib-1.0.0-1.el 392 kB/s | 43 kB 00:00 2023-11-16T16:36:18.514 INFO:teuthology.orchestra.run.smithi155.stdout:(112/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.3 MB/s | 560 kB 00:00 2023-11-16T16:36:18.570 INFO:teuthology.orchestra.run.smithi002.stdout:(110/131): python3-pycparser-2.14-14.el8.noarch 768 kB/s | 109 kB 00:00 2023-11-16T16:36:18.572 INFO:teuthology.orchestra.run.smithi155.stdout:(113/131): libaio-devel-0.3.112-1.el8.x86_64.rp 112 kB/s | 19 kB 00:00 2023-11-16T16:36:18.595 INFO:teuthology.orchestra.run.smithi002.stdout:(111/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.6 MB/s | 229 kB 00:00 2023-11-16T16:36:18.614 INFO:teuthology.orchestra.run.smithi155.stdout:(114/131): python3-oauthlib-2.1.0-1.el8.noarch. 807 kB/s | 155 kB 00:00 2023-11-16T16:36:18.656 INFO:teuthology.orchestra.run.smithi155.stdout:(115/131): python3-cryptography-3.2.1-5.el8.x86 3.8 MB/s | 559 kB 00:00 2023-11-16T16:36:18.665 INFO:teuthology.orchestra.run.smithi002.stdout:(112/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 2.9 MB/s | 560 kB 00:00 2023-11-16T16:36:18.696 INFO:teuthology.orchestra.run.smithi002.stdout:(113/131): libaio-devel-0.3.112-1.el8.x86_64.rp 149 kB/s | 19 kB 00:00 2023-11-16T16:36:18.706 INFO:teuthology.orchestra.run.smithi155.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 133 kB/s | 18 kB 00:00 2023-11-16T16:36:18.746 INFO:teuthology.orchestra.run.smithi002.stdout:(114/131): python3-oauthlib-2.1.0-1.el8.noarch. 1.0 MB/s | 155 kB 00:00 2023-11-16T16:36:18.767 INFO:teuthology.orchestra.run.smithi155.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.9 MB/s | 605 kB 00:00 2023-11-16T16:36:18.792 INFO:teuthology.orchestra.run.smithi155.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.7 MB/s | 238 kB 00:00 2023-11-16T16:36:18.822 INFO:teuthology.orchestra.run.smithi002.stdout:(115/131): python3-cryptography-3.2.1-5.el8.x86 3.5 MB/s | 559 kB 00:00 2023-11-16T16:36:18.847 INFO:teuthology.orchestra.run.smithi002.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 119 kB/s | 18 kB 00:00 2023-11-16T16:36:18.859 INFO:teuthology.orchestra.run.smithi155.stdout:(119/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 15 MB/s | 2.3 MB 00:00 2023-11-16T16:36:18.880 INFO:teuthology.orchestra.run.smithi002.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 4.4 MB/s | 605 kB 00:00 2023-11-16T16:36:18.885 INFO:teuthology.orchestra.run.smithi155.stdout:(120/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 616 kB/s | 72 kB 00:00 2023-11-16T16:36:18.952 INFO:teuthology.orchestra.run.smithi155.stdout:(121/131): fuse-2.9.7-15.el8.x86_64.rpm 521 kB/s | 83 kB 00:00 2023-11-16T16:36:18.973 INFO:teuthology.orchestra.run.smithi002.stdout:(118/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 787 kB/s | 72 kB 00:00 2023-11-16T16:36:19.006 INFO:teuthology.orchestra.run.smithi002.stdout:(119/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.3 MB/s | 238 kB 00:00 2023-11-16T16:36:19.019 INFO:teuthology.orchestra.run.smithi155.stdout:(122/131): elfutils-libelf-devel-0.186-1.el8.x8 377 kB/s | 60 kB 00:00 2023-11-16T16:36:19.044 INFO:teuthology.orchestra.run.smithi155.stdout:(123/131): libuuid-devel-2.32.1-35.el8.x86_64.r 618 kB/s | 98 kB 00:00 2023-11-16T16:36:19.069 INFO:teuthology.orchestra.run.smithi155.stdout:(124/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 493 kB/s | 58 kB 00:00 2023-11-16T16:36:19.081 INFO:teuthology.orchestra.run.smithi002.stdout:(120/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 9.9 MB/s | 2.3 MB 00:00 2023-11-16T16:36:19.135 INFO:teuthology.orchestra.run.smithi002.stdout:(121/131): fuse-2.9.7-15.el8.x86_64.rpm 511 kB/s | 83 kB 00:00 2023-11-16T16:36:19.144 INFO:teuthology.orchestra.run.smithi155.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 6.3 MB/s | 644 kB 00:00 2023-11-16T16:36:19.170 INFO:teuthology.orchestra.run.smithi155.stdout:(126/131): libcom_err-devel-1.45.6-4.el8.x86_64 256 kB/s | 39 kB 00:00 2023-11-16T16:36:19.174 INFO:teuthology.orchestra.run.smithi002.stdout:(122/131): elfutils-libelf-devel-0.186-1.el8.x8 358 kB/s | 60 kB 00:00 2023-11-16T16:36:19.199 INFO:teuthology.orchestra.run.smithi002.stdout:(123/131): libuuid-devel-2.32.1-35.el8.x86_64.r 837 kB/s | 98 kB 00:00 2023-11-16T16:36:19.228 INFO:teuthology.orchestra.run.smithi155.stdout:(127/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.1 MB/s | 171 kB 00:00 2023-11-16T16:36:19.262 INFO:teuthology.orchestra.run.smithi155.stdout:(128/131): NetworkManager-initscripts-updown-1. 1.1 MB/s | 138 kB 00:00 2023-11-16T16:36:19.291 INFO:teuthology.orchestra.run.smithi002.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 331 kB/s | 39 kB 00:00 2023-11-16T16:36:19.316 INFO:teuthology.orchestra.run.smithi002.stdout:(125/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 319 kB/s | 58 kB 00:00 2023-11-16T16:36:19.367 INFO:teuthology.orchestra.run.smithi002.stdout:(126/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 3.8 MB/s | 644 kB 00:00 2023-11-16T16:36:19.408 INFO:teuthology.orchestra.run.smithi002.stdout:(127/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.4 MB/s | 171 kB 00:00 2023-11-16T16:36:19.429 INFO:teuthology.orchestra.run.smithi155.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 4.7 MB/s | 798 kB 00:00 2023-11-16T16:36:19.434 INFO:teuthology.orchestra.run.smithi002.stdout:(128/131): NetworkManager-initscripts-updown-1. 1.1 MB/s | 138 kB 00:00 2023-11-16T16:36:19.609 INFO:teuthology.orchestra.run.smithi002.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 4.5 MB/s | 798 kB 00:00 2023-11-16T16:36:19.671 INFO:teuthology.orchestra.run.smithi155.stdout:(130/131): librbd1-18.0.0-3905.g5e4a743a.el8.x8 8.1 MB/s | 3.6 MB 00:00 2023-11-16T16:36:19.729 INFO:teuthology.orchestra.run.smithi155.stdout:(131/131): librados2-18.0.0-3905.g5e4a743a.el8. 6.5 MB/s | 3.7 MB 00:00 2023-11-16T16:36:19.734 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2023-11-16T16:36:19.734 INFO:teuthology.orchestra.run.smithi155.stdout:Total 9.8 MB/s | 197 MB 00:20 2023-11-16T16:36:19.959 INFO:teuthology.orchestra.run.smithi002.stdout:(130/131): librbd1-18.0.0-3905.g5e4a743a.el8.x8 6.5 MB/s | 3.6 MB 00:00 2023-11-16T16:36:20.310 INFO:teuthology.orchestra.run.smithi002.stdout:(131/131): librados2-18.0.0-3905.g5e4a743a.el8. 3.9 MB/s | 3.7 MB 00:00 2023-11-16T16:36:20.313 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-11-16T16:36:20.313 INFO:teuthology.orchestra.run.smithi002.stdout:Total 12 MB/s | 197 MB 00:16 2023-11-16T16:36:20.580 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2023-11-16T16:36:20.702 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2023-11-16T16:36:20.702 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2023-11-16T16:36:20.958 INFO:teuthology.orchestra.run.smithi141.stdout: Preparing : 1/1 2023-11-16T16:36:21.177 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-11-16T16:36:21.347 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-11-16T16:36:21.560 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-11-16T16:36:21.720 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-16T16:36:21.786 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2023-11-16T16:36:21.790 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2023-11-16T16:36:21.827 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-16T16:36:22.075 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2023-11-16T16:36:22.120 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-16T16:36:22.223 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2023-11-16T16:36:22.223 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2023-11-16T16:36:22.521 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-16T16:36:22.737 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T16:36:23.133 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T16:36:23.328 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T16:36:23.459 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T16:36:23.644 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T16:36:23.748 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T16:36:23.798 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2023-11-16T16:36:23.804 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2023-11-16T16:36:23.929 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 9/134 2023-11-16T16:36:24.148 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-11-16T16:36:24.381 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-11-16T16:36:24.772 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-11-16T16:36:25.036 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-11-16T16:36:25.170 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 14/134 2023-11-16T16:36:25.278 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 14/134 2023-11-16T16:36:25.670 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 15/134 2023-11-16T16:36:25.773 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 15/134 2023-11-16T16:36:25.911 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-16T16:36:26.064 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-16T16:36:26.220 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-16T16:36:26.503 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2023-11-16T16:36:26.658 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-11-16T16:36:26.814 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-11-16T16:36:26.836 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-16T16:36:27.011 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-16T16:36:27.042 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-11-16T16:36:27.202 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-16T16:36:27.294 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-16T16:36:27.543 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-16T16:36:27.608 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-16T16:36:27.803 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-11-16T16:36:27.925 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-16T16:36:28.118 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T16:36:28.504 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-11-16T16:36:28.545 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T16:36:28.710 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T16:36:28.758 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-11-16T16:36:28.836 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T16:36:28.976 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T16:36:29.004 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-11-16T16:36:29.039 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2023-11-16T16:36:29.059 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T16:36:29.231 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-11-16T16:36:29.244 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 9/134 2023-11-16T16:36:29.249 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-11-16T16:36:29.408 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-11-16T16:36:29.432 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-11-16T16:36:29.496 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-11-16T16:36:29.608 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-11-16T16:36:29.679 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-11-16T16:36:29.687 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-11-16T16:36:29.842 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-16T16:36:29.862 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-11-16T16:36:29.995 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-16T16:36:30.050 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8. 26/134 2023-11-16T16:36:30.067 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-11-16T16:36:30.180 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el 27/134 2023-11-16T16:36:30.267 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-16T16:36:30.280 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-11-16T16:36:30.409 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 14/134 2023-11-16T16:36:30.555 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 14/134 2023-11-16T16:36:30.701 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 15/134 2023-11-16T16:36:30.751 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-16T16:36:30.778 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 15/134 2023-11-16T16:36:30.884 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-16T16:36:31.029 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-16T16:36:31.130 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_6 28/134 2023-11-16T16:36:31.158 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T16:36:31.176 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-16T16:36:31.598 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-11-16T16:36:31.632 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T16:36:31.776 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-16T16:36:31.808 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T16:36:31.836 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-11-16T16:36:31.951 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-16T16:36:31.962 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T16:36:32.134 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-11-16T16:36:32.174 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T16:36:32.296 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T16:36:32.310 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-16T16:36:32.517 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 9/134 2023-11-16T16:36:32.535 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-16T16:36:32.734 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-11-16T16:36:32.795 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-11-16T16:36:33.056 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-11-16T16:36:33.429 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-11-16T16:36:33.492 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-11-16T16:36:33.674 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-11-16T16:36:33.713 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-11-16T16:36:33.885 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-11-16T16:36:33.891 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 14/134 2023-11-16T16:36:34.020 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 14/134 2023-11-16T16:36:34.090 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-16T16:36:34.138 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-11-16T16:36:34.166 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 15/134 2023-11-16T16:36:34.267 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 15/134 2023-11-16T16:36:34.298 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-11-16T16:36:34.433 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-16T16:36:34.435 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-11-16T16:36:34.489 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-11-16T16:36:34.598 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-16T16:36:34.638 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-11-16T16:36:34.744 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-11-16T16:36:34.783 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-16T16:36:34.837 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-11-16T16:36:34.924 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8. 26/134 2023-11-16T16:36:35.053 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el 27/134 2023-11-16T16:36:35.074 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 36/134 2023-11-16T16:36:35.263 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_6 37/134 2023-11-16T16:36:35.425 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-11-16T16:36:35.513 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-16T16:36:35.586 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-11-16T16:36:35.749 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-16T16:36:35.795 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-11-16T16:36:35.946 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-11-16T16:36:35.972 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_6 28/134 2023-11-16T16:36:36.159 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-11-16T16:36:36.274 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-11-16T16:36:36.349 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-11-16T16:36:36.462 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-16T16:36:36.528 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-11-16T16:36:36.691 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-11-16T16:36:36.751 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-11-16T16:36:36.763 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-11-16T16:36:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-16T16:36:37.045 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-11-16T16:36:37.235 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-11-16T16:36:37.472 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-11-16T16:36:37.580 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-11-16T16:36:37.705 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-11-16T16:36:37.849 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-11-16T16:36:38.024 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-11-16T16:36:38.124 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-11-16T16:36:38.267 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-11-16T16:36:38.340 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-11-16T16:36:38.458 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-11-16T16:36:38.567 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-11-16T16:36:38.672 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-11-16T16:36:38.713 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-16T16:36:38.806 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-11-16T16:36:38.863 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-11-16T16:36:39.026 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-11-16T16:36:39.028 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-11-16T16:36:39.064 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-11-16T16:36:39.211 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-11-16T16:36:39.240 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8. 26/134 2023-11-16T16:36:39.325 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-11-16T16:36:39.368 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-11-16T16:36:39.412 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el 27/134 2023-11-16T16:36:39.571 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-11-16T16:36:39.572 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 36/134 2023-11-16T16:36:39.761 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_6 37/134 2023-11-16T16:36:39.838 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-11-16T16:36:39.915 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-11-16T16:36:40.057 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-11-16T16:36:40.071 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-11-16T16:36:40.224 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-11-16T16:36:40.269 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-11-16T16:36:40.403 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-11-16T16:36:40.410 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-11-16T16:36:40.489 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_6 28/134 2023-11-16T16:36:40.583 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-11-16T16:36:40.607 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-11-16T16:36:40.658 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-11-16T16:36:40.805 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-11-16T16:36:40.880 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-11-16T16:36:40.894 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-11-16T16:36:41.095 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-11-16T16:36:41.144 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-11-16T16:36:41.185 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-11-16T16:36:41.283 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-16T16:36:41.361 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-16T16:36:41.425 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-11-16T16:36:41.452 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-11-16T16:36:41.580 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-11-16T16:36:41.648 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-16T16:36:41.735 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-11-16T16:36:41.934 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-11-16T16:36:41.935 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-11-16T16:36:42.114 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-11-16T16:36:42.169 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-11-16T16:36:42.381 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-11-16T16:36:42.404 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-11-16T16:36:42.616 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-11-16T16:36:42.708 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-11-16T16:36:42.788 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-11-16T16:36:42.890 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-11-16T16:36:42.936 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-11-16T16:36:43.085 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-11-16T16:36:43.135 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-11-16T16:36:43.321 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-11-16T16:36:43.337 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-11-16T16:36:43.503 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-11-16T16:36:43.511 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-11-16T16:36:43.653 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-11-16T16:36:43.660 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-11-16T16:36:43.797 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-11-16T16:36:43.854 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-16T16:36:43.954 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-11-16T16:36:44.021 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-11-16T16:36:44.167 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-11-16T16:36:44.204 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-11-16T16:36:44.287 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-11-16T16:36:44.431 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-11-16T16:36:44.481 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-11-16T16:36:44.488 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-11-16T16:36:44.631 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-11-16T16:36:44.697 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-11-16T16:36:44.738 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-11-16T16:36:44.864 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-11-16T16:36:44.890 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 36/134 2023-11-16T16:36:44.990 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-11-16T16:36:45.017 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-11-16T16:36:45.095 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_6 37/134 2023-11-16T16:36:45.173 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-11-16T16:36:45.207 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-11-16T16:36:45.266 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-11-16T16:36:45.299 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-11-16T16:36:45.417 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-11-16T16:36:45.477 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-11-16T16:36:45.503 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-11-16T16:36:45.694 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-11-16T16:36:45.701 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-11-16T16:36:45.832 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-16T16:36:45.847 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-11-16T16:36:45.911 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-16T16:36:46.066 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-11-16T16:36:46.136 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-11-16T16:36:46.299 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-11-16T16:36:46.325 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-11-16T16:36:46.456 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-11-16T16:36:46.507 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-11-16T16:36:46.643 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-11-16T16:36:46.751 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-11-16T16:36:46.778 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-11-16T16:36:46.911 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-11-16T16:36:47.057 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-11-16T16:36:47.073 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-11-16T16:36:47.114 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-11-16T16:36:47.281 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-11-16T16:36:47.286 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-11-16T16:36:47.398 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-11-16T16:36:47.427 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-11-16T16:36:47.541 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-11-16T16:36:47.618 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-11-16T16:36:47.668 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-11-16T16:36:47.818 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-11-16T16:36:47.854 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-11-16T16:36:47.857 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-11-16T16:36:48.027 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-11-16T16:36:48.193 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-11-16T16:36:48.199 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-11-16T16:36:48.329 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-11-16T16:36:48.449 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-11-16T16:36:48.546 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-11-16T16:36:48.618 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-11-16T16:36:48.650 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-11-16T16:36:48.752 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-11-16T16:36:48.840 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 92/134 2023-11-16T16:36:48.844 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-11-16T16:36:48.913 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 92/134 2023-11-16T16:36:48.938 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-11-16T16:36:49.040 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-11-16T16:36:49.090 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 93/134 2023-11-16T16:36:49.188 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-11-16T16:36:49.204 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-11-16T16:36:49.414 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-11-16T16:36:49.580 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-11-16T16:36:49.607 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-11-16T16:36:49.797 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-11-16T16:36:49.954 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-11-16T16:36:50.216 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-11-16T16:36:50.565 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-11-16T16:36:50.783 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-11-16T16:36:50.880 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-11-16T16:36:50.962 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-11-16T16:36:51.135 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-11-16T16:36:51.189 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-11-16T16:36:51.291 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-11-16T16:36:51.389 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-11-16T16:36:51.601 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-11-16T16:36:51.691 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-11-16T16:36:51.821 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-11-16T16:36:51.950 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-11-16T16:36:51.980 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-16T16:36:52.102 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-16T16:36:52.128 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-11-16T16:36:52.339 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-11-16T16:36:52.501 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-11-16T16:36:52.709 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-11-16T16:36:52.726 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T16:36:52.900 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-11-16T16:36:52.902 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-11-16T16:36:52.932 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T16:36:53.064 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 92/134 2023-11-16T16:36:53.137 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 92/134 2023-11-16T16:36:53.228 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-11-16T16:36:53.298 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 93/134 2023-11-16T16:36:53.473 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-11-16T16:36:53.517 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T16:36:53.628 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-11-16T16:36:53.785 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 95/134 2023-11-16T16:36:53.788 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-11-16T16:36:53.863 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 95/134 2023-11-16T16:36:53.998 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 96/134 2023-11-16T16:36:54.026 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-11-16T16:36:54.217 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-11-16T16:36:54.440 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-11-16T16:36:54.618 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-11-16T16:36:54.772 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-11-16T16:36:55.045 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-11-16T16:36:55.305 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-11-16T16:36:55.505 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-11-16T16:36:55.788 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-11-16T16:36:56.015 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-11-16T16:36:56.226 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-11-16T16:36:56.460 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-11-16T16:36:56.942 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T16:36:57.190 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T16:36:57.778 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T16:36:57.912 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-11-16T16:36:58.014 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 95/134 2023-11-16T16:36:58.096 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 95/134 2023-11-16T16:36:58.245 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-11-16T16:36:58.272 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 96/134 2023-11-16T16:36:58.529 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-11-16T16:36:58.889 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-11-16T16:36:59.216 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-11-16T16:36:59.585 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-11-16T16:37:00.433 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-11-16T16:37:00.595 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 92/134 2023-11-16T16:37:00.694 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 92/134 2023-11-16T16:37:00.871 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 93/134 2023-11-16T16:37:06.243 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T16:37:06.479 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T16:37:07.824 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 94/134 2023-11-16T16:37:08.181 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 95/134 2023-11-16T16:37:08.355 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 95/134 2023-11-16T16:37:08.494 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 96/134 2023-11-16T16:37:12.603 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 96/134 2023-11-16T16:37:12.604 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /sys 2023-11-16T16:37:12.604 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /proc 2023-11-16T16:37:12.604 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /dev 2023-11-16T16:37:12.604 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /run 2023-11-16T16:37:12.604 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /mnt 2023-11-16T16:37:12.604 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /var/tmp 2023-11-16T16:37:12.604 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /home 2023-11-16T16:37:12.604 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /tmp 2023-11-16T16:37:12.604 INFO:teuthology.orchestra.run.smithi141.stdout:skipping the directory /dev 2023-11-16T16:37:12.604 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-11-16T16:37:12.961 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 97/134 2023-11-16T16:37:14.521 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 97/134 2023-11-16T16:37:14.731 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 98/134 2023-11-16T16:37:14.949 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 98/134 2023-11-16T16:37:15.119 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 99/134 2023-11-16T16:37:15.570 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 99/134 2023-11-16T16:37:15.736 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 100/134 2023-11-16T16:37:15.754 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 100/134 2023-11-16T16:37:17.366 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 96/134 2023-11-16T16:37:17.366 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /sys 2023-11-16T16:37:17.366 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /proc 2023-11-16T16:37:17.366 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /dev 2023-11-16T16:37:17.366 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /run 2023-11-16T16:37:17.366 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /mnt 2023-11-16T16:37:17.366 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /var/tmp 2023-11-16T16:37:17.366 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /home 2023-11-16T16:37:17.367 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /tmp 2023-11-16T16:37:17.367 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /dev 2023-11-16T16:37:17.367 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-11-16T16:37:17.450 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 101/134 2023-11-16T16:37:17.641 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 101/134 2023-11-16T16:37:17.719 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 97/134 2023-11-16T16:37:17.775 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.e 102/134 2023-11-16T16:37:17.975 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a. 103/134 2023-11-16T16:37:18.160 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 104/134 2023-11-16T16:37:18.372 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 104/134 2023-11-16T16:37:18.696 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 105/134 2023-11-16T16:37:19.277 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 97/134 2023-11-16T16:37:19.443 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 98/134 2023-11-16T16:37:19.658 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 105/134 2023-11-16T16:37:19.662 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 98/134 2023-11-16T16:37:19.810 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 106/134 2023-11-16T16:37:19.827 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 99/134 2023-11-16T16:37:19.900 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 106/134 2023-11-16T16:37:20.277 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 99/134 2023-11-16T16:37:20.282 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el 107/134 2023-11-16T16:37:20.436 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 100/134 2023-11-16T16:37:20.453 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 100/134 2023-11-16T16:37:20.516 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 108/134 2023-11-16T16:37:20.551 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 108/134 2023-11-16T16:37:20.701 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 109/134 2023-11-16T16:37:20.712 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 109/134 2023-11-16T16:37:21.993 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 110/134 2023-11-16T16:37:22.128 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 101/134 2023-11-16T16:37:22.188 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 111/134 2023-11-16T16:37:22.222 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 111/134 2023-11-16T16:37:22.296 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 101/134 2023-11-16T16:37:22.401 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 112/134 2023-11-16T16:37:22.408 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.e 102/134 2023-11-16T16:37:22.453 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 112/134 2023-11-16T16:37:22.582 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a. 103/134 2023-11-16T16:37:22.584 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 113/134 2023-11-16T16:37:22.751 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 104/134 2023-11-16T16:37:22.929 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 113/134 2023-11-16T16:37:22.967 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 104/134 2023-11-16T16:37:23.126 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 114/134 2023-11-16T16:37:23.259 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 105/134 2023-11-16T16:37:24.219 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 105/134 2023-11-16T16:37:24.376 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 106/134 2023-11-16T16:37:24.467 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 106/134 2023-11-16T16:37:24.820 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el 107/134 2023-11-16T16:37:25.053 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 108/134 2023-11-16T16:37:25.089 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 108/134 2023-11-16T16:37:25.226 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 109/134 2023-11-16T16:37:25.238 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 109/134 2023-11-16T16:37:26.461 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 110/134 2023-11-16T16:37:26.670 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 111/134 2023-11-16T16:37:26.705 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 111/134 2023-11-16T16:37:26.835 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 112/134 2023-11-16T16:37:26.882 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 112/134 2023-11-16T16:37:27.009 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 113/134 2023-11-16T16:37:27.026 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 114/134 2023-11-16T16:37:27.318 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 115/134 2023-11-16T16:37:27.353 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 113/134 2023-11-16T16:37:27.492 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-11-16T16:37:27.501 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 114/134 2023-11-16T16:37:27.686 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-11-16T16:37:28.568 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-11-16T16:37:28.890 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-11-16T16:37:29.094 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 119/134 2023-11-16T16:37:29.312 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-16T16:37:29.487 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_ 121/134 2023-11-16T16:37:29.642 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T16:37:29.899 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 96/134 2023-11-16T16:37:29.899 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /sys 2023-11-16T16:37:29.899 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /proc 2023-11-16T16:37:29.899 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /dev 2023-11-16T16:37:29.899 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /run 2023-11-16T16:37:29.899 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /mnt 2023-11-16T16:37:29.900 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /var/tmp 2023-11-16T16:37:29.900 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /home 2023-11-16T16:37:29.900 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /tmp 2023-11-16T16:37:29.900 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /dev 2023-11-16T16:37:29.900 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:37:30.283 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 97/134 2023-11-16T16:37:31.409 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 114/134 2023-11-16T16:37:31.675 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 115/134 2023-11-16T16:37:31.825 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-11-16T16:37:31.851 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T16:37:32.022 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-11-16T16:37:32.092 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 97/134 2023-11-16T16:37:32.108 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T16:37:32.125 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T16:37:32.283 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 124/134 2023-11-16T16:37:32.294 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 98/134 2023-11-16T16:37:32.575 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 98/134 2023-11-16T16:37:32.590 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 125/134 2023-11-16T16:37:32.791 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 99/134 2023-11-16T16:37:32.886 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 126/134 2023-11-16T16:37:32.906 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-11-16T16:37:33.198 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-11-16T16:37:33.218 INFO:teuthology.orchestra.run.smithi141.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-11-16T16:37:33.340 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 99/134 2023-11-16T16:37:33.374 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T16:37:33.383 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T16:37:33.403 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 119/134 2023-11-16T16:37:33.566 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 100/134 2023-11-16T16:37:33.581 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-11-16T16:37:33.596 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-16T16:37:33.610 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 100/134 2023-11-16T16:37:33.749 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-11-16T16:37:33.754 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_ 121/134 2023-11-16T16:37:33.837 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-11-16T16:37:33.901 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T16:37:34.065 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-11-16T16:37:34.078 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-11-16T16:37:34.078 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T16:37:34.260 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T16:37:34.260 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T16:37:34.479 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T16:37:34.723 INFO:teuthology.orchestra.run.smithi141.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T16:37:35.217 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T16:37:35.218 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 1/134 2023-11-16T16:37:35.218 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 2/134 2023-11-16T16:37:35.218 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 3/134 2023-11-16T16:37:35.218 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 4/134 2023-11-16T16:37:35.218 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 5/134 2023-11-16T16:37:35.218 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T16:37:35.218 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T16:37:35.218 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T16:37:35.218 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 9/134 2023-11-16T16:37:35.218 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 10/134 2023-11-16T16:37:35.218 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 11/134 2023-11-16T16:37:35.219 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 12/134 2023-11-16T16:37:35.219 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 13/134 2023-11-16T16:37:35.219 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_ 14/134 2023-11-16T16:37:35.219 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 15/134 2023-11-16T16:37:35.219 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 16/134 2023-11-16T16:37:35.219 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_6 17/134 2023-11-16T16:37:35.219 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 18/134 2023-11-16T16:37:35.219 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 19/134 2023-11-16T16:37:35.219 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el 20/134 2023-11-16T16:37:35.219 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8. 21/134 2023-11-16T16:37:35.219 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_6 22/134 2023-11-16T16:37:35.219 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 23/134 2023-11-16T16:37:35.219 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 24/134 2023-11-16T16:37:35.220 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 25/134 2023-11-16T16:37:35.220 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 26/134 2023-11-16T16:37:35.220 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 27/134 2023-11-16T16:37:35.220 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 28/134 2023-11-16T16:37:35.220 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a. 29/134 2023-11-16T16:37:35.220 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 30/134 2023-11-16T16:37:35.220 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 31/134 2023-11-16T16:37:35.220 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 32/134 2023-11-16T16:37:35.220 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el 33/134 2023-11-16T16:37:35.220 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 34/134 2023-11-16T16:37:35.220 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.e 35/134 2023-11-16T16:37:35.221 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 36/134 2023-11-16T16:37:35.221 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 37/134 2023-11-16T16:37:35.221 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 38/134 2023-11-16T16:37:35.221 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-11-16T16:37:35.221 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-11-16T16:37:35.221 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-11-16T16:37:35.221 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-11-16T16:37:35.221 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-11-16T16:37:35.221 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-11-16T16:37:35.221 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-11-16T16:37:35.221 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-11-16T16:37:35.221 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-11-16T16:37:35.221 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-11-16T16:37:35.222 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-11-16T16:37:35.222 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-11-16T16:37:35.222 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-11-16T16:37:35.222 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-11-16T16:37:35.222 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-11-16T16:37:35.222 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-11-16T16:37:35.222 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-11-16T16:37:35.222 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-11-16T16:37:35.222 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-11-16T16:37:35.222 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-11-16T16:37:35.222 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-11-16T16:37:35.223 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-11-16T16:37:35.223 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-11-16T16:37:35.223 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-11-16T16:37:35.223 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-11-16T16:37:35.223 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-11-16T16:37:35.223 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-11-16T16:37:35.223 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-11-16T16:37:35.223 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-11-16T16:37:35.223 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-11-16T16:37:35.224 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-11-16T16:37:35.224 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-11-16T16:37:35.224 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-11-16T16:37:35.224 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-11-16T16:37:35.224 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-11-16T16:37:35.224 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-11-16T16:37:35.224 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-11-16T16:37:35.224 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-11-16T16:37:35.224 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-11-16T16:37:35.224 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-11-16T16:37:35.224 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-11-16T16:37:35.224 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-11-16T16:37:35.224 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-11-16T16:37:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-11-16T16:37:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-11-16T16:37:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-11-16T16:37:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-11-16T16:37:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-11-16T16:37:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-11-16T16:37:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-11-16T16:37:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-11-16T16:37:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-11-16T16:37:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-11-16T16:37:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-11-16T16:37:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-11-16T16:37:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-11-16T16:37:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-11-16T16:37:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-11-16T16:37:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-11-16T16:37:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-11-16T16:37:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-11-16T16:37:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-11-16T16:37:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-11-16T16:37:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-11-16T16:37:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-11-16T16:37:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-11-16T16:37:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-11-16T16:37:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-11-16T16:37:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-11-16T16:37:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-11-16T16:37:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-11-16T16:37:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-11-16T16:37:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-11-16T16:37:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-11-16T16:37:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-11-16T16:37:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-11-16T16:37:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-11-16T16:37:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-11-16T16:37:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-11-16T16:37:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-11-16T16:37:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-11-16T16:37:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-16T16:37:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-11-16T16:37:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T16:37:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-11-16T16:37:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-11-16T16:37:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-11-16T16:37:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-11-16T16:37:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-11-16T16:37:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T16:37:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 129/134 2023-11-16T16:37:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-11-16T16:37:35.229 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 131/134 2023-11-16T16:37:35.229 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T16:37:35.229 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-11-16T16:37:36.117 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T16:37:36.366 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T16:37:36.385 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T16:37:36.547 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 101/134 2023-11-16T16:37:36.550 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 124/134 2023-11-16T16:37:36.730 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 101/134 2023-11-16T16:37:36.866 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 125/134 2023-11-16T16:37:36.881 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.e 102/134 2023-11-16T16:37:37.071 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a. 103/134 2023-11-16T16:37:37.196 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 126/134 2023-11-16T16:37:37.248 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 104/134 2023-11-16T16:37:37.476 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-11-16T16:37:37.520 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 104/134 2023-11-16T16:37:37.616 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T16:37:37.624 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T16:37:37.701 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T16:37:37.701 INFO:teuthology.orchestra.run.smithi141.stdout:Last metadata expiration check: 0:01:51 ago on Thu 16 Nov 2023 04:35:46 PM UTC. 2023-11-16T16:37:37.709 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 105/134 2023-11-16T16:37:37.804 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-11-16T16:37:38.007 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-11-16T16:37:38.088 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-11-16T16:37:38.283 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-11-16T16:37:38.295 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-11-16T16:37:38.295 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T16:37:38.486 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T16:37:38.486 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T16:37:38.704 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T16:37:38.873 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T16:37:38.873 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 105/134 2023-11-16T16:37:39.165 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 106/134 2023-11-16T16:37:39.301 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 106/134 2023-11-16T16:37:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T16:37:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 1/134 2023-11-16T16:37:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 2/134 2023-11-16T16:37:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 3/134 2023-11-16T16:37:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 4/134 2023-11-16T16:37:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 5/134 2023-11-16T16:37:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T16:37:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T16:37:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T16:37:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 9/134 2023-11-16T16:37:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 10/134 2023-11-16T16:37:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 11/134 2023-11-16T16:37:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 12/134 2023-11-16T16:37:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 13/134 2023-11-16T16:37:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_ 14/134 2023-11-16T16:37:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 15/134 2023-11-16T16:37:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 16/134 2023-11-16T16:37:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_6 17/134 2023-11-16T16:37:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 18/134 2023-11-16T16:37:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 19/134 2023-11-16T16:37:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el 20/134 2023-11-16T16:37:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8. 21/134 2023-11-16T16:37:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_6 22/134 2023-11-16T16:37:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 23/134 2023-11-16T16:37:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 24/134 2023-11-16T16:37:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 25/134 2023-11-16T16:37:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 26/134 2023-11-16T16:37:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 27/134 2023-11-16T16:37:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 28/134 2023-11-16T16:37:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a. 29/134 2023-11-16T16:37:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 30/134 2023-11-16T16:37:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 31/134 2023-11-16T16:37:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 32/134 2023-11-16T16:37:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el 33/134 2023-11-16T16:37:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 34/134 2023-11-16T16:37:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.e 35/134 2023-11-16T16:37:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 36/134 2023-11-16T16:37:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 37/134 2023-11-16T16:37:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 38/134 2023-11-16T16:37:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-11-16T16:37:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-11-16T16:37:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-11-16T16:37:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-11-16T16:37:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-11-16T16:37:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-11-16T16:37:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-11-16T16:37:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-11-16T16:37:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-11-16T16:37:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-11-16T16:37:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-11-16T16:37:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-11-16T16:37:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-11-16T16:37:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-11-16T16:37:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-11-16T16:37:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-11-16T16:37:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-11-16T16:37:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-11-16T16:37:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-11-16T16:37:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-11-16T16:37:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-11-16T16:37:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-11-16T16:37:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-11-16T16:37:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-11-16T16:37:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-11-16T16:37:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-11-16T16:37:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-11-16T16:37:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-11-16T16:37:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-11-16T16:37:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-11-16T16:37:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-11-16T16:37:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-11-16T16:37:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-11-16T16:37:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-11-16T16:37:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-11-16T16:37:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-11-16T16:37:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-11-16T16:37:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-11-16T16:37:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-11-16T16:37:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-11-16T16:37:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-11-16T16:37:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-11-16T16:37:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-11-16T16:37:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-11-16T16:37:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-11-16T16:37:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-11-16T16:37:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-11-16T16:37:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-11-16T16:37:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-11-16T16:37:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-11-16T16:37:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-11-16T16:37:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-11-16T16:37:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-11-16T16:37:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-11-16T16:37:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-11-16T16:37:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-11-16T16:37:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-11-16T16:37:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-11-16T16:37:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-11-16T16:37:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-11-16T16:37:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-11-16T16:37:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-11-16T16:37:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-11-16T16:37:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-11-16T16:37:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-11-16T16:37:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-11-16T16:37:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-11-16T16:37:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-11-16T16:37:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-11-16T16:37:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-11-16T16:37:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-11-16T16:37:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-11-16T16:37:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-11-16T16:37:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-11-16T16:37:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-11-16T16:37:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-11-16T16:37:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-11-16T16:37:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-11-16T16:37:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-11-16T16:37:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-11-16T16:37:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-11-16T16:37:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-16T16:37:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-11-16T16:37:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T16:37:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-11-16T16:37:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-11-16T16:37:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-11-16T16:37:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-11-16T16:37:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-11-16T16:37:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T16:37:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 129/134 2023-11-16T16:37:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-11-16T16:37:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 131/134 2023-11-16T16:37:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T16:37:39.434 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-11-16T16:37:39.676 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el 107/134 2023-11-16T16:37:39.887 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 108/134 2023-11-16T16:37:39.955 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 108/134 2023-11-16T16:37:40.165 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 109/134 2023-11-16T16:37:40.195 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 109/134 2023-11-16T16:37:41.595 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 110/134 2023-11-16T16:37:41.774 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 111/134 2023-11-16T16:37:41.838 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 111/134 2023-11-16T16:37:41.905 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T16:37:41.905 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:54 ago on Thu 16 Nov 2023 04:35:47 PM UTC. 2023-11-16T16:37:42.114 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 112/134 2023-11-16T16:37:42.199 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 112/134 2023-11-16T16:37:42.398 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 113/134 2023-11-16T16:37:42.834 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 113/134 2023-11-16T16:37:42.998 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 114/134 2023-11-16T16:37:44.939 INFO:teuthology.orchestra.run.smithi141.stdout:Installed products updated. 2023-11-16T16:37:45.152 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-11-16T16:37:45.152 INFO:teuthology.orchestra.run.smithi141.stdout:Upgraded: 2023-11-16T16:37:45.153 INFO:teuthology.orchestra.run.smithi141.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-11-16T16:37:45.153 INFO:teuthology.orchestra.run.smithi141.stdout: librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.153 INFO:teuthology.orchestra.run.smithi141.stdout: librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.153 INFO:teuthology.orchestra.run.smithi141.stdout:Installed: 2023-11-16T16:37:45.153 INFO:teuthology.orchestra.run.smithi141.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-11-16T16:37:45.153 INFO:teuthology.orchestra.run.smithi141.stdout: automake-1.16.1-7.el8.noarch 2023-11-16T16:37:45.153 INFO:teuthology.orchestra.run.smithi141.stdout: bison-3.0.4-10.el8.x86_64 2023-11-16T16:37:45.153 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.154 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.154 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.154 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.154 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:45.154 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-immutable-object-cache-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.154 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.154 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.155 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:45.155 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:45.155 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:45.155 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:45.155 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:45.155 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.155 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.155 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:45.156 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.156 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.156 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.156 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:45.156 INFO:teuthology.orchestra.run.smithi141.stdout: cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:45.156 INFO:teuthology.orchestra.run.smithi141.stdout: cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.156 INFO:teuthology.orchestra.run.smithi141.stdout: cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:45.156 INFO:teuthology.orchestra.run.smithi141.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-11-16T16:37:45.157 INFO:teuthology.orchestra.run.smithi141.stdout: flex-2.6.1-9.el8.x86_64 2023-11-16T16:37:45.157 INFO:teuthology.orchestra.run.smithi141.stdout: fuse-2.9.7-15.el8.x86_64 2023-11-16T16:37:45.157 INFO:teuthology.orchestra.run.smithi141.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-16T16:37:45.157 INFO:teuthology.orchestra.run.smithi141.stdout: jq-1.6-3.el8.x86_64 2023-11-16T16:37:45.157 INFO:teuthology.orchestra.run.smithi141.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-16T16:37:45.157 INFO:teuthology.orchestra.run.smithi141.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-11-16T16:37:45.157 INFO:teuthology.orchestra.run.smithi141.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-16T16:37:45.157 INFO:teuthology.orchestra.run.smithi141.stdout: libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.158 INFO:teuthology.orchestra.run.smithi141.stdout: libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.158 INFO:teuthology.orchestra.run.smithi141.stdout: libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.158 INFO:teuthology.orchestra.run.smithi141.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-11-16T16:37:45.158 INFO:teuthology.orchestra.run.smithi141.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-16T16:37:45.158 INFO:teuthology.orchestra.run.smithi141.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-16T16:37:45.158 INFO:teuthology.orchestra.run.smithi141.stdout: libpq-13.5-1.el8.x86_64 2023-11-16T16:37:45.158 INFO:teuthology.orchestra.run.smithi141.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-16T16:37:45.158 INFO:teuthology.orchestra.run.smithi141.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-16T16:37:45.159 INFO:teuthology.orchestra.run.smithi141.stdout: librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.159 INFO:teuthology.orchestra.run.smithi141.stdout: libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.159 INFO:teuthology.orchestra.run.smithi141.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-16T16:37:45.159 INFO:teuthology.orchestra.run.smithi141.stdout: librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.159 INFO:teuthology.orchestra.run.smithi141.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-11-16T16:37:45.159 INFO:teuthology.orchestra.run.smithi141.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-16T16:37:45.159 INFO:teuthology.orchestra.run.smithi141.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-16T16:37:45.159 INFO:teuthology.orchestra.run.smithi141.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-16T16:37:45.160 INFO:teuthology.orchestra.run.smithi141.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-11-16T16:37:45.160 INFO:teuthology.orchestra.run.smithi141.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-11-16T16:37:45.160 INFO:teuthology.orchestra.run.smithi141.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-16T16:37:45.160 INFO:teuthology.orchestra.run.smithi141.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-16T16:37:45.160 INFO:teuthology.orchestra.run.smithi141.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-16T16:37:45.160 INFO:teuthology.orchestra.run.smithi141.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-16T16:37:45.160 INFO:teuthology.orchestra.run.smithi141.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-11-16T16:37:45.160 INFO:teuthology.orchestra.run.smithi141.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-11-16T16:37:45.161 INFO:teuthology.orchestra.run.smithi141.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-11-16T16:37:45.161 INFO:teuthology.orchestra.run.smithi141.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-11-16T16:37:45.161 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T16:37:45.161 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T16:37:45.161 INFO:teuthology.orchestra.run.smithi141.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T16:37:45.161 INFO:teuthology.orchestra.run.smithi141.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-16T16:37:45.161 INFO:teuthology.orchestra.run.smithi141.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-16T16:37:45.161 INFO:teuthology.orchestra.run.smithi141.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-16T16:37:45.162 INFO:teuthology.orchestra.run.smithi141.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-16T16:37:45.162 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-16T16:37:45.162 INFO:teuthology.orchestra.run.smithi141.stdout: python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.162 INFO:teuthology.orchestra.run.smithi141.stdout: python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.162 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.162 INFO:teuthology.orchestra.run.smithi141.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-16T16:37:45.162 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-16T16:37:45.162 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-16T16:37:45.163 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-16T16:37:45.163 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-16T16:37:45.163 INFO:teuthology.orchestra.run.smithi141.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-16T16:37:45.163 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-16T16:37:45.163 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-16T16:37:45.163 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-16T16:37:45.163 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-16T16:37:45.163 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-16T16:37:45.164 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-16T16:37:45.164 INFO:teuthology.orchestra.run.smithi141.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-16T16:37:45.164 INFO:teuthology.orchestra.run.smithi141.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-16T16:37:45.164 INFO:teuthology.orchestra.run.smithi141.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-16T16:37:45.164 INFO:teuthology.orchestra.run.smithi141.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-16T16:37:45.164 INFO:teuthology.orchestra.run.smithi141.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-16T16:37:45.164 INFO:teuthology.orchestra.run.smithi141.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-16T16:37:45.164 INFO:teuthology.orchestra.run.smithi141.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-16T16:37:45.165 INFO:teuthology.orchestra.run.smithi141.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-16T16:37:45.165 INFO:teuthology.orchestra.run.smithi141.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-16T16:37:45.165 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-16T16:37:45.165 INFO:teuthology.orchestra.run.smithi141.stdout: python3-portend-2.6-1.el8.noarch 2023-11-16T16:37:45.165 INFO:teuthology.orchestra.run.smithi141.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-16T16:37:45.165 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-16T16:37:45.165 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-16T16:37:45.165 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-16T16:37:45.166 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-16T16:37:45.166 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-16T16:37:45.166 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.166 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.166 INFO:teuthology.orchestra.run.smithi141.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-16T16:37:45.166 INFO:teuthology.orchestra.run.smithi141.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-16T16:37:45.166 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.166 INFO:teuthology.orchestra.run.smithi141.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-16T16:37:45.167 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-16T16:37:45.167 INFO:teuthology.orchestra.run.smithi141.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-16T16:37:45.167 INFO:teuthology.orchestra.run.smithi141.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-16T16:37:45.167 INFO:teuthology.orchestra.run.smithi141.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-16T16:37:45.167 INFO:teuthology.orchestra.run.smithi141.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-16T16:37:45.167 INFO:teuthology.orchestra.run.smithi141.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-16T16:37:45.167 INFO:teuthology.orchestra.run.smithi141.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-16T16:37:45.167 INFO:teuthology.orchestra.run.smithi141.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-16T16:37:45.167 INFO:teuthology.orchestra.run.smithi141.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-16T16:37:45.168 INFO:teuthology.orchestra.run.smithi141.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-16T16:37:45.168 INFO:teuthology.orchestra.run.smithi141.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-16T16:37:45.168 INFO:teuthology.orchestra.run.smithi141.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-16T16:37:45.168 INFO:teuthology.orchestra.run.smithi141.stdout: rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.168 INFO:teuthology.orchestra.run.smithi141.stdout: rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.168 INFO:teuthology.orchestra.run.smithi141.stdout: rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:45.168 INFO:teuthology.orchestra.run.smithi141.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-16T16:37:45.168 INFO:teuthology.orchestra.run.smithi141.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-16T16:37:45.168 INFO:teuthology.orchestra.run.smithi141.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-16T16:37:45.169 INFO:teuthology.orchestra.run.smithi141.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-11-16T16:37:45.169 INFO:teuthology.orchestra.run.smithi141.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-16T16:37:45.169 INFO:teuthology.orchestra.run.smithi141.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-11-16T16:37:45.169 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-11-16T16:37:45.169 INFO:teuthology.orchestra.run.smithi141.stdout:Complete! 2023-11-16T16:37:45.344 DEBUG:teuthology.parallel:result is None 2023-11-16T16:37:47.529 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 114/134 2023-11-16T16:37:47.846 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 115/134 2023-11-16T16:37:47.988 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-11-16T16:37:48.268 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-11-16T16:37:49.438 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2023-11-16T16:37:49.571 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-11-16T16:37:49.656 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-11-16T16:37:49.656 INFO:teuthology.orchestra.run.smithi155.stdout:Upgraded: 2023-11-16T16:37:49.656 INFO:teuthology.orchestra.run.smithi155.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-11-16T16:37:49.656 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.656 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.656 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2023-11-16T16:37:49.657 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-11-16T16:37:49.657 INFO:teuthology.orchestra.run.smithi155.stdout: automake-1.16.1-7.el8.noarch 2023-11-16T16:37:49.657 INFO:teuthology.orchestra.run.smithi155.stdout: bison-3.0.4-10.el8.x86_64 2023-11-16T16:37:49.657 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.657 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.657 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.657 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.657 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:49.657 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.657 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.657 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.657 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:49.657 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:49.657 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:49.658 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:49.658 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:49.658 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.658 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.658 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:49.658 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.658 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.658 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.658 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:49.658 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:49.658 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.658 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:37:49.658 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-11-16T16:37:49.659 INFO:teuthology.orchestra.run.smithi155.stdout: flex-2.6.1-9.el8.x86_64 2023-11-16T16:37:49.659 INFO:teuthology.orchestra.run.smithi155.stdout: fuse-2.9.7-15.el8.x86_64 2023-11-16T16:37:49.659 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-16T16:37:49.659 INFO:teuthology.orchestra.run.smithi155.stdout: jq-1.6-3.el8.x86_64 2023-11-16T16:37:49.659 INFO:teuthology.orchestra.run.smithi155.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-16T16:37:49.659 INFO:teuthology.orchestra.run.smithi155.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-11-16T16:37:49.659 INFO:teuthology.orchestra.run.smithi155.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-16T16:37:49.659 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.659 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.659 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.659 INFO:teuthology.orchestra.run.smithi155.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-11-16T16:37:49.659 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-16T16:37:49.659 INFO:teuthology.orchestra.run.smithi155.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-16T16:37:49.659 INFO:teuthology.orchestra.run.smithi155.stdout: libpq-13.5-1.el8.x86_64 2023-11-16T16:37:49.660 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-16T16:37:49.660 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-16T16:37:49.660 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.660 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.660 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-16T16:37:49.660 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.660 INFO:teuthology.orchestra.run.smithi155.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-11-16T16:37:49.660 INFO:teuthology.orchestra.run.smithi155.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-16T16:37:49.660 INFO:teuthology.orchestra.run.smithi155.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-16T16:37:49.660 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-16T16:37:49.660 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-11-16T16:37:49.660 INFO:teuthology.orchestra.run.smithi155.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-11-16T16:37:49.660 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-16T16:37:49.660 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-16T16:37:49.661 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-16T16:37:49.661 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-16T16:37:49.661 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-11-16T16:37:49.661 INFO:teuthology.orchestra.run.smithi155.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-11-16T16:37:49.661 INFO:teuthology.orchestra.run.smithi155.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-11-16T16:37:49.661 INFO:teuthology.orchestra.run.smithi155.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-11-16T16:37:49.661 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T16:37:49.661 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T16:37:49.661 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T16:37:49.661 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-16T16:37:49.661 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-16T16:37:49.661 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-16T16:37:49.661 INFO:teuthology.orchestra.run.smithi155.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-16T16:37:49.661 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-16T16:37:49.662 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.662 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.662 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.662 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-16T16:37:49.662 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-16T16:37:49.662 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-16T16:37:49.662 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-16T16:37:49.662 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-16T16:37:49.662 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-16T16:37:49.662 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-16T16:37:49.662 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-16T16:37:49.662 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-16T16:37:49.663 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-16T16:37:49.663 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-16T16:37:49.663 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-16T16:37:49.663 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-16T16:37:49.663 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-16T16:37:49.663 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-16T16:37:49.663 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-16T16:37:49.663 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-16T16:37:49.663 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-16T16:37:49.663 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-16T16:37:49.663 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-16T16:37:49.663 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-16T16:37:49.663 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-16T16:37:49.663 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend-2.6-1.el8.noarch 2023-11-16T16:37:49.664 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-16T16:37:49.664 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-16T16:37:49.664 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-16T16:37:49.664 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-16T16:37:49.664 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-16T16:37:49.664 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-16T16:37:49.664 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.664 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.664 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-16T16:37:49.664 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-16T16:37:49.664 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.664 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-16T16:37:49.664 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-16T16:37:49.664 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-16T16:37:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-16T16:37:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-16T16:37:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-16T16:37:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-16T16:37:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-16T16:37:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-16T16:37:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-16T16:37:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-16T16:37:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-16T16:37:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-16T16:37:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:37:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-16T16:37:49.665 INFO:teuthology.orchestra.run.smithi155.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-16T16:37:49.666 INFO:teuthology.orchestra.run.smithi155.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-16T16:37:49.666 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-11-16T16:37:49.666 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-16T16:37:49.666 INFO:teuthology.orchestra.run.smithi155.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-11-16T16:37:49.666 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-11-16T16:37:49.666 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2023-11-16T16:37:49.817 DEBUG:teuthology.parallel:result is None 2023-11-16T16:37:49.916 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-11-16T16:37:50.262 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 119/134 2023-11-16T16:37:50.468 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-16T16:37:50.669 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_ 121/134 2023-11-16T16:37:50.814 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T16:37:54.463 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T16:37:54.746 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T16:37:54.786 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-16T16:37:55.064 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 124/134 2023-11-16T16:37:55.281 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 125/134 2023-11-16T16:37:55.747 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 126/134 2023-11-16T16:37:56.048 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-11-16T16:37:56.220 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T16:37:56.247 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T16:37:56.507 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-11-16T16:37:56.686 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-11-16T16:37:56.810 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-11-16T16:37:57.062 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-11-16T16:37:57.094 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-11-16T16:37:57.094 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T16:37:57.344 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T16:37:57.344 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T16:37:57.612 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-16T16:37:57.863 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T16:37:58.256 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T16:37:58.256 INFO:teuthology.orchestra.run.smithi002.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-11-16T16:37:58.256 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:37:58.926 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 1/134 2023-11-16T16:37:58.926 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 2/134 2023-11-16T16:37:58.926 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 3/134 2023-11-16T16:37:58.926 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 4/134 2023-11-16T16:37:58.926 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3905.g5e4a7 5/134 2023-11-16T16:37:58.926 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 6/134 2023-11-16T16:37:58.926 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 7/134 2023-11-16T16:37:58.926 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 8/134 2023-11-16T16:37:58.926 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 9/134 2023-11-16T16:37:58.927 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 10/134 2023-11-16T16:37:58.927 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 11/134 2023-11-16T16:37:58.927 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 12/134 2023-11-16T16:37:58.927 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 13/134 2023-11-16T16:37:58.927 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_ 14/134 2023-11-16T16:37:58.927 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 15/134 2023-11-16T16:37:58.927 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 16/134 2023-11-16T16:37:58.927 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_6 17/134 2023-11-16T16:37:58.927 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86 18/134 2023-11-16T16:37:58.927 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 19/134 2023-11-16T16:37:58.927 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el 20/134 2023-11-16T16:37:58.927 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8. 21/134 2023-11-16T16:37:58.928 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_6 22/134 2023-11-16T16:37:58.928 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 23/134 2023-11-16T16:37:58.928 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 24/134 2023-11-16T16:37:58.928 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 25/134 2023-11-16T16:37:58.928 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 26/134 2023-11-16T16:37:58.928 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 27/134 2023-11-16T16:37:58.928 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 28/134 2023-11-16T16:37:58.928 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a. 29/134 2023-11-16T16:37:58.928 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noa 30/134 2023-11-16T16:37:58.928 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.n 31/134 2023-11-16T16:37:58.929 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4 32/134 2023-11-16T16:37:58.929 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el 33/134 2023-11-16T16:37:58.929 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 34/134 2023-11-16T16:37:58.929 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.e 35/134 2023-11-16T16:37:58.929 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 36/134 2023-11-16T16:37:58.929 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 37/134 2023-11-16T16:37:58.929 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 38/134 2023-11-16T16:37:58.929 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-11-16T16:37:58.929 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-11-16T16:37:58.929 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-11-16T16:37:58.929 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-11-16T16:37:58.929 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-11-16T16:37:58.930 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-11-16T16:37:58.930 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-11-16T16:37:58.930 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-11-16T16:37:58.930 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-11-16T16:37:58.930 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-11-16T16:37:58.930 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-11-16T16:37:58.930 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-11-16T16:37:58.930 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-11-16T16:37:58.930 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-11-16T16:37:58.930 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-11-16T16:37:58.930 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-11-16T16:37:58.931 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-11-16T16:37:58.931 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-11-16T16:37:58.931 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-11-16T16:37:58.931 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-11-16T16:37:58.931 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-11-16T16:37:58.931 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-11-16T16:37:58.931 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-11-16T16:37:58.931 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-11-16T16:37:58.931 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-11-16T16:37:58.931 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-11-16T16:37:58.931 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-11-16T16:37:58.931 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-11-16T16:37:58.932 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-11-16T16:37:58.932 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-11-16T16:37:58.932 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-11-16T16:37:58.932 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-11-16T16:37:58.932 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-11-16T16:37:58.932 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-11-16T16:37:58.932 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-11-16T16:37:58.932 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-11-16T16:37:58.932 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-11-16T16:37:58.932 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-11-16T16:37:58.932 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-11-16T16:37:58.932 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-11-16T16:37:58.933 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-11-16T16:37:58.933 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-11-16T16:37:58.933 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-11-16T16:37:58.933 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-11-16T16:37:58.933 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-11-16T16:37:58.933 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-11-16T16:37:58.933 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-11-16T16:37:58.933 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-11-16T16:37:58.933 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-11-16T16:37:58.933 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-11-16T16:37:58.933 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-11-16T16:37:58.933 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-11-16T16:37:58.934 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-11-16T16:37:58.934 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-11-16T16:37:58.934 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-11-16T16:37:58.934 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-11-16T16:37:58.934 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-11-16T16:37:58.934 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-11-16T16:37:58.934 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-11-16T16:37:58.934 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-11-16T16:37:58.934 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-11-16T16:37:58.934 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-11-16T16:37:58.934 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-11-16T16:37:58.934 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-11-16T16:37:58.935 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-11-16T16:37:58.935 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-11-16T16:37:58.935 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-11-16T16:37:58.935 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-11-16T16:37:58.935 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-11-16T16:37:58.935 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-11-16T16:37:58.935 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-11-16T16:37:58.935 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-11-16T16:37:58.935 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-11-16T16:37:58.935 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-11-16T16:37:58.935 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-11-16T16:37:58.935 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-11-16T16:37:58.936 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-11-16T16:37:58.936 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-11-16T16:37:58.936 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-11-16T16:37:58.936 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-11-16T16:37:58.936 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-11-16T16:37:58.936 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-16T16:37:58.936 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-11-16T16:37:58.936 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-16T16:37:58.936 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-11-16T16:37:58.936 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-11-16T16:37:58.936 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-11-16T16:37:58.936 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-11-16T16:37:58.936 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-11-16T16:37:58.937 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-16T16:37:58.937 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 129/134 2023-11-16T16:37:58.937 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-11-16T16:37:58.937 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 131/134 2023-11-16T16:37:58.937 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-16T16:37:58.937 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-11-16T16:38:01.656 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-16T16:38:01.656 INFO:teuthology.orchestra.run.smithi002.stdout:Last metadata expiration check: 0:02:11 ago on Thu 16 Nov 2023 04:35:50 PM UTC. 2023-11-16T16:38:07.918 INFO:teuthology.orchestra.run.smithi002.stdout:Installed products updated. 2023-11-16T16:38:08.161 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:38:08.162 INFO:teuthology.orchestra.run.smithi002.stdout:Upgraded: 2023-11-16T16:38:08.162 INFO:teuthology.orchestra.run.smithi002.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-11-16T16:38:08.162 INFO:teuthology.orchestra.run.smithi002.stdout: librados2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.162 INFO:teuthology.orchestra.run.smithi002.stdout: librbd1-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.162 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2023-11-16T16:38:08.162 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-11-16T16:38:08.162 INFO:teuthology.orchestra.run.smithi002.stdout: automake-1.16.1-7.el8.noarch 2023-11-16T16:38:08.162 INFO:teuthology.orchestra.run.smithi002.stdout: bison-3.0.4-10.el8.x86_64 2023-11-16T16:38:08.163 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.163 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.163 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.163 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.163 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-grafana-dashboards-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:38:08.163 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-immutable-object-cache-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.163 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mds-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.163 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.164 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:38:08.164 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:38:08.164 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:38:08.164 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:38:08.164 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-rook-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:38:08.164 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mon-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.164 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-osd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.164 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-prometheus-alerts-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:38:08.164 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-radosgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.165 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-selinux-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.165 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-test-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.165 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-volume-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:38:08.165 INFO:teuthology.orchestra.run.smithi002.stdout: cephadm-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:38:08.165 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.165 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-top-2:18.0.0-3905.g5e4a743a.el8.noarch 2023-11-16T16:38:08.165 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-11-16T16:38:08.165 INFO:teuthology.orchestra.run.smithi002.stdout: flex-2.6.1-9.el8.x86_64 2023-11-16T16:38:08.165 INFO:teuthology.orchestra.run.smithi002.stdout: fuse-2.9.7-15.el8.x86_64 2023-11-16T16:38:08.166 INFO:teuthology.orchestra.run.smithi002.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-16T16:38:08.166 INFO:teuthology.orchestra.run.smithi002.stdout: jq-1.6-3.el8.x86_64 2023-11-16T16:38:08.166 INFO:teuthology.orchestra.run.smithi002.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-16T16:38:08.166 INFO:teuthology.orchestra.run.smithi002.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-11-16T16:38:08.166 INFO:teuthology.orchestra.run.smithi002.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-16T16:38:08.166 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-devel-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.166 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.166 INFO:teuthology.orchestra.run.smithi002.stdout: libcephsqlite-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.167 INFO:teuthology.orchestra.run.smithi002.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-11-16T16:38:08.167 INFO:teuthology.orchestra.run.smithi002.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-16T16:38:08.167 INFO:teuthology.orchestra.run.smithi002.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-16T16:38:08.167 INFO:teuthology.orchestra.run.smithi002.stdout: libpq-13.5-1.el8.x86_64 2023-11-16T16:38:08.167 INFO:teuthology.orchestra.run.smithi002.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-16T16:38:08.167 INFO:teuthology.orchestra.run.smithi002.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-16T16:38:08.167 INFO:teuthology.orchestra.run.smithi002.stdout: librados-devel-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.168 INFO:teuthology.orchestra.run.smithi002.stdout: libradosstriper1-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.168 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-16T16:38:08.168 INFO:teuthology.orchestra.run.smithi002.stdout: librgw2-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.168 INFO:teuthology.orchestra.run.smithi002.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-11-16T16:38:08.168 INFO:teuthology.orchestra.run.smithi002.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-16T16:38:08.168 INFO:teuthology.orchestra.run.smithi002.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-16T16:38:08.168 INFO:teuthology.orchestra.run.smithi002.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-16T16:38:08.168 INFO:teuthology.orchestra.run.smithi002.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-11-16T16:38:08.168 INFO:teuthology.orchestra.run.smithi002.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-11-16T16:38:08.169 INFO:teuthology.orchestra.run.smithi002.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-16T16:38:08.169 INFO:teuthology.orchestra.run.smithi002.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-16T16:38:08.169 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-16T16:38:08.169 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-16T16:38:08.169 INFO:teuthology.orchestra.run.smithi002.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-11-16T16:38:08.169 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-11-16T16:38:08.169 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-11-16T16:38:08.169 INFO:teuthology.orchestra.run.smithi002.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-11-16T16:38:08.169 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T16:38:08.170 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T16:38:08.170 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-16T16:38:08.170 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-16T16:38:08.170 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-16T16:38:08.170 INFO:teuthology.orchestra.run.smithi002.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-16T16:38:08.170 INFO:teuthology.orchestra.run.smithi002.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-16T16:38:08.170 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-16T16:38:08.170 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.170 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.171 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cephfs-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.171 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-16T16:38:08.171 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-16T16:38:08.171 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-16T16:38:08.171 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-16T16:38:08.171 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-16T16:38:08.171 INFO:teuthology.orchestra.run.smithi002.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-16T16:38:08.171 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-16T16:38:08.172 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-16T16:38:08.172 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-16T16:38:08.172 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-16T16:38:08.172 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-16T16:38:08.172 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-16T16:38:08.172 INFO:teuthology.orchestra.run.smithi002.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-16T16:38:08.172 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-16T16:38:08.172 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-16T16:38:08.172 INFO:teuthology.orchestra.run.smithi002.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-16T16:38:08.173 INFO:teuthology.orchestra.run.smithi002.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-16T16:38:08.173 INFO:teuthology.orchestra.run.smithi002.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-16T16:38:08.173 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-16T16:38:08.173 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-16T16:38:08.173 INFO:teuthology.orchestra.run.smithi002.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-16T16:38:08.173 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-16T16:38:08.173 INFO:teuthology.orchestra.run.smithi002.stdout: python3-portend-2.6-1.el8.noarch 2023-11-16T16:38:08.173 INFO:teuthology.orchestra.run.smithi002.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-16T16:38:08.173 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-16T16:38:08.174 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-16T16:38:08.174 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-16T16:38:08.174 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-16T16:38:08.174 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-16T16:38:08.174 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rados-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.174 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.174 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-16T16:38:08.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-16T16:38:08.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rgw-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-16T16:38:08.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-16T16:38:08.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-16T16:38:08.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-16T16:38:08.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-16T16:38:08.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-16T16:38:08.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-16T16:38:08.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-16T16:38:08.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-16T16:38:08.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-16T16:38:08.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-16T16:38:08.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-16T16:38:08.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-16T16:38:08.176 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-fuse-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.176 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-mirror-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.176 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-nbd-2:18.0.0-3905.g5e4a743a.el8.x86_64 2023-11-16T16:38:08.177 INFO:teuthology.orchestra.run.smithi002.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-16T16:38:08.177 INFO:teuthology.orchestra.run.smithi002.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-16T16:38:08.177 INFO:teuthology.orchestra.run.smithi002.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-16T16:38:08.177 INFO:teuthology.orchestra.run.smithi002.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-11-16T16:38:08.177 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-16T16:38:08.177 INFO:teuthology.orchestra.run.smithi002.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-11-16T16:38:08.178 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:38:08.178 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2023-11-16T16:38:08.476 DEBUG:teuthology.parallel:result is None 2023-11-16T16:38:08.476 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:38:08.697 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-16T16:38:08.752 INFO:teuthology.orchestra.run.smithi002.stdout:18.0.0-3905.g5e4a743a.el8 2023-11-16T16:38:08.753 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3905.g5e4a743a.el8 2023-11-16T16:38:08.753 INFO:teuthology.task.install:The correct ceph version 18.0.0-3905.g5e4a743a is installed. 2023-11-16T16:38:08.753 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:38:08.962 DEBUG:teuthology.orchestra.run.smithi141:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-16T16:38:08.992 INFO:teuthology.orchestra.run.smithi141.stdout:18.0.0-3905.g5e4a743a.el8 2023-11-16T16:38:08.993 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3905.g5e4a743a.el8 2023-11-16T16:38:08.993 INFO:teuthology.task.install:The correct ceph version 18.0.0-3905.g5e4a743a is installed. 2023-11-16T16:38:08.993 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:38:09.132 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-16T16:38:09.160 INFO:teuthology.orchestra.run.smithi155.stdout:18.0.0-3905.g5e4a743a.el8 2023-11-16T16:38:09.161 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3905.g5e4a743a.el8 2023-11-16T16:38:09.161 INFO:teuthology.task.install:The correct ceph version 18.0.0-3905.g5e4a743a is installed. 2023-11-16T16:38:09.161 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-16T16:38:09.161 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T16:38:09.161 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-16T16:38:09.261 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-11-16T16:38:09.261 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-16T16:38:09.308 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-11-16T16:38:09.308 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-16T16:38:09.356 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-16T16:38:09.356 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T16:38:09.356 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/daemon-helper 2023-11-16T16:38:09.427 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-16T16:38:09.538 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-11-16T16:38:09.538 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/usr/bin/daemon-helper 2023-11-16T16:38:09.570 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-16T16:38:09.642 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-11-16T16:38:09.642 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/daemon-helper 2023-11-16T16:38:09.675 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-16T16:38:09.745 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-16T16:38:09.746 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T16:38:09.746 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-16T16:38:09.815 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-16T16:38:09.925 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-11-16T16:38:09.925 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-16T16:38:09.957 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-16T16:38:10.027 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-11-16T16:38:10.027 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-16T16:38:10.060 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-16T16:38:10.131 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-16T16:38:10.132 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T16:38:10.132 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/stdin-killer 2023-11-16T16:38:10.202 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-16T16:38:10.311 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-11-16T16:38:10.312 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/usr/bin/stdin-killer 2023-11-16T16:38:10.344 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-16T16:38:10.415 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-11-16T16:38:10.415 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/stdin-killer 2023-11-16T16:38:10.447 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-16T16:38:10.519 INFO:teuthology.run_tasks:Running task cephadm... 2023-11-16T16:38:10.596 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2023-11-16T16:38:10.596 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:38:10.596 INFO:tasks.cephadm:Cluster fsid is 875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:38:10.596 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-11-16T16:38:10.597 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.2', 'mon.b': '172.21.15.141', 'mon.c': '172.21.15.155'} 2023-11-16T16:38:10.597 INFO:tasks.cephadm:First mon is mon.a on smithi002 2023-11-16T16:38:10.597 INFO:tasks.cephadm:First mgr is x 2023-11-16T16:38:10.597 INFO:tasks.cephadm:Normalizing hostnames... 2023-11-16T16:38:10.597 DEBUG:teuthology.orchestra.run.smithi002:> sudo hostname $(hostname -s) 2023-11-16T16:38:10.669 DEBUG:teuthology.orchestra.run.smithi141:> sudo hostname $(hostname -s) 2023-11-16T16:38:10.701 DEBUG:teuthology.orchestra.run.smithi155:> sudo hostname $(hostname -s) 2023-11-16T16:38:10.733 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-11-16T16:38:10.734 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:38:10.876 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '5e4a743ad7fd1422ebd6c4dc950a21a303279207', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/74186/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-3905-g5e4a743a', 'node_name': '172.21.5.32+adami02', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-3905.g5e4a743a'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-11-16 00:50:05.467524', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-choffman-62793', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2023-11-16T16:38:11.093 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-choffman-62793, sha1 5e4a743ad7fd1422ebd6c4dc950a21a303279207 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:38:11.093 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/x86_64/flavors/default/cephadm 2023-11-16T16:38:11.094 DEBUG:teuthology.orchestra.run.smithi002:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-16T16:38:11.510 INFO:teuthology.orchestra.run.smithi002.stdout:-rw-rw-r--. 1 ubuntu ubuntu 384133 Nov 16 16:38 /home/ubuntu/cephtest/cephadm 2023-11-16T16:38:11.511 DEBUG:teuthology.orchestra.run.smithi141:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-16T16:38:12.051 INFO:teuthology.orchestra.run.smithi141.stdout:-rw-rw-r--. 1 ubuntu ubuntu 384133 Nov 16 16:38 /home/ubuntu/cephtest/cephadm 2023-11-16T16:38:12.052 DEBUG:teuthology.orchestra.run.smithi155:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-choffman-62793/5e4a743ad7fd1422ebd6c4dc950a21a303279207/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-16T16:38:12.438 INFO:teuthology.orchestra.run.smithi155.stdout:-rw-rw-r--. 1 ubuntu ubuntu 384133 Nov 16 16:38 /home/ubuntu/cephtest/cephadm 2023-11-16T16:38:12.438 DEBUG:teuthology.orchestra.run.smithi002:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-16T16:38:12.490 DEBUG:teuthology.orchestra.run.smithi141:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-16T16:38:12.512 DEBUG:teuthology.orchestra.run.smithi155:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-16T16:38:12.547 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 on all hosts... 2023-11-16T16:38:12.547 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 pull 2023-11-16T16:38:12.552 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 pull 2023-11-16T16:38:12.554 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 pull 2023-11-16T16:38:13.848 INFO:teuthology.orchestra.run.smithi141.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207... 2023-11-16T16:38:14.041 INFO:teuthology.orchestra.run.smithi155.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207... 2023-11-16T16:38:16.883 INFO:teuthology.orchestra.run.smithi002.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207... 2023-11-16T16:38:41.417 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2023-11-16T16:38:41.418 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_version": "ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev)", 2023-11-16T16:38:41.418 INFO:teuthology.orchestra.run.smithi155.stdout: "image_id": "d5b1f852d682af5926771a6e4760786b6147f99518b76fd91b07a1b9e99bfe99", 2023-11-16T16:38:41.418 INFO:teuthology.orchestra.run.smithi155.stdout: "repo_digests": [ 2023-11-16T16:38:41.418 INFO:teuthology.orchestra.run.smithi155.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:209515a30c262c9be4300b2075ceb5fd8912e5a4eccd3c9021b5585705456b42" 2023-11-16T16:38:41.418 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2023-11-16T16:38:41.418 INFO:teuthology.orchestra.run.smithi155.stdout:} 2023-11-16T16:38:51.392 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-11-16T16:38:51.393 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_version": "ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev)", 2023-11-16T16:38:51.393 INFO:teuthology.orchestra.run.smithi141.stdout: "image_id": "d5b1f852d682af5926771a6e4760786b6147f99518b76fd91b07a1b9e99bfe99", 2023-11-16T16:38:51.393 INFO:teuthology.orchestra.run.smithi141.stdout: "repo_digests": [ 2023-11-16T16:38:51.393 INFO:teuthology.orchestra.run.smithi141.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:209515a30c262c9be4300b2075ceb5fd8912e5a4eccd3c9021b5585705456b42" 2023-11-16T16:38:51.393 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-11-16T16:38:51.393 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-11-16T16:39:33.077 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T16:39:33.077 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev)", 2023-11-16T16:39:33.078 INFO:teuthology.orchestra.run.smithi002.stdout: "image_id": "d5b1f852d682af5926771a6e4760786b6147f99518b76fd91b07a1b9e99bfe99", 2023-11-16T16:39:33.078 INFO:teuthology.orchestra.run.smithi002.stdout: "repo_digests": [ 2023-11-16T16:39:33.078 INFO:teuthology.orchestra.run.smithi002.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:209515a30c262c9be4300b2075ceb5fd8912e5a4eccd3c9021b5585705456b42" 2023-11-16T16:39:33.078 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T16:39:33.078 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T16:39:33.104 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /etc/ceph 2023-11-16T16:39:33.179 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /etc/ceph 2023-11-16T16:39:33.216 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /etc/ceph 2023-11-16T16:39:33.254 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 777 /etc/ceph 2023-11-16T16:39:33.324 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 777 /etc/ceph 2023-11-16T16:39:33.354 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 777 /etc/ceph 2023-11-16T16:39:33.385 INFO:tasks.cephadm:Writing seed config... 2023-11-16T16:39:33.386 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-11-16T16:39:33.386 INFO:tasks.cephadm: override: [client] debug client = 20 2023-11-16T16:39:33.386 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-11-16T16:39:33.386 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-11-16T16:39:33.386 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-11-16T16:39:33.386 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2023-11-16T16:39:33.386 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-11-16T16:39:33.387 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-11-16T16:39:33.387 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-11-16T16:39:33.387 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-11-16T16:39:33.387 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-11-16T16:39:33.387 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-11-16T16:39:33.387 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-11-16T16:39:33.387 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-11-16T16:39:33.387 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-11-16T16:39:33.387 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-11-16T16:39:33.387 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-11-16T16:39:33.387 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2023-11-16T16:39:33.388 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-11-16T16:39:33.388 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-11-16T16:39:33.388 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-11-16T16:39:33.388 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-11-16T16:39:33.388 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2023-11-16T16:39:33.388 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-11-16T16:39:33.388 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-11-16T16:39:33.388 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-11-16T16:39:33.388 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-11-16T16:39:33.388 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-11-16T16:39:33.389 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-11-16T16:39:33.389 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-11-16T16:39:33.389 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2023-11-16T16:39:33.389 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-11-16T16:39:33.389 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2023-11-16T16:39:33.389 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2023-11-16T16:39:33.389 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-11-16T16:39:33.389 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-11-16T16:39:33.389 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2023-11-16T16:39:33.389 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-11-16T16:39:33.390 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-11-16T16:39:33.390 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-11-16T16:39:33.390 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-11-16T16:39:33.390 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-11-16T16:39:33.390 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-11-16T16:39:33.390 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-11-16T16:39:33.390 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2023-11-16T16:39:33.391 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T16:39:33.391 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-11-16T16:39:33.434 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 875fcd24-849e-11ee-8dbe-212e2dc638e7 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-11-16T16:39:33.435 DEBUG:teuthology.orchestra.run.smithi002:mon.a> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mon.a.service 2023-11-16T16:39:33.480 DEBUG:teuthology.orchestra.run.smithi002:mgr.x> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mgr.x.service 2023-11-16T16:39:33.524 INFO:tasks.cephadm:Bootstrapping... 2023-11-16T16:39:33.524 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 -v bootstrap --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.2 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-11-16T16:39:33.547 INFO:journalctl@ceph.mon.a.smithi002.stdout:-- Logs begin at Thu 2023-11-16 16:19:53 UTC. -- 2023-11-16T16:39:33.594 INFO:journalctl@ceph.mgr.x.smithi002.stdout:-- Logs begin at Thu 2023-11-16 16:19:53 UTC. -- 2023-11-16T16:39:33.875 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2023-11-16T16:39:33.876 INFO:teuthology.orchestra.run.smithi002.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207', '-v', 'bootstrap', '--fsid', '875fcd24-849e-11ee-8dbe-212e2dc638e7', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.2', '--skip-admin-label'] 2023-11-16T16:39:34.051 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout 4.1.1 2023-11-16T16:39:34.052 INFO:teuthology.orchestra.run.smithi002.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-11-16T16:39:34.052 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying podman|docker is present... 2023-11-16T16:39:34.226 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout 4.1.1 2023-11-16T16:39:34.227 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying lvm2 is present... 2023-11-16T16:39:34.227 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying time synchronization is in place... 2023-11-16T16:39:34.241 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-11-16T16:39:34.242 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-11-16T16:39:34.255 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-11-16T16:39:34.255 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2023-11-16T16:39:34.269 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout enabled 2023-11-16T16:39:34.282 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout active 2023-11-16T16:39:34.283 INFO:teuthology.orchestra.run.smithi002.stdout:Unit chronyd.service is enabled and running 2023-11-16T16:39:34.283 INFO:teuthology.orchestra.run.smithi002.stdout:Repeating the final host check... 2023-11-16T16:39:34.359 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout 4.1.1 2023-11-16T16:39:34.359 INFO:teuthology.orchestra.run.smithi002.stdout:podman (/bin/podman) version 4.1.1 is present 2023-11-16T16:39:34.359 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl is present 2023-11-16T16:39:34.359 INFO:teuthology.orchestra.run.smithi002.stdout:lvcreate is present 2023-11-16T16:39:34.372 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-11-16T16:39:34.372 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-11-16T16:39:34.385 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-11-16T16:39:34.386 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2023-11-16T16:39:34.399 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout enabled 2023-11-16T16:39:34.413 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout active 2023-11-16T16:39:34.413 INFO:teuthology.orchestra.run.smithi002.stdout:Unit chronyd.service is enabled and running 2023-11-16T16:39:34.413 INFO:teuthology.orchestra.run.smithi002.stdout:Host looks OK 2023-11-16T16:39:34.413 INFO:teuthology.orchestra.run.smithi002.stdout:Cluster fsid: 875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:39:34.413 INFO:teuthology.orchestra.run.smithi002.stdout:Acquiring lock 140447489075688 on /run/cephadm/875fcd24-849e-11ee-8dbe-212e2dc638e7.lock 2023-11-16T16:39:34.414 INFO:teuthology.orchestra.run.smithi002.stdout:Lock 140447489075688 acquired on /run/cephadm/875fcd24-849e-11ee-8dbe-212e2dc638e7.lock 2023-11-16T16:39:34.414 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying IP 172.21.15.2 port 3300 ... 2023-11-16T16:39:34.415 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying IP 172.21.15.2 port 6789 ... 2023-11-16T16:39:34.415 INFO:teuthology.orchestra.run.smithi002.stdout:Base mon IP(s) is [172.21.15.2:3300, 172.21.15.2:6789], mon addrv is [v2:172.21.15.2:3300,v1:172.21.15.2:6789] 2023-11-16T16:39:34.422 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2023-11-16T16:39:34.422 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.2 metric 100 2023-11-16T16:39:34.429 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-11-16T16:39:34.429 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-11-16T16:39:34.430 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1696sec hoplimit 64 pref medium 2023-11-16T16:39:34.436 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-11-16T16:39:34.436 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-11-16T16:39:34.436 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-11-16T16:39:34.437 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2023-11-16T16:39:34.437 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1920/64 scope link 2023-11-16T16:39:34.437 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-11-16T16:39:34.437 INFO:teuthology.orchestra.run.smithi002.stdout:Mon IP `172.21.15.2` is in CIDR network `172.21.0.0/20` 2023-11-16T16:39:34.438 INFO:teuthology.orchestra.run.smithi002.stdout:Mon IP `172.21.15.2` is in CIDR network `172.21.0.0/20` 2023-11-16T16:39:34.438 INFO:teuthology.orchestra.run.smithi002.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-11-16T16:39:34.438 INFO:teuthology.orchestra.run.smithi002.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-11-16T16:39:34.439 INFO:teuthology.orchestra.run.smithi002.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207... 2023-11-16T16:39:34.915 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout d5b1f852d682af5926771a6e4760786b6147f99518b76fd91b07a1b9e99bfe99 2023-11-16T16:39:34.915 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207... 2023-11-16T16:39:34.915 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Getting image source signatures 2023-11-16T16:39:34.915 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Copying blob sha256:3a8f6e475435f9e3b5d92795b454ceb30085f6221d261249c26042723227dea7 2023-11-16T16:39:34.915 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Copying blob sha256:837e66e3ff89804ed1bb42786f80e6c23534ff65fe48526d5007449f4e1f79b0 2023-11-16T16:39:34.915 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Copying config sha256:d5b1f852d682af5926771a6e4760786b6147f99518b76fd91b07a1b9e99bfe99 2023-11-16T16:39:34.915 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Writing manifest to image destination 2023-11-16T16:39:34.916 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Storing signatures 2023-11-16T16:39:35.894 INFO:teuthology.orchestra.run.smithi002.stdout:ceph: stdout ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev) 2023-11-16T16:39:35.895 INFO:teuthology.orchestra.run.smithi002.stdout:Ceph version: ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev) 2023-11-16T16:39:35.895 INFO:teuthology.orchestra.run.smithi002.stdout:Extracting ceph user uid/gid from container image... 2023-11-16T16:39:36.772 INFO:teuthology.orchestra.run.smithi002.stdout:stat: stdout 167 167 2023-11-16T16:39:36.772 INFO:teuthology.orchestra.run.smithi002.stdout:Creating initial keys... 2023-11-16T16:39:37.651 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph-authtool: stdout AQDJRVZl2RDPCxAAWcdHSzo/hEB1fjHDJEp9mg== 2023-11-16T16:39:38.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph-authtool: stdout AQDKRVZlwsbhAhAAJWSNNhlFliwVX1Ql2Btp7A== 2023-11-16T16:39:39.305 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph-authtool: stdout AQDKRVZlV1z/NhAACsNLyslr2o5Mb6YYzvYuDw== 2023-11-16T16:39:39.306 INFO:teuthology.orchestra.run.smithi002.stdout:Creating initial monmap... 2023-11-16T16:39:40.186 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-16T16:39:40.186 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-11-16T16:39:40.186 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:39:40.186 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-16T16:39:40.187 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool for a [v2:172.21.15.2:3300,v1:172.21.15.2:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-16T16:39:40.187 INFO:teuthology.orchestra.run.smithi002.stdout:setting min_mon_release = pacific 2023-11-16T16:39:40.187 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: set fsid to 875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:39:40.187 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-16T16:39:40.187 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:39:40.187 INFO:teuthology.orchestra.run.smithi002.stdout:Creating mon... 2023-11-16T16:39:41.065 INFO:teuthology.orchestra.run.smithi002.stdout:create mon.a on 2023-11-16T16:39:41.294 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-11-16T16:39:41.501 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-11-16T16:39:41.723 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7.target. 2023-11-16T16:39:41.723 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7.target. 2023-11-16T16:39:42.026 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mon.a 2023-11-16T16:39:42.026 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to reset failed state of unit ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mon.a.service: Unit ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mon.a.service not loaded. 2023-11-16T16:39:42.232 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7.target.wants/ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mon.a.service → /etc/systemd/system/ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@.service. 2023-11-16T16:39:42.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:42 smithi002 systemd[1]: Starting Ceph mon.a for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:39:43.033 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:42 smithi002 podman[128168]: 2023-11-16T16:39:43.033 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:42 smithi002 bash[128168]: dacd39a8bef8b7daab2ed2a78a45df4132100d7cf57f44a8d84fb3356139f0af 2023-11-16T16:39:43.058 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-16T16:39:43.058 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2023-11-16T16:39:43.074 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-16T16:39:43.075 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2023-11-16T16:39:43.075 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2023-11-16T16:39:43.075 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to enable service . firewalld.service is not available 2023-11-16T16:39:43.075 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mon to start... 2023-11-16T16:39:43.076 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mon... 2023-11-16T16:39:43.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:43 smithi002 ceph-mon[128208]: mkfs 875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:39:43.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:43 smithi002 systemd[1]: Started Ceph mon.a for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:39:43.358 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:43 smithi002 ceph-mon[128208]: mon.a is new leader, mons a in quorum (ranks 0) 2023-11-16T16:39:44.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout cluster: 2023-11-16T16:39:44.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout id: 875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:39:44.169 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-11-16T16:39:44.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T16:39:44.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout services: 2023-11-16T16:39:44.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.690458s) 2023-11-16T16:39:44.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-11-16T16:39:44.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-11-16T16:39:44.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T16:39:44.170 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout data: 2023-11-16T16:39:44.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-11-16T16:39:44.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-11-16T16:39:44.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-11-16T16:39:44.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout pgs: 2023-11-16T16:39:44.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T16:39:44.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.631+0000 7f8c8a70b700 1 Processor -- start 2023-11-16T16:39:44.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.631+0000 7f8c8a70b700 1 -- start start 2023-11-16T16:39:44.171 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.632+0000 7f8c8a70b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c8407be70 0x7f8c8407a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:44.172 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.632+0000 7f8c8a70b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8407a8a0 con 0x7f8c8407be70 2023-11-16T16:39:44.172 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.633+0000 7f8c83fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c8407be70 0x7f8c8407a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:44.172 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.633+0000 7f8c83fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c8407be70 0x7f8c8407a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32980/0 (socket says 172.21.15.2:32980) 2023-11-16T16:39:44.172 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.633+0000 7f8c83fff700 1 -- 172.21.15.2:0/1362419145 learned_addr learned my addr 172.21.15.2:0/1362419145 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:39:44.172 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.633+0000 7f8c83fff700 1 -- 172.21.15.2:0/1362419145 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8407a9e0 con 0x7f8c8407be70 2023-11-16T16:39:44.172 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.633+0000 7f8c83fff700 1 --2- 172.21.15.2:0/1362419145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c8407be70 0x7f8c8407a2d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8c740098a0 tx=0x7f8c74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c15d4b02a25d8ec server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:44.172 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.634+0000 7f8c82ffd700 1 -- 172.21.15.2:0/1362419145 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c74004160 con 0x7f8c8407be70 2023-11-16T16:39:44.172 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.634+0000 7f8c82ffd700 1 -- 172.21.15.2:0/1362419145 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c740042c0 con 0x7f8c8407be70 2023-11-16T16:39:44.173 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.634+0000 7f8c82ffd700 1 -- 172.21.15.2:0/1362419145 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c740045a0 con 0x7f8c8407be70 2023-11-16T16:39:44.173 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.634+0000 7f8c8a70b700 1 -- 172.21.15.2:0/1362419145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c8407be70 msgr2=0x7f8c8407a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:44.173 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.634+0000 7f8c8a70b700 1 --2- 172.21.15.2:0/1362419145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c8407be70 0x7f8c8407a2d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8c740098a0 tx=0x7f8c74009bb0 comp rx=0 tx=0).stop 2023-11-16T16:39:44.173 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.634+0000 7f8c8a70b700 1 -- 172.21.15.2:0/1362419145 shutdown_connections 2023-11-16T16:39:44.173 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.634+0000 7f8c8a70b700 1 --2- 172.21.15.2:0/1362419145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c8407be70 0x7f8c8407a2d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:44.173 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.634+0000 7f8c8a70b700 1 -- 172.21.15.2:0/1362419145 >> 172.21.15.2:0/1362419145 conn(0x7f8c84100fd0 msgr2=0x7f8c84103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:44.173 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.634+0000 7f8c8a70b700 1 -- 172.21.15.2:0/1362419145 shutdown_connections 2023-11-16T16:39:44.173 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.634+0000 7f8c8a70b700 1 -- 172.21.15.2:0/1362419145 wait complete. 2023-11-16T16:39:44.174 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.635+0000 7f8c8a70b700 1 Processor -- start 2023-11-16T16:39:44.174 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.636+0000 7f8c8a70b700 1 -- start start 2023-11-16T16:39:44.174 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.636+0000 7f8c8a70b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c8407be70 0x7f8c8419b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:44.174 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.636+0000 7f8c8a70b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8419b6c0 con 0x7f8c8407be70 2023-11-16T16:39:44.174 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.636+0000 7f8c83fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c8407be70 0x7f8c8419b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:44.174 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.636+0000 7f8c83fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c8407be70 0x7f8c8419b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32996/0 (socket says 172.21.15.2:32996) 2023-11-16T16:39:44.175 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.636+0000 7f8c83fff700 1 -- 172.21.15.2:0/3197119453 learned_addr learned my addr 172.21.15.2:0/3197119453 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:39:44.175 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.637+0000 7f8c83fff700 1 -- 172.21.15.2:0/3197119453 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c74009550 con 0x7f8c8407be70 2023-11-16T16:39:44.175 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.637+0000 7f8c83fff700 1 --2- 172.21.15.2:0/3197119453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c8407be70 0x7f8c8419b180 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8c8407b340 tx=0x7f8c74004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:44.175 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.637+0000 7f8c817fa700 1 -- 172.21.15.2:0/3197119453 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c74011630 con 0x7f8c8407be70 2023-11-16T16:39:44.175 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.637+0000 7f8c817fa700 1 -- 172.21.15.2:0/3197119453 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c74011790 con 0x7f8c8407be70 2023-11-16T16:39:44.175 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.637+0000 7f8c817fa700 1 -- 172.21.15.2:0/3197119453 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c74011a30 con 0x7f8c8407be70 2023-11-16T16:39:44.175 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.637+0000 7f8c8a70b700 1 -- 172.21.15.2:0/3197119453 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c8407aee0 con 0x7f8c8407be70 2023-11-16T16:39:44.175 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.637+0000 7f8c8a70b700 1 -- 172.21.15.2:0/3197119453 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8407ad50 con 0x7f8c8407be70 2023-11-16T16:39:44.176 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.637+0000 7f8c817fa700 1 -- 172.21.15.2:0/3197119453 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8c74022070 con 0x7f8c8407be70 2023-11-16T16:39:44.176 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.638+0000 7f8c817fa700 1 -- 172.21.15.2:0/3197119453 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8c7401c450 con 0x7f8c8407be70 2023-11-16T16:39:44.176 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.638+0000 7f8c8a70b700 1 -- 172.21.15.2:0/3197119453 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c84194fb0 con 0x7f8c8407be70 2023-11-16T16:39:44.176 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.641+0000 7f8c817fa700 1 -- 172.21.15.2:0/3197119453 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f8c7401b450 con 0x7f8c8407be70 2023-11-16T16:39:44.176 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.717+0000 7f8c8a70b700 1 -- 172.21.15.2:0/3197119453 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f8c84062120 con 0x7f8c8407be70 2023-11-16T16:39:44.176 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.717+0000 7f8c817fa700 1 -- 172.21.15.2:0/3197119453 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f8c84062120 con 0x7f8c8407be70 2023-11-16T16:39:44.176 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.718+0000 7f8c8a70b700 1 -- 172.21.15.2:0/3197119453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c8407be70 msgr2=0x7f8c8419b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:44.176 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.718+0000 7f8c8a70b700 1 --2- 172.21.15.2:0/3197119453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c8407be70 0x7f8c8419b180 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8c8407b340 tx=0x7f8c74004750 comp rx=0 tx=0).stop 2023-11-16T16:39:44.177 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.718+0000 7f8c8a70b700 1 -- 172.21.15.2:0/3197119453 shutdown_connections 2023-11-16T16:39:44.177 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.718+0000 7f8c8a70b700 1 --2- 172.21.15.2:0/3197119453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c8407be70 0x7f8c8419b180 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:44.177 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.718+0000 7f8c8a70b700 1 -- 172.21.15.2:0/3197119453 >> 172.21.15.2:0/3197119453 conn(0x7f8c84100fd0 msgr2=0x7f8c841029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:44.177 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.718+0000 7f8c8a70b700 1 -- 172.21.15.2:0/3197119453 shutdown_connections 2023-11-16T16:39:44.177 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:43.718+0000 7f8c8a70b700 1 -- 172.21.15.2:0/3197119453 wait complete. 2023-11-16T16:39:44.177 INFO:teuthology.orchestra.run.smithi002.stdout:mon is available 2023-11-16T16:39:44.177 INFO:teuthology.orchestra.run.smithi002.stdout:Assimilating anything we can from ceph.conf... 2023-11-16T16:39:44.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:44 smithi002 ceph-mon[128208]: mon.a is new leader, mons a in quorum (ranks 0) 2023-11-16T16:39:44.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:44 smithi002 ceph-mon[128208]: monmap e1: 1 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0]} removed_ranks: {} 2023-11-16T16:39:44.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:44 smithi002 ceph-mon[128208]: fsmap 2023-11-16T16:39:44.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:44 smithi002 ceph-mon[128208]: osdmap e1: 0 total, 0 up, 0 in 2023-11-16T16:39:44.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:44 smithi002 ceph-mon[128208]: mgrmap e1: no daemons active 2023-11-16T16:39:44.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:44 smithi002 ceph-mon[128208]: from='client.? 172.21.15.2:0/3197119453' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-11-16T16:39:45.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:45 smithi002 ceph-mon[128208]: from='client.? 172.21.15.2:0/2308651112' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-16T16:39:45.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:45 smithi002 ceph-mon[128208]: from='client.? 172.21.15.2:0/2308651112' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-16T16:39:45.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T16:39:45.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [global] 2023-11-16T16:39:45.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout fsid = 875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:39:45.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.2:3300,v1:172.21.15.2:6789] 2023-11-16T16:39:45.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-11-16T16:39:45.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-11-16T16:39:45.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-11-16T16:39:45.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-11-16T16:39:45.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T16:39:45.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [mgr] 2023-11-16T16:39:45.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-11-16T16:39:45.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T16:39:45.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [osd] 2023-11-16T16:39:45.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-11-16T16:39:45.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-11-16T16:39:45.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-11-16T16:39:45.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-11-16T16:39:45.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-11-16T16:39:45.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.754+0000 7f0a9c74e700 1 Processor -- start 2023-11-16T16:39:45.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.755+0000 7f0a9c74e700 1 -- start start 2023-11-16T16:39:45.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.755+0000 7f0a9c74e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a94105a20 0x7f0a94105e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:45.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.755+0000 7f0a9c74e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a94106400 con 0x7f0a94105a20 2023-11-16T16:39:45.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.756+0000 7f0a9a4ea700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a94105a20 0x7f0a94105e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:45.354 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.756+0000 7f0a9a4ea700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a94105a20 0x7f0a94105e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38708/0 (socket says 172.21.15.2:38708) 2023-11-16T16:39:45.354 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.756+0000 7f0a9a4ea700 1 -- 172.21.15.2:0/1984544234 learned_addr learned my addr 172.21.15.2:0/1984544234 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:39:45.354 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.756+0000 7f0a9a4ea700 1 -- 172.21.15.2:0/1984544234 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a94106c20 con 0x7f0a94105a20 2023-11-16T16:39:45.354 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.756+0000 7f0a9a4ea700 1 --2- 172.21.15.2:0/1984544234 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a94105a20 0x7f0a94105e30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0a900098a0 tx=0x7f0a90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec66b57e6493f6b1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:45.354 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.757+0000 7f0a994e8700 1 -- 172.21.15.2:0/1984544234 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a90004160 con 0x7f0a94105a20 2023-11-16T16:39:45.354 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.757+0000 7f0a994e8700 1 -- 172.21.15.2:0/1984544234 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a900042c0 con 0x7f0a94105a20 2023-11-16T16:39:45.354 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.757+0000 7f0a994e8700 1 -- 172.21.15.2:0/1984544234 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a900045a0 con 0x7f0a94105a20 2023-11-16T16:39:45.354 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.757+0000 7f0a9c74e700 1 -- 172.21.15.2:0/1984544234 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a94105a20 msgr2=0x7f0a94105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:45.355 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.757+0000 7f0a9c74e700 1 --2- 172.21.15.2:0/1984544234 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a94105a20 0x7f0a94105e30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0a900098a0 tx=0x7f0a90009bb0 comp rx=0 tx=0).stop 2023-11-16T16:39:45.355 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.757+0000 7f0a9c74e700 1 -- 172.21.15.2:0/1984544234 shutdown_connections 2023-11-16T16:39:45.355 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.757+0000 7f0a9c74e700 1 --2- 172.21.15.2:0/1984544234 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a94105a20 0x7f0a94105e30 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:45.355 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.757+0000 7f0a9c74e700 1 -- 172.21.15.2:0/1984544234 >> 172.21.15.2:0/1984544234 conn(0x7f0a94100fd0 msgr2=0x7f0a94103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:45.355 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.757+0000 7f0a9c74e700 1 -- 172.21.15.2:0/1984544234 shutdown_connections 2023-11-16T16:39:45.355 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.757+0000 7f0a9c74e700 1 -- 172.21.15.2:0/1984544234 wait complete. 2023-11-16T16:39:45.355 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.758+0000 7f0a9c74e700 1 Processor -- start 2023-11-16T16:39:45.355 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.759+0000 7f0a9c74e700 1 -- start start 2023-11-16T16:39:45.356 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.759+0000 7f0a9c74e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a94105a20 0x7f0a9407bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:45.356 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.759+0000 7f0a9c74e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9407c140 con 0x7f0a94105a20 2023-11-16T16:39:45.356 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.759+0000 7f0a9a4ea700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a94105a20 0x7f0a9407bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:45.356 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.759+0000 7f0a9a4ea700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a94105a20 0x7f0a9407bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38722/0 (socket says 172.21.15.2:38722) 2023-11-16T16:39:45.356 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.759+0000 7f0a9a4ea700 1 -- 172.21.15.2:0/2308651112 learned_addr learned my addr 172.21.15.2:0/2308651112 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:39:45.356 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.759+0000 7f0a9a4ea700 1 -- 172.21.15.2:0/2308651112 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a90009550 con 0x7f0a94105a20 2023-11-16T16:39:45.356 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.760+0000 7f0a9a4ea700 1 --2- 172.21.15.2:0/2308651112 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a94105a20 0x7f0a9407bc00 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0a90004000 tx=0x7f0a90004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:45.356 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.760+0000 7f0a8b7fe700 1 -- 172.21.15.2:0/2308651112 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a90011630 con 0x7f0a94105a20 2023-11-16T16:39:45.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.760+0000 7f0a8b7fe700 1 -- 172.21.15.2:0/2308651112 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a90011790 con 0x7f0a94105a20 2023-11-16T16:39:45.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.760+0000 7f0a8b7fe700 1 -- 172.21.15.2:0/2308651112 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a90011a30 con 0x7f0a94105a20 2023-11-16T16:39:45.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.760+0000 7f0a9c74e700 1 -- 172.21.15.2:0/2308651112 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a94107120 con 0x7f0a94105a20 2023-11-16T16:39:45.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.760+0000 7f0a9c74e700 1 -- 172.21.15.2:0/2308651112 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a94106f90 con 0x7f0a94105a20 2023-11-16T16:39:45.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.760+0000 7f0a8b7fe700 1 -- 172.21.15.2:0/2308651112 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0a90022070 con 0x7f0a94105a20 2023-11-16T16:39:45.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.760+0000 7f0a8b7fe700 1 -- 172.21.15.2:0/2308651112 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0a9001c450 con 0x7f0a94105a20 2023-11-16T16:39:45.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.761+0000 7f0a9c74e700 1 -- 172.21.15.2:0/2308651112 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a94062120 con 0x7f0a94105a20 2023-11-16T16:39:45.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.764+0000 7f0a8b7fe700 1 -- 172.21.15.2:0/2308651112 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f0a9001b450 con 0x7f0a94105a20 2023-11-16T16:39:45.357 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.836+0000 7f0a9c74e700 1 -- 172.21.15.2:0/2308651112 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f0a941925a0 con 0x7f0a94105a20 2023-11-16T16:39:45.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.841+0000 7f0a8b7fe700 1 -- 172.21.15.2:0/2308651112 <== mon.0 v2:172.21.15.2:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a9001b5f0 con 0x7f0a94105a20 2023-11-16T16:39:45.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.841+0000 7f0a8b7fe700 1 -- 172.21.15.2:0/2308651112 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+427 (secure 0 0 0) 0x7f0a90024790 con 0x7f0a94105a20 2023-11-16T16:39:45.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.842+0000 7f0a9c74e700 1 -- 172.21.15.2:0/2308651112 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a94105a20 msgr2=0x7f0a9407bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:45.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.842+0000 7f0a9c74e700 1 --2- 172.21.15.2:0/2308651112 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a94105a20 0x7f0a9407bc00 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0a90004000 tx=0x7f0a90004750 comp rx=0 tx=0).stop 2023-11-16T16:39:45.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.842+0000 7f0a9c74e700 1 -- 172.21.15.2:0/2308651112 shutdown_connections 2023-11-16T16:39:45.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.842+0000 7f0a9c74e700 1 --2- 172.21.15.2:0/2308651112 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a94105a20 0x7f0a9407bc00 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:45.358 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.842+0000 7f0a9c74e700 1 -- 172.21.15.2:0/2308651112 >> 172.21.15.2:0/2308651112 conn(0x7f0a94100fd0 msgr2=0x7f0a9418d400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:45.359 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.842+0000 7f0a9c74e700 1 -- 172.21.15.2:0/2308651112 shutdown_connections 2023-11-16T16:39:45.359 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:44.842+0000 7f0a9c74e700 1 -- 172.21.15.2:0/2308651112 wait complete. 2023-11-16T16:39:45.359 INFO:teuthology.orchestra.run.smithi002.stdout:Generating new minimal ceph.conf... 2023-11-16T16:39:46.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:46 smithi002 ceph-mon[128208]: from='client.? 172.21.15.2:0/2674350944' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:39:46.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.888+0000 7fd6d3d01700 1 Processor -- start 2023-11-16T16:39:46.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.888+0000 7fd6d3d01700 1 -- start start 2023-11-16T16:39:46.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.889+0000 7fd6d3d01700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc07be30 0x7fd6cc07a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:46.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.889+0000 7fd6d3d01700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc07a860 con 0x7fd6cc07be30 2023-11-16T16:39:46.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.889+0000 7fd6d1a9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc07be30 0x7fd6cc07a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:46.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.889+0000 7fd6d1a9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc07be30 0x7fd6cc07a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38738/0 (socket says 172.21.15.2:38738) 2023-11-16T16:39:46.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.889+0000 7fd6d1a9d700 1 -- 172.21.15.2:0/4179936670 learned_addr learned my addr 172.21.15.2:0/4179936670 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:39:46.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.889+0000 7fd6d1a9d700 1 -- 172.21.15.2:0/4179936670 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc07a9a0 con 0x7fd6cc07be30 2023-11-16T16:39:46.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.890+0000 7fd6d1a9d700 1 --2- 172.21.15.2:0/4179936670 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc07be30 0x7fd6cc07a290 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80098a0 tx=0x7fd6c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32f1f56e8ab5923 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:46.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.890+0000 7fd6d0a9b700 1 -- 172.21.15.2:0/4179936670 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6c8004160 con 0x7fd6cc07be30 2023-11-16T16:39:46.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.890+0000 7fd6d0a9b700 1 -- 172.21.15.2:0/4179936670 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd6c80042c0 con 0x7fd6cc07be30 2023-11-16T16:39:46.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.890+0000 7fd6d0a9b700 1 -- 172.21.15.2:0/4179936670 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6c8011a00 con 0x7fd6cc07be30 2023-11-16T16:39:46.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.890+0000 7fd6d3d01700 1 -- 172.21.15.2:0/4179936670 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc07be30 msgr2=0x7fd6cc07a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:46.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.890+0000 7fd6d3d01700 1 --2- 172.21.15.2:0/4179936670 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc07be30 0x7fd6cc07a290 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80098a0 tx=0x7fd6c8009bb0 comp rx=0 tx=0).stop 2023-11-16T16:39:46.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.891+0000 7fd6d3d01700 1 -- 172.21.15.2:0/4179936670 shutdown_connections 2023-11-16T16:39:46.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.891+0000 7fd6d3d01700 1 --2- 172.21.15.2:0/4179936670 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc07be30 0x7fd6cc07a290 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:46.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.891+0000 7fd6d3d01700 1 -- 172.21.15.2:0/4179936670 >> 172.21.15.2:0/4179936670 conn(0x7fd6cc100fd0 msgr2=0x7fd6cc103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:46.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.891+0000 7fd6d3d01700 1 -- 172.21.15.2:0/4179936670 shutdown_connections 2023-11-16T16:39:46.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.891+0000 7fd6d3d01700 1 -- 172.21.15.2:0/4179936670 wait complete. 2023-11-16T16:39:46.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.892+0000 7fd6d3d01700 1 Processor -- start 2023-11-16T16:39:46.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.892+0000 7fd6d3d01700 1 -- start start 2023-11-16T16:39:46.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.892+0000 7fd6d3d01700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc07be30 0x7fd6cc19b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:46.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.892+0000 7fd6d3d01700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc19be70 con 0x7fd6cc07be30 2023-11-16T16:39:46.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.892+0000 7fd6d1a9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc07be30 0x7fd6cc19b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:46.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.892+0000 7fd6d1a9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc07be30 0x7fd6cc19b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38750/0 (socket says 172.21.15.2:38750) 2023-11-16T16:39:46.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.892+0000 7fd6d1a9d700 1 -- 172.21.15.2:0/2674350944 learned_addr learned my addr 172.21.15.2:0/2674350944 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:39:46.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.893+0000 7fd6d1a9d700 1 -- 172.21.15.2:0/2674350944 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c8009550 con 0x7fd6cc07be30 2023-11-16T16:39:46.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.893+0000 7fd6d1a9d700 1 --2- 172.21.15.2:0/2674350944 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc07be30 0x7fd6cc19b930 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8006af0 tx=0x7fd6c8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:46.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.893+0000 7fd6c2ffd700 1 -- 172.21.15.2:0/2674350944 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6c8003a40 con 0x7fd6cc07be30 2023-11-16T16:39:46.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.893+0000 7fd6c2ffd700 1 -- 172.21.15.2:0/2674350944 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd6c8003ba0 con 0x7fd6cc07be30 2023-11-16T16:39:46.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.893+0000 7fd6c2ffd700 1 -- 172.21.15.2:0/2674350944 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6c801a6e0 con 0x7fd6cc07be30 2023-11-16T16:39:46.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.893+0000 7fd6d3d01700 1 -- 172.21.15.2:0/2674350944 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6cc10bf40 con 0x7fd6cc07be30 2023-11-16T16:39:46.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.894+0000 7fd6d3d01700 1 -- 172.21.15.2:0/2674350944 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6cc10bdb0 con 0x7fd6cc07be30 2023-11-16T16:39:46.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.894+0000 7fd6c2ffd700 1 -- 172.21.15.2:0/2674350944 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd6c801ab60 con 0x7fd6cc07be30 2023-11-16T16:39:46.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.894+0000 7fd6c2ffd700 1 -- 172.21.15.2:0/2674350944 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd6c8019700 con 0x7fd6cc07be30 2023-11-16T16:39:46.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.895+0000 7fd6d3d01700 1 -- 172.21.15.2:0/2674350944 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6cc10c0d0 con 0x7fd6cc07be30 2023-11-16T16:39:46.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.898+0000 7fd6c2ffd700 1 -- 172.21.15.2:0/2674350944 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7fd6c802dde0 con 0x7fd6cc07be30 2023-11-16T16:39:46.437 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.971+0000 7fd6d3d01700 1 -- 172.21.15.2:0/2674350944 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd6cc19c470 con 0x7fd6cc07be30 2023-11-16T16:39:46.437 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.972+0000 7fd6c2ffd700 1 -- 172.21.15.2:0/2674350944 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+173 (secure 0 0 0) 0x7fd6c8024a60 con 0x7fd6cc07be30 2023-11-16T16:39:46.437 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.972+0000 7fd6d3d01700 1 -- 172.21.15.2:0/2674350944 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc07be30 msgr2=0x7fd6cc19b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:46.437 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.972+0000 7fd6d3d01700 1 --2- 172.21.15.2:0/2674350944 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc07be30 0x7fd6cc19b930 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8006af0 tx=0x7fd6c8011ad0 comp rx=0 tx=0).stop 2023-11-16T16:39:46.437 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.973+0000 7fd6d3d01700 1 -- 172.21.15.2:0/2674350944 shutdown_connections 2023-11-16T16:39:46.437 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.973+0000 7fd6d3d01700 1 --2- 172.21.15.2:0/2674350944 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc07be30 0x7fd6cc19b930 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:46.437 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.973+0000 7fd6d3d01700 1 -- 172.21.15.2:0/2674350944 >> 172.21.15.2:0/2674350944 conn(0x7fd6cc100fd0 msgr2=0x7fd6cc1026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:46.437 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.973+0000 7fd6d3d01700 1 -- 172.21.15.2:0/2674350944 shutdown_connections 2023-11-16T16:39:46.438 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:45.973+0000 7fd6d3d01700 1 -- 172.21.15.2:0/2674350944 wait complete. 2023-11-16T16:39:46.438 INFO:teuthology.orchestra.run.smithi002.stdout:Restarting the monitor... 2023-11-16T16:39:46.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:46 smithi002 systemd[1]: Stopping Ceph mon.a for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:39:47.187 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:46 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mon-a[128184]: 2023-11-16T16:39:46.904+0000 7f34c9c92700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T16:39:47.187 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:46 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mon-a[128184]: 2023-11-16T16:39:46.904+0000 7f34c9c92700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-11-16T16:39:47.453 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:47 smithi002 bash[128544]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mon-a 2023-11-16T16:39:48.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:47 smithi002 systemd[1]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mon.a.service: Succeeded. 2023-11-16T16:39:48.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:47 smithi002 systemd[1]: Stopped Ceph mon.a for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:39:48.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:47 smithi002 systemd[1]: Starting Ceph mon.a for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:39:48.429 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 podman[128672]: 2023-11-16T16:39:48.429 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 bash[128672]: c8cc70f11b4275a496820c0e98c4130ac2ac611491b5cbce8aaa4149cb5c049c 2023-11-16T16:39:48.429 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: set uid:gid to 167:167 (ceph:ceph) 2023-11-16T16:39:48.440 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev), process ceph-mon, pid 7 2023-11-16T16:39:48.440 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: pidfile_write: ignore empty --pid-file 2023-11-16T16:39:48.483 INFO:teuthology.orchestra.run.smithi002.stdout:Setting mon public_network to 172.21.0.0/20 2023-11-16T16:39:48.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: load: jerasure load: lrc 2023-11-16T16:39:48.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: RocksDB version: 7.9.2 2023-11-16T16:39:48.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Git sha 0 2023-11-16T16:39:48.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Compile date 2023-11-15 23:53:54 2023-11-16T16:39:48.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: DB SUMMARY 2023-11-16T16:39:48.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: DB Session ID: VLPLXJP6LST5XVTOVDYK 2023-11-16T16:39:48.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: CURRENT file: CURRENT 2023-11-16T16:39:48.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: IDENTITY file: IDENTITY 2023-11-16T16:39:48.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-11-16T16:39:48.754 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-11-16T16:39:48.754 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 98717 ; 2023-11-16T16:39:48.754 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.error_if_exists: 0 2023-11-16T16:39:48.754 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.create_if_missing: 0 2023-11-16T16:39:48.754 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.paranoid_checks: 1 2023-11-16T16:39:48.754 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.flush_verify_memtable_count: 1 2023-11-16T16:39:48.754 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-11-16T16:39:48.754 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-11-16T16:39:48.754 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.env: 0x55645cc22720 2023-11-16T16:39:48.755 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.fs: PosixFileSystem 2023-11-16T16:39:48.755 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.info_log: 0x55645e4773c0 2023-11-16T16:39:48.755 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_file_opening_threads: 16 2023-11-16T16:39:48.755 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.statistics: (nil) 2023-11-16T16:39:48.755 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.use_fsync: 0 2023-11-16T16:39:48.755 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_log_file_size: 0 2023-11-16T16:39:48.755 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-16T16:39:48.755 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.log_file_time_to_roll: 0 2023-11-16T16:39:48.755 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.keep_log_file_num: 1000 2023-11-16T16:39:48.755 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.recycle_log_file_num: 0 2023-11-16T16:39:48.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.allow_fallocate: 1 2023-11-16T16:39:48.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.allow_mmap_reads: 0 2023-11-16T16:39:48.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.allow_mmap_writes: 0 2023-11-16T16:39:48.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.use_direct_reads: 0 2023-11-16T16:39:48.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-16T16:39:48.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.create_missing_column_families: 0 2023-11-16T16:39:48.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.db_log_dir: 2023-11-16T16:39:48.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.wal_dir: 2023-11-16T16:39:48.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.table_cache_numshardbits: 6 2023-11-16T16:39:48.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.WAL_ttl_seconds: 0 2023-11-16T16:39:48.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.WAL_size_limit_MB: 0 2023-11-16T16:39:48.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-16T16:39:48.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-16T16:39:48.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.is_fd_close_on_exec: 1 2023-11-16T16:39:48.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.advise_random_on_open: 1 2023-11-16T16:39:48.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.db_write_buffer_size: 0 2023-11-16T16:39:48.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.write_buffer_manager: 0x55645d6e83c0 2023-11-16T16:39:48.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-16T16:39:48.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-16T16:39:48.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.use_adaptive_mutex: 0 2023-11-16T16:39:48.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.rate_limiter: (nil) 2023-11-16T16:39:48.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-16T16:39:48.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.wal_recovery_mode: 2 2023-11-16T16:39:48.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.enable_thread_tracking: 0 2023-11-16T16:39:48.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.enable_pipelined_write: 0 2023-11-16T16:39:48.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.unordered_write: 0 2023-11-16T16:39:48.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-16T16:39:48.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-16T16:39:48.758 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-16T16:39:48.759 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-16T16:39:48.759 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.row_cache: None 2023-11-16T16:39:48.759 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.wal_filter: None 2023-11-16T16:39:48.759 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-16T16:39:48.759 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.allow_ingest_behind: 0 2023-11-16T16:39:48.759 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.two_write_queues: 0 2023-11-16T16:39:48.759 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.manual_wal_flush: 0 2023-11-16T16:39:48.759 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.wal_compression: 0 2023-11-16T16:39:48.759 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.atomic_flush: 0 2023-11-16T16:39:48.759 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-16T16:39:48.760 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.persist_stats_to_disk: 0 2023-11-16T16:39:48.760 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.write_dbid_to_manifest: 0 2023-11-16T16:39:48.760 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.log_readahead_size: 0 2023-11-16T16:39:48.760 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-11-16T16:39:48.760 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.best_efforts_recovery: 0 2023-11-16T16:39:48.760 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-11-16T16:39:48.760 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-11-16T16:39:48.760 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.allow_data_in_errors: 0 2023-11-16T16:39:48.760 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.db_host_id: __hostname__ 2023-11-16T16:39:48.761 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.enforce_single_del_contracts: true 2023-11-16T16:39:48.761 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_background_jobs: 2 2023-11-16T16:39:48.761 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_background_compactions: -1 2023-11-16T16:39:48.761 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_subcompactions: 1 2023-11-16T16:39:48.761 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-16T16:39:48.761 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-16T16:39:48.761 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.delayed_write_rate : 16777216 2023-11-16T16:39:48.761 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_total_wal_size: 0 2023-11-16T16:39:48.761 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-16T16:39:48.762 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.stats_dump_period_sec: 600 2023-11-16T16:39:48.762 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.stats_persist_period_sec: 600 2023-11-16T16:39:48.762 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-16T16:39:48.762 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_open_files: -1 2023-11-16T16:39:48.762 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.bytes_per_sync: 0 2023-11-16T16:39:48.762 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.wal_bytes_per_sync: 0 2023-11-16T16:39:48.762 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.strict_bytes_per_sync: 0 2023-11-16T16:39:48.762 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compaction_readahead_size: 0 2023-11-16T16:39:48.762 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_background_flushes: -1 2023-11-16T16:39:48.763 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Compression algorithms supported: 2023-11-16T16:39:48.763 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-16T16:39:48.763 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: kZSTD supported: 0 2023-11-16T16:39:48.763 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: kXpressCompression supported: 0 2023-11-16T16:39:48.763 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: kLZ4HCCompression supported: 1 2023-11-16T16:39:48.763 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: kZlibCompression supported: 1 2023-11-16T16:39:48.763 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: kSnappyCompression supported: 1 2023-11-16T16:39:48.763 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: kLZ4Compression supported: 1 2023-11-16T16:39:48.763 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: kBZip2Compression supported: 0 2023-11-16T16:39:48.763 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Fast CRC32 supported: Supported on x86 2023-11-16T16:39:48.764 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: DMutex implementation: pthread_mutex_t 2023-11-16T16:39:48.764 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-11-16T16:39:48.764 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-11-16T16:39:48.764 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-16T16:39:48.764 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.merge_operator: 2023-11-16T16:39:48.764 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compaction_filter: None 2023-11-16T16:39:48.764 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compaction_filter_factory: None 2023-11-16T16:39:48.764 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.sst_partitioner_factory: None 2023-11-16T16:39:48.764 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.memtable_factory: SkipListFactory 2023-11-16T16:39:48.764 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.table_factory: BlockBasedTable 2023-11-16T16:39:48.765 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55645e4774e0) 2023-11-16T16:39:48.765 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks: 1 2023-11-16T16:39:48.765 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-16T16:39:48.765 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-16T16:39:48.765 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_top_level_index_and_filter: 1 2023-11-16T16:39:48.765 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_type: 0 2023-11-16T16:39:48.765 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_index_type: 0 2023-11-16T16:39:48.765 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_shortening: 1 2023-11-16T16:39:48.765 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_hash_table_util_ratio: 0.750000 2023-11-16T16:39:48.765 INFO:journalctl@ceph.mon.a.smithi002.stdout: checksum: 4 2023-11-16T16:39:48.766 INFO:journalctl@ceph.mon.a.smithi002.stdout: no_block_cache: 0 2023-11-16T16:39:48.766 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache: 0x55645d7891f0 2023-11-16T16:39:48.766 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_name: BinnedLRUCache 2023-11-16T16:39:48.766 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_options: 2023-11-16T16:39:48.766 INFO:journalctl@ceph.mon.a.smithi002.stdout: capacity : 536870912 2023-11-16T16:39:48.766 INFO:journalctl@ceph.mon.a.smithi002.stdout: num_shard_bits : 4 2023-11-16T16:39:48.766 INFO:journalctl@ceph.mon.a.smithi002.stdout: strict_capacity_limit : 0 2023-11-16T16:39:48.766 INFO:journalctl@ceph.mon.a.smithi002.stdout: high_pri_pool_ratio: 0.000 2023-11-16T16:39:48.766 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_compressed: (nil) 2023-11-16T16:39:48.767 INFO:journalctl@ceph.mon.a.smithi002.stdout: persistent_cache: (nil) 2023-11-16T16:39:48.767 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size: 4096 2023-11-16T16:39:48.767 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size_deviation: 10 2023-11-16T16:39:48.767 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_restart_interval: 16 2023-11-16T16:39:48.767 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_block_restart_interval: 1 2023-11-16T16:39:48.767 INFO:journalctl@ceph.mon.a.smithi002.stdout: metadata_block_size: 4096 2023-11-16T16:39:48.767 INFO:journalctl@ceph.mon.a.smithi002.stdout: partition_filters: 0 2023-11-16T16:39:48.767 INFO:journalctl@ceph.mon.a.smithi002.stdout: use_delta_encoding: 1 2023-11-16T16:39:48.767 INFO:journalctl@ceph.mon.a.smithi002.stdout: filter_policy: bloomfilter 2023-11-16T16:39:48.767 INFO:journalctl@ceph.mon.a.smithi002.stdout: whole_key_filtering: 1 2023-11-16T16:39:48.768 INFO:journalctl@ceph.mon.a.smithi002.stdout: verify_compression: 0 2023-11-16T16:39:48.768 INFO:journalctl@ceph.mon.a.smithi002.stdout: read_amp_bytes_per_bit: 0 2023-11-16T16:39:48.768 INFO:journalctl@ceph.mon.a.smithi002.stdout: format_version: 5 2023-11-16T16:39:48.768 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_index_compression: 1 2023-11-16T16:39:48.768 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_align: 0 2023-11-16T16:39:48.768 INFO:journalctl@ceph.mon.a.smithi002.stdout: max_auto_readahead_size: 262144 2023-11-16T16:39:48.768 INFO:journalctl@ceph.mon.a.smithi002.stdout: prepopulate_block_cache: 0 2023-11-16T16:39:48.768 INFO:journalctl@ceph.mon.a.smithi002.stdout: initial_auto_readahead_size: 8192 2023-11-16T16:39:48.768 INFO:journalctl@ceph.mon.a.smithi002.stdout: num_file_reads_for_auto_readahead: 2 2023-11-16T16:39:48.769 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.write_buffer_size: 33554432 2023-11-16T16:39:48.769 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_write_buffer_number: 2 2023-11-16T16:39:48.769 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compression: NoCompression 2023-11-16T16:39:48.769 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.bottommost_compression: Disabled 2023-11-16T16:39:48.769 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.prefix_extractor: nullptr 2023-11-16T16:39:48.769 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-16T16:39:48.769 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.num_levels: 7 2023-11-16T16:39:48.769 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-16T16:39:48.769 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-16T16:39:48.769 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-16T16:39:48.770 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-16T16:39:48.770 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-16T16:39:48.770 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-16T16:39:48.770 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-16T16:39:48.770 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-16T16:39:48.770 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-11-16T16:39:48.770 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-16T16:39:48.770 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-11-16T16:39:48.770 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-11-16T16:39:48.770 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compression_opts.window_bits: -14 2023-11-16T16:39:48.771 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compression_opts.level: 32767 2023-11-16T16:39:48.771 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compression_opts.strategy: 0 2023-11-16T16:39:48.771 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-16T16:39:48.771 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-16T16:39:48.771 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-11-16T16:39:48.771 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-11-16T16:39:48.771 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compression_opts.enabled: false 2023-11-16T16:39:48.771 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-11-16T16:39:48.771 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-16T16:39:48.771 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-16T16:39:48.771 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-16T16:39:48.772 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.target_file_size_base: 67108864 2023-11-16T16:39:48.772 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.target_file_size_multiplier: 1 2023-11-16T16:39:48.772 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-16T16:39:48.772 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-16T16:39:48.772 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-16T16:39:48.772 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-16T16:39:48.772 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-16T16:39:48.772 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-16T16:39:48.772 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-16T16:39:48.772 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-16T16:39:48.773 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-16T16:39:48.773 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-16T16:39:48.773 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-16T16:39:48.773 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-16T16:39:48.773 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-11-16T16:39:48.773 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.arena_block_size: 1048576 2023-11-16T16:39:48.773 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-16T16:39:48.773 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-16T16:39:48.773 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.disable_auto_compactions: 0 2023-11-16T16:39:48.774 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-16T16:39:48.774 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-16T16:39:48.774 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-16T16:39:48.774 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-16T16:39:48.774 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-16T16:39:48.774 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-16T16:39:48.774 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-16T16:39:48.774 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-16T16:39:48.774 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-16T16:39:48.774 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-16T16:39:48.775 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.table_properties_collectors: 2023-11-16T16:39:48.775 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.inplace_update_support: 0 2023-11-16T16:39:48.775 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.inplace_update_num_locks: 10000 2023-11-16T16:39:48.775 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-16T16:39:48.775 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-16T16:39:48.775 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.memtable_huge_page_size: 0 2023-11-16T16:39:48.775 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.bloom_locality: 0 2023-11-16T16:39:48.775 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.max_successive_merges: 0 2023-11-16T16:39:48.775 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.optimize_filters_for_hits: 0 2023-11-16T16:39:48.775 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.paranoid_file_checks: 0 2023-11-16T16:39:48.776 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.force_consistency_checks: 1 2023-11-16T16:39:48.776 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.report_bg_io_stats: 0 2023-11-16T16:39:48.776 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.ttl: 2592000 2023-11-16T16:39:48.776 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.periodic_compaction_seconds: 0 2023-11-16T16:39:48.776 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-11-16T16:39:48.776 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-11-16T16:39:48.776 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.enable_blob_files: false 2023-11-16T16:39:48.776 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.min_blob_size: 0 2023-11-16T16:39:48.776 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.blob_file_size: 268435456 2023-11-16T16:39:48.776 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.blob_compression_type: NoCompression 2023-11-16T16:39:48.776 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.enable_blob_garbage_collection: false 2023-11-16T16:39:48.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-11-16T16:39:48.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-11-16T16:39:48.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-11-16T16:39:48.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.blob_file_starting_level: 0 2023-11-16T16:39:48.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-11-16T16:39:48.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-11-16T16:39:48.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-11-16T16:39:48.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4dc770b3-e535-466e-959f-c58945eeb265 2023-11-16T16:39:48.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700152788437853, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-11-16T16:39:48.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-11-16T16:39:48.778 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700152788439615, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93970, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91969, "index_size": 283, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13658, "raw_average_key_size": 50, "raw_value_size": 84510, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1700152788, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4dc770b3-e535-466e-959f-c58945eeb265", "db_session_id": "VLPLXJP6LST5XVTOVDYK", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-11-16T16:39:48.778 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700152788439765, "job": 1, "event": "recovery_finished"} 2023-11-16T16:39:48.778 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-11-16T16:39:48.778 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-16T16:39:48.778 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55645d826000 2023-11-16T16:39:48.778 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: DB pointer 0x55645d812000 2023-11-16T16:39:48.778 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-11-16T16:39:48.778 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-11-16T16:39:48.778 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** DB Stats ** 2023-11-16T16:39:48.778 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T16:39:48.779 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-16T16:39:48.779 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T16:39:48.779 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T16:39:48.779 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-16T16:39:48.779 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T16:39:48.779 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T16:39:48.779 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2023-11-16T16:39:48.779 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2023-11-16T16:39:48.779 INFO:journalctl@ceph.mon.a.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T16:39:48.779 INFO:journalctl@ceph.mon.a.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-11-16T16:39:48.779 INFO:journalctl@ceph.mon.a.smithi002.stdout: L0 2/0 93.60 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 82.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T16:39:48.780 INFO:journalctl@ceph.mon.a.smithi002.stdout: Sum 2/0 93.60 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 82.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T16:39:48.780 INFO:journalctl@ceph.mon.a.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 82.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T16:39:48.780 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2023-11-16T16:39:48.780 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2023-11-16T16:39:48.780 INFO:journalctl@ceph.mon.a.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T16:39:48.780 INFO:journalctl@ceph.mon.a.smithi002.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T16:39:48.780 INFO:journalctl@ceph.mon.a.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 82.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-16T16:39:48.780 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2023-11-16T16:39:48.780 INFO:journalctl@ceph.mon.a.smithi002.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-11-16T16:39:48.780 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2023-11-16T16:39:48.781 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T16:39:48.781 INFO:journalctl@ceph.mon.a.smithi002.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T16:39:48.781 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T16:39:48.781 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T16:39:48.781 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T16:39:48.781 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2023-11-16T16:39:48.781 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative compaction: 0.00 GB write, 12.35 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T16:39:48.781 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval compaction: 0.00 GB write, 12.35 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T16:39:48.781 INFO:journalctl@ceph.mon.a.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T16:39:48.781 INFO:journalctl@ceph.mon.a.smithi002.stdout: Block cache BinnedLRUCache@0x55645d7891f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2023-11-16T16:39:48.781 INFO:journalctl@ceph.mon.a.smithi002.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-11-16T16:39:48.782 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2023-11-16T16:39:48.782 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2023-11-16T16:39:48.782 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: starting mon.a rank 0 at public addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] at bind addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:39:48.782 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: mon.a@-1(???) e1 preinit fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:39:48.782 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: mon.a@-1(???).mds e1 new map 2023-11-16T16:39:48.782 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: mon.a@-1(???).mds e1 print_map 2023-11-16T16:39:48.782 INFO:journalctl@ceph.mon.a.smithi002.stdout: e1 2023-11-16T16:39:48.782 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-11-16T16:39:48.782 INFO:journalctl@ceph.mon.a.smithi002.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T16:39:48.782 INFO:journalctl@ceph.mon.a.smithi002.stdout: legacy client fscid: -1 2023-11-16T16:39:48.783 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2023-11-16T16:39:48.783 INFO:journalctl@ceph.mon.a.smithi002.stdout: No filesystems configured 2023-11-16T16:39:48.783 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-16T16:39:48.783 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T16:39:48.783 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T16:39:48.783 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T16:39:48.783 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-11-16T16:39:48.783 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: expand_channel_meta expand map: {default=false} 2023-11-16T16:39:48.783 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: expand_channel_meta from 'false' to 'false' 2023-11-16T16:39:48.783 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: expand_channel_meta expanded map: {default=false} 2023-11-16T16:39:48.783 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: expand_channel_meta expand map: {default=info} 2023-11-16T16:39:48.784 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: expand_channel_meta from 'info' to 'info' 2023-11-16T16:39:48.784 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: expand_channel_meta expanded map: {default=info} 2023-11-16T16:39:48.784 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: expand_channel_meta expand map: {default=daemon} 2023-11-16T16:39:48.784 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: expand_channel_meta from 'daemon' to 'daemon' 2023-11-16T16:39:48.784 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: expand_channel_meta expanded map: {default=daemon} 2023-11-16T16:39:48.784 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: expand_channel_meta expand map: {default=debug} 2023-11-16T16:39:48.784 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: expand_channel_meta from 'debug' to 'debug' 2023-11-16T16:39:48.784 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: expand_channel_meta expanded map: {default=debug} 2023-11-16T16:39:48.784 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 systemd[1]: Started Ceph mon.a for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:39:48.784 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: mon.a is new leader, mons a in quorum (ranks 0) 2023-11-16T16:39:48.784 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: monmap e1: 1 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0]} removed_ranks: {} 2023-11-16T16:39:48.785 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: fsmap 2023-11-16T16:39:48.785 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: osdmap e1: 0 total, 0 up, 0 in 2023-11-16T16:39:48.785 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:48 smithi002 ceph-mon[128712]: mgrmap e1: no daemons active 2023-11-16T16:39:49.472 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.026+0000 7f73cd6c0700 1 Processor -- start 2023-11-16T16:39:49.473 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.026+0000 7f73cd6c0700 1 -- start start 2023-11-16T16:39:49.473 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.026+0000 7f73cd6c0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73c8105a20 0x7f73c8105e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:49.473 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.027+0000 7f73cd6c0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c8106400 con 0x7f73c8105a20 2023-11-16T16:39:49.473 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.027+0000 7f73c6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73c8105a20 0x7f73c8105e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:49.473 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.027+0000 7f73c6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73c8105a20 0x7f73c8105e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38752/0 (socket says 172.21.15.2:38752) 2023-11-16T16:39:49.473 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.027+0000 7f73c6ffd700 1 -- 172.21.15.2:0/2398732098 learned_addr learned my addr 172.21.15.2:0/2398732098 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:39:49.473 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.028+0000 7f73c6ffd700 1 -- 172.21.15.2:0/2398732098 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c8106c20 con 0x7f73c8105a20 2023-11-16T16:39:49.474 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.028+0000 7f73c6ffd700 1 --2- 172.21.15.2:0/2398732098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73c8105a20 0x7f73c8105e30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f73b80098a0 tx=0x7f73b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bfcb9f02a359872 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:49.474 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.028+0000 7f73c5ffb700 1 -- 172.21.15.2:0/2398732098 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73b8004160 con 0x7f73c8105a20 2023-11-16T16:39:49.474 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.028+0000 7f73c5ffb700 1 -- 172.21.15.2:0/2398732098 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73b80042c0 con 0x7f73c8105a20 2023-11-16T16:39:49.474 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.028+0000 7f73c5ffb700 1 -- 172.21.15.2:0/2398732098 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73b8011a00 con 0x7f73c8105a20 2023-11-16T16:39:49.474 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.029+0000 7f73cd6c0700 1 -- 172.21.15.2:0/2398732098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73c8105a20 msgr2=0x7f73c8105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:49.474 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.029+0000 7f73cd6c0700 1 --2- 172.21.15.2:0/2398732098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73c8105a20 0x7f73c8105e30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f73b80098a0 tx=0x7f73b8009bb0 comp rx=0 tx=0).stop 2023-11-16T16:39:49.474 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.029+0000 7f73cd6c0700 1 -- 172.21.15.2:0/2398732098 shutdown_connections 2023-11-16T16:39:49.474 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.029+0000 7f73cd6c0700 1 --2- 172.21.15.2:0/2398732098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73c8105a20 0x7f73c8105e30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:49.475 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.029+0000 7f73cd6c0700 1 -- 172.21.15.2:0/2398732098 >> 172.21.15.2:0/2398732098 conn(0x7f73c8100fd0 msgr2=0x7f73c8103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:49.475 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.029+0000 7f73cd6c0700 1 -- 172.21.15.2:0/2398732098 shutdown_connections 2023-11-16T16:39:49.475 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.029+0000 7f73cd6c0700 1 -- 172.21.15.2:0/2398732098 wait complete. 2023-11-16T16:39:49.475 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.030+0000 7f73cd6c0700 1 Processor -- start 2023-11-16T16:39:49.475 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.030+0000 7f73cd6c0700 1 -- start start 2023-11-16T16:39:49.475 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.030+0000 7f73cd6c0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73c8105a20 0x7f73c819bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:49.475 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.030+0000 7f73cd6c0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c819c090 con 0x7f73c8105a20 2023-11-16T16:39:49.475 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.031+0000 7f73c6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73c8105a20 0x7f73c819bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:49.475 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.031+0000 7f73c6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73c8105a20 0x7f73c819bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38766/0 (socket says 172.21.15.2:38766) 2023-11-16T16:39:49.476 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.031+0000 7f73c6ffd700 1 -- 172.21.15.2:0/2604478474 learned_addr learned my addr 172.21.15.2:0/2604478474 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:39:49.476 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.031+0000 7f73c6ffd700 1 -- 172.21.15.2:0/2604478474 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b8009550 con 0x7f73c8105a20 2023-11-16T16:39:49.476 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.031+0000 7f73c6ffd700 1 --2- 172.21.15.2:0/2604478474 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73c8105a20 0x7f73c819bb50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f73b8006af0 tx=0x7f73b8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:49.476 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.031+0000 7f73affff700 1 -- 172.21.15.2:0/2604478474 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73b8003a40 con 0x7f73c8105a20 2023-11-16T16:39:49.476 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.032+0000 7f73affff700 1 -- 172.21.15.2:0/2604478474 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73b8003ba0 con 0x7f73c8105a20 2023-11-16T16:39:49.476 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.032+0000 7f73cd6c0700 1 -- 172.21.15.2:0/2604478474 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c8109e50 con 0x7f73c8105a20 2023-11-16T16:39:49.476 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.032+0000 7f73affff700 1 -- 172.21.15.2:0/2604478474 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73b801a700 con 0x7f73c8105a20 2023-11-16T16:39:49.476 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.032+0000 7f73cd6c0700 1 -- 172.21.15.2:0/2604478474 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c8109fe0 con 0x7f73c8105a20 2023-11-16T16:39:49.476 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.032+0000 7f73affff700 1 -- 172.21.15.2:0/2604478474 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f73b801ac80 con 0x7f73c8105a20 2023-11-16T16:39:49.477 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.032+0000 7f73affff700 1 -- 172.21.15.2:0/2604478474 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f73b8019700 con 0x7f73c8105a20 2023-11-16T16:39:49.477 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.033+0000 7f73cd6c0700 1 -- 172.21.15.2:0/2604478474 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c8107a90 con 0x7f73c8105a20 2023-11-16T16:39:49.477 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.036+0000 7f73affff700 1 -- 172.21.15.2:0/2604478474 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f73c8107a90 con 0x7f73c8105a20 2023-11-16T16:39:49.477 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.108+0000 7f73cd6c0700 1 -- 172.21.15.2:0/2604478474 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f73c8062120 con 0x7f73c8105a20 2023-11-16T16:39:49.477 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.111+0000 7f73affff700 1 -- 172.21.15.2:0/2604478474 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f73b8024ad0 con 0x7f73c8105a20 2023-11-16T16:39:49.477 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.112+0000 7f73cd6c0700 1 -- 172.21.15.2:0/2604478474 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73c8105a20 msgr2=0x7f73c819bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:49.477 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.112+0000 7f73cd6c0700 1 --2- 172.21.15.2:0/2604478474 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73c8105a20 0x7f73c819bb50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f73b8006af0 tx=0x7f73b8011ad0 comp rx=0 tx=0).stop 2023-11-16T16:39:49.478 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.112+0000 7f73cd6c0700 1 -- 172.21.15.2:0/2604478474 shutdown_connections 2023-11-16T16:39:49.478 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.112+0000 7f73cd6c0700 1 --2- 172.21.15.2:0/2604478474 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73c8105a20 0x7f73c819bb50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:49.478 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.112+0000 7f73cd6c0700 1 -- 172.21.15.2:0/2604478474 >> 172.21.15.2:0/2604478474 conn(0x7f73c8100fd0 msgr2=0x7f73c8102640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:49.478 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.112+0000 7f73cd6c0700 1 -- 172.21.15.2:0/2604478474 shutdown_connections 2023-11-16T16:39:49.478 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:49.112+0000 7f73cd6c0700 1 -- 172.21.15.2:0/2604478474 wait complete. 2023-11-16T16:39:49.478 INFO:teuthology.orchestra.run.smithi002.stdout:Wrote config to /etc/ceph/ceph.conf 2023-11-16T16:39:49.478 INFO:teuthology.orchestra.run.smithi002.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-11-16T16:39:49.478 INFO:teuthology.orchestra.run.smithi002.stdout:Creating mgr... 2023-11-16T16:39:49.479 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying port 9283 ... 2023-11-16T16:39:49.479 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying port 8765 ... 2023-11-16T16:39:49.768 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mgr.x 2023-11-16T16:39:49.768 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to reset failed state of unit ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mgr.x.service: Unit ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mgr.x.service not loaded. 2023-11-16T16:39:49.972 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7.target.wants/ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mgr.x.service → /etc/systemd/system/ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@.service. 2023-11-16T16:39:50.113 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:50 smithi002 systemd[1]: Starting Ceph mgr.x for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:39:50.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:50 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2604478474' entity='client.admin' 2023-11-16T16:39:50.781 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:50 smithi002 podman[128943]: 2023-11-16T16:39:50.782 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:50 smithi002 bash[128943]: 98af492427e68573cba515ce29ddf170ee62e047d603e710d4575f10bb9bcd4c 2023-11-16T16:39:50.809 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-16T16:39:50.810 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2023-11-16T16:39:50.829 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-16T16:39:50.829 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2023-11-16T16:39:50.829 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2023-11-16T16:39:50.829 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to enable service . firewalld.service is not available 2023-11-16T16:39:50.851 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-16T16:39:50.851 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2023-11-16T16:39:50.870 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-16T16:39:50.871 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2023-11-16T16:39:50.871 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2023-11-16T16:39:50.871 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-11-16T16:39:50.871 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr to start... 2023-11-16T16:39:50.871 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr... 2023-11-16T16:39:51.042 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:50 smithi002 systemd[1]: Started Ceph mgr.x for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:39:51.042 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:50 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:50.930+0000 7effe3b5d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T16:39:51.042 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:51 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:51.039+0000 7effe3b5d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T16:39:51.406 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:51 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:51.158+0000 7effe3b5d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T16:39:51.406 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:51 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:51.289+0000 7effe3b5d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T16:39:51.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:51 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1807026123' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-16T16:39:51.906 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:51 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:51.533+0000 7effe3b5d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T16:39:52.058 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T16:39:52.058 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2023-11-16T16:39:52.058 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsid": "875fcd24-849e-11ee-8dbe-212e2dc638e7", 2023-11-16T16:39:52.058 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "health": { 2023-11-16T16:39:52.058 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-11-16T16:39:52.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-11-16T16:39:52.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-11-16T16:39:52.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:39:52.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-11-16T16:39:52.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-11-16T16:39:52.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 0 2023-11-16T16:39:52.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T16:39:52.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-11-16T16:39:52.059 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "a" 2023-11-16T16:39:52.060 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T16:39:52.060 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2023-11-16T16:39:52.061 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "monmap": { 2023-11-16T16:39:52.061 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:39:52.061 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-11-16T16:39:52.061 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-11-16T16:39:52.061 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:39:52.061 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-11-16T16:39:52.061 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:39:52.061 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-11-16T16:39:52.061 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-11-16T16:39:52.061 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-11-16T16:39:52.069 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-11-16T16:39:52.069 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-11-16T16:39:52.069 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-11-16T16:39:52.069 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:39:52.069 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-11-16T16:39:52.069 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-11-16T16:39:52.069 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-11-16T16:39:52.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-11-16T16:39:52.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-11-16T16:39:52.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-11-16T16:39:52.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-11-16T16:39:52.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-11-16T16:39:52.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-11-16T16:39:52.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:39:52.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-11-16T16:39:52.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:39:52.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-11-16T16:39:52.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-11-16T16:39:52.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:39:52.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-11-16T16:39:52.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": false, 2023-11-16T16:39:52.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-11-16T16:39:52.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modules": [ 2023-11-16T16:39:52.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "iostat", 2023-11-16T16:39:52.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "nfs", 2023-11-16T16:39:52.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "restful" 2023-11-16T16:39:52.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T16:39:52.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T16:39:52.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:39:52.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-11-16T16:39:52.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:39:52.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modified": "2023-11-16T16:39:43.029922+0000", 2023-11-16T16:39:52.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T16:39:52.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:39:52.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-11-16T16:39:52.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2023-11-16T16:39:52.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.457+0000 7f642b0b4700 1 Processor -- start 2023-11-16T16:39:52.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.457+0000 7f642b0b4700 1 -- start start 2023-11-16T16:39:52.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.457+0000 7f642b0b4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642407be30 0x7f642407a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:52.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.457+0000 7f642b0b4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642407a860 con 0x7f642407be30 2023-11-16T16:39:52.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.458+0000 7f6428e50700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642407be30 0x7f642407a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:52.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.458+0000 7f6428e50700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642407be30 0x7f642407a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38794/0 (socket says 172.21.15.2:38794) 2023-11-16T16:39:52.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.458+0000 7f6428e50700 1 -- 172.21.15.2:0/1708554128 learned_addr learned my addr 172.21.15.2:0/1708554128 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:39:52.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.458+0000 7f6428e50700 1 -- 172.21.15.2:0/1708554128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642407a9a0 con 0x7f642407be30 2023-11-16T16:39:52.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.458+0000 7f6428e50700 1 --2- 172.21.15.2:0/1708554128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642407be30 0x7f642407a290 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f64180098a0 tx=0x7f6418009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67be9146816ec48a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:52.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.459+0000 7f64237fe700 1 -- 172.21.15.2:0/1708554128 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6418004160 con 0x7f642407be30 2023-11-16T16:39:52.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.459+0000 7f64237fe700 1 -- 172.21.15.2:0/1708554128 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64180042c0 con 0x7f642407be30 2023-11-16T16:39:52.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.459+0000 7f64237fe700 1 -- 172.21.15.2:0/1708554128 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6418011a00 con 0x7f642407be30 2023-11-16T16:39:52.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.459+0000 7f642b0b4700 1 -- 172.21.15.2:0/1708554128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642407be30 msgr2=0x7f642407a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:52.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.459+0000 7f642b0b4700 1 --2- 172.21.15.2:0/1708554128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642407be30 0x7f642407a290 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f64180098a0 tx=0x7f6418009bb0 comp rx=0 tx=0).stop 2023-11-16T16:39:52.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.459+0000 7f642b0b4700 1 -- 172.21.15.2:0/1708554128 shutdown_connections 2023-11-16T16:39:52.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.459+0000 7f642b0b4700 1 --2- 172.21.15.2:0/1708554128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642407be30 0x7f642407a290 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:52.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.459+0000 7f642b0b4700 1 -- 172.21.15.2:0/1708554128 >> 172.21.15.2:0/1708554128 conn(0x7f6424100fd0 msgr2=0x7f6424103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:52.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.460+0000 7f642b0b4700 1 -- 172.21.15.2:0/1708554128 shutdown_connections 2023-11-16T16:39:52.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.460+0000 7f642b0b4700 1 -- 172.21.15.2:0/1708554128 wait complete. 2023-11-16T16:39:52.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.461+0000 7f642b0b4700 1 Processor -- start 2023-11-16T16:39:52.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.461+0000 7f642b0b4700 1 -- start start 2023-11-16T16:39:52.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.461+0000 7f642b0b4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642407be30 0x7f642419b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:39:52.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.461+0000 7f642b0b4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642419be70 con 0x7f642407be30 2023-11-16T16:39:52.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.461+0000 7f6428e50700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642407be30 0x7f642419b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:39:52.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.461+0000 7f6428e50700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642407be30 0x7f642419b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38806/0 (socket says 172.21.15.2:38806) 2023-11-16T16:39:52.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.461+0000 7f6428e50700 1 -- 172.21.15.2:0/1807026123 learned_addr learned my addr 172.21.15.2:0/1807026123 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:39:52.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.462+0000 7f6428e50700 1 -- 172.21.15.2:0/1807026123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6418009550 con 0x7f642407be30 2023-11-16T16:39:52.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.462+0000 7f6428e50700 1 --2- 172.21.15.2:0/1807026123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642407be30 0x7f642419b930 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6418006af0 tx=0x7f6418011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:39:52.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.462+0000 7f6421ffb700 1 -- 172.21.15.2:0/1807026123 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6418003a40 con 0x7f642407be30 2023-11-16T16:39:52.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.462+0000 7f6421ffb700 1 -- 172.21.15.2:0/1807026123 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6418003ba0 con 0x7f642407be30 2023-11-16T16:39:52.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.462+0000 7f642b0b4700 1 -- 172.21.15.2:0/1807026123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642410bf40 con 0x7f642407be30 2023-11-16T16:39:52.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.462+0000 7f6421ffb700 1 -- 172.21.15.2:0/1807026123 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f641801a6e0 con 0x7f642407be30 2023-11-16T16:39:52.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.463+0000 7f642b0b4700 1 -- 172.21.15.2:0/1807026123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642410bdb0 con 0x7f642407be30 2023-11-16T16:39:52.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.463+0000 7f6421ffb700 1 -- 172.21.15.2:0/1807026123 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f641801a950 con 0x7f642407be30 2023-11-16T16:39:52.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.463+0000 7f6421ffb700 1 -- 172.21.15.2:0/1807026123 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f641802dd10 con 0x7f642407be30 2023-11-16T16:39:52.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.464+0000 7f642b0b4700 1 -- 172.21.15.2:0/1807026123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6424062120 con 0x7f642407be30 2023-11-16T16:39:52.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.466+0000 7f6421ffb700 1 -- 172.21.15.2:0/1807026123 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f641802d3f0 con 0x7f642407be30 2023-11-16T16:39:52.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.555+0000 7f642b0b4700 1 -- 172.21.15.2:0/1807026123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f642419c3b0 con 0x7f642407be30 2023-11-16T16:39:52.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.555+0000 7f6421ffb700 1 -- 172.21.15.2:0/1807026123 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f6418024a60 con 0x7f642407be30 2023-11-16T16:39:52.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.556+0000 7f642b0b4700 1 -- 172.21.15.2:0/1807026123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642407be30 msgr2=0x7f642419b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:39:52.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.556+0000 7f642b0b4700 1 --2- 172.21.15.2:0/1807026123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642407be30 0x7f642419b930 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6418006af0 tx=0x7f6418011ad0 comp rx=0 tx=0).stop 2023-11-16T16:39:52.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.556+0000 7f642b0b4700 1 -- 172.21.15.2:0/1807026123 shutdown_connections 2023-11-16T16:39:52.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.556+0000 7f642b0b4700 1 --2- 172.21.15.2:0/1807026123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642407be30 0x7f642419b930 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:39:52.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.556+0000 7f642b0b4700 1 -- 172.21.15.2:0/1807026123 >> 172.21.15.2:0/1807026123 conn(0x7f6424100fd0 msgr2=0x7f64241026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:39:52.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.556+0000 7f642b0b4700 1 -- 172.21.15.2:0/1807026123 shutdown_connections 2023-11-16T16:39:52.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:51.556+0000 7f642b0b4700 1 -- 172.21.15.2:0/1807026123 wait complete. 2023-11-16T16:39:52.075 INFO:teuthology.orchestra.run.smithi002.stdout:mgr not available, waiting (1/15)... 2023-11-16T16:39:52.252 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:51 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:51.962+0000 7effe3b5d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T16:39:52.252 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:52 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:52.082+0000 7effe3b5d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T16:39:52.252 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:52 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:52.250+0000 7effe3b5d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T16:39:52.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:52 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:52.378+0000 7effe3b5d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T16:39:53.405 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:52 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:52.942+0000 7effe3b5d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T16:39:54.155 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:53 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:53.751+0000 7effe3b5d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T16:39:54.156 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:53 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:53.965+0000 7effe3b5d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T16:39:55.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:54 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3896446977' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-16T16:39:55.656 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:55 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:54.991+0000 7effe3b5d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:39:55.656 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:55 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:55.115+0000 7effe3b5d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T16:39:55.656 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:55 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:55.345+0000 7effe3b5d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:39:56.034 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:55 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:55.745+0000 7effe3b5d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T16:39:56.406 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:56 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:56.032+0000 7effe3b5d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T16:39:56.406 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:56 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:56.235+0000 7effe3b5d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T16:39:56.905 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:56 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:56.602+0000 7effe3b5d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T16:39:57.906 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:57 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:57.586+0000 7effe3b5d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T16:39:57.906 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:57 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:57.781+0000 7effe3b5d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T16:39:58.405 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:58 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:58.043+0000 7effe3b5d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T16:39:58.405 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:58 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:58.212+0000 7effe3b5d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T16:39:58.905 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:58 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:58.520+0000 7effe3b5d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T16:40:00.156 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:39:59 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:39:59.721+0000 7effe3b5d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T16:40:02.970 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:59 smithi002 ceph-mon[128712]: Activating manager daemon x 2023-11-16T16:40:02.970 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:59 smithi002 ceph-mon[128712]: mgrmap e2: x(active, starting, since 0.00418928s) 2023-11-16T16:40:02.970 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:59 smithi002 ceph-mon[128712]: from='mgr.14100 172.21.15.2:0/2185769504' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T16:40:02.970 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:59 smithi002 ceph-mon[128712]: from='mgr.14100 172.21.15.2:0/2185769504' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T16:40:02.970 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:59 smithi002 ceph-mon[128712]: from='mgr.14100 172.21.15.2:0/2185769504' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T16:40:02.970 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:59 smithi002 ceph-mon[128712]: from='mgr.14100 172.21.15.2:0/2185769504' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:40:02.971 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:59 smithi002 ceph-mon[128712]: from='mgr.14100 172.21.15.2:0/2185769504' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T16:40:02.971 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:59 smithi002 ceph-mon[128712]: Manager daemon x is now available 2023-11-16T16:40:02.971 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:59 smithi002 ceph-mon[128712]: from='mgr.14100 172.21.15.2:0/2185769504' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T16:40:02.971 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:59 smithi002 ceph-mon[128712]: from='mgr.14100 172.21.15.2:0/2185769504' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T16:40:02.971 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:59 smithi002 ceph-mon[128712]: from='mgr.14100 172.21.15.2:0/2185769504' entity='mgr.x' 2023-11-16T16:40:02.971 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:59 smithi002 ceph-mon[128712]: from='mgr.14100 172.21.15.2:0/2185769504' entity='mgr.x' 2023-11-16T16:40:02.971 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:39:59 smithi002 ceph-mon[128712]: from='mgr.14100 172.21.15.2:0/2185769504' entity='mgr.x' 2023-11-16T16:40:02.971 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:01 smithi002 ceph-mon[128712]: mgrmap e3: x(active, since 1.00971s) 2023-11-16T16:40:02.981 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T16:40:02.981 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2023-11-16T16:40:02.981 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsid": "875fcd24-849e-11ee-8dbe-212e2dc638e7", 2023-11-16T16:40:02.982 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "health": { 2023-11-16T16:40:02.982 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-11-16T16:40:02.982 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-11-16T16:40:02.982 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-11-16T16:40:02.982 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:02.982 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-11-16T16:40:02.982 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-11-16T16:40:02.982 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 0 2023-11-16T16:40:02.982 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T16:40:02.982 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-11-16T16:40:02.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "a" 2023-11-16T16:40:02.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T16:40:02.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2023-11-16T16:40:02.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "monmap": { 2023-11-16T16:40:02.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:40:02.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-11-16T16:40:02.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-11-16T16:40:02.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:02.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-11-16T16:40:02.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:40:02.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-11-16T16:40:02.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-11-16T16:40:02.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-11-16T16:40:02.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-11-16T16:40:02.983 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-11-16T16:40:02.984 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-11-16T16:40:02.984 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:02.984 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-11-16T16:40:02.984 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-11-16T16:40:02.984 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-11-16T16:40:02.984 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-11-16T16:40:02.984 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-11-16T16:40:02.984 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-11-16T16:40:02.984 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-11-16T16:40:02.984 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-11-16T16:40:02.984 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-11-16T16:40:02.984 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:02.984 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-11-16T16:40:02.985 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:40:02.985 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-11-16T16:40:02.985 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-11-16T16:40:02.985 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:02.985 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-11-16T16:40:02.985 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": false, 2023-11-16T16:40:02.985 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-11-16T16:40:02.985 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modules": [ 2023-11-16T16:40:02.985 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "iostat", 2023-11-16T16:40:02.985 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "nfs", 2023-11-16T16:40:02.985 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "restful" 2023-11-16T16:40:02.985 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T16:40:02.985 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T16:40:02.986 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:02.986 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-11-16T16:40:02.986 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:40:02.986 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modified": "2023-11-16T16:39:43.029922+0000", 2023-11-16T16:40:02.986 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T16:40:02.986 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:02.986 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-11-16T16:40:02.986 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2023-11-16T16:40:02.986 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.636+0000 7f529e2e3700 1 Processor -- start 2023-11-16T16:40:02.986 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.637+0000 7f529e2e3700 1 -- start start 2023-11-16T16:40:02.986 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.637+0000 7f529e2e3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f529807be30 0x7f529807a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:02.986 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.637+0000 7f529e2e3700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529807a860 con 0x7f529807be30 2023-11-16T16:40:02.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.637+0000 7f5297fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f529807be30 0x7f529807a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:02.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.638+0000 7f5297fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f529807be30 0x7f529807a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32954/0 (socket says 172.21.15.2:32954) 2023-11-16T16:40:02.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.638+0000 7f5297fff700 1 -- 172.21.15.2:0/2270512882 learned_addr learned my addr 172.21.15.2:0/2270512882 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:02.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.638+0000 7f5297fff700 1 -- 172.21.15.2:0/2270512882 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529807a9a0 con 0x7f529807be30 2023-11-16T16:40:02.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.638+0000 7f5297fff700 1 --2- 172.21.15.2:0/2270512882 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f529807be30 0x7f529807a290 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f52880098a0 tx=0x7f5288009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=743920c9d44c97da server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:02.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.638+0000 7f5296ffd700 1 -- 172.21.15.2:0/2270512882 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5288004160 con 0x7f529807be30 2023-11-16T16:40:02.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.638+0000 7f5296ffd700 1 -- 172.21.15.2:0/2270512882 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52880042c0 con 0x7f529807be30 2023-11-16T16:40:02.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.638+0000 7f5296ffd700 1 -- 172.21.15.2:0/2270512882 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5288011a00 con 0x7f529807be30 2023-11-16T16:40:02.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.639+0000 7f529e2e3700 1 -- 172.21.15.2:0/2270512882 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f529807be30 msgr2=0x7f529807a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:02.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.639+0000 7f529e2e3700 1 --2- 172.21.15.2:0/2270512882 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f529807be30 0x7f529807a290 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f52880098a0 tx=0x7f5288009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:02.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.639+0000 7f529e2e3700 1 -- 172.21.15.2:0/2270512882 shutdown_connections 2023-11-16T16:40:02.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.639+0000 7f529e2e3700 1 --2- 172.21.15.2:0/2270512882 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f529807be30 0x7f529807a290 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:02.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.639+0000 7f529e2e3700 1 -- 172.21.15.2:0/2270512882 >> 172.21.15.2:0/2270512882 conn(0x7f5298100fd0 msgr2=0x7f5298103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:02.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.639+0000 7f529e2e3700 1 -- 172.21.15.2:0/2270512882 shutdown_connections 2023-11-16T16:40:02.988 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.640+0000 7f529e2e3700 1 -- 172.21.15.2:0/2270512882 wait complete. 2023-11-16T16:40:02.988 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.641+0000 7f529e2e3700 1 Processor -- start 2023-11-16T16:40:02.988 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.641+0000 7f529e2e3700 1 -- start start 2023-11-16T16:40:02.988 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.642+0000 7f529e2e3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f529807be30 0x7f529819b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:02.988 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.642+0000 7f529e2e3700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529819be70 con 0x7f529807be30 2023-11-16T16:40:02.988 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.642+0000 7f5297fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f529807be30 0x7f529819b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:02.988 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.642+0000 7f5297fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f529807be30 0x7f529819b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32966/0 (socket says 172.21.15.2:32966) 2023-11-16T16:40:02.988 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.642+0000 7f5297fff700 1 -- 172.21.15.2:0/3896446977 learned_addr learned my addr 172.21.15.2:0/3896446977 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:02.988 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.642+0000 7f5297fff700 1 -- 172.21.15.2:0/3896446977 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5288009550 con 0x7f529807be30 2023-11-16T16:40:02.988 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.643+0000 7f5297fff700 1 --2- 172.21.15.2:0/3896446977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f529807be30 0x7f529819b930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5288006af0 tx=0x7f5288011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:02.988 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.643+0000 7f52957fa700 1 -- 172.21.15.2:0/3896446977 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5288003a40 con 0x7f529807be30 2023-11-16T16:40:02.988 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.643+0000 7f52957fa700 1 -- 172.21.15.2:0/3896446977 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5288003ba0 con 0x7f529807be30 2023-11-16T16:40:02.988 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.643+0000 7f529e2e3700 1 -- 172.21.15.2:0/3896446977 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f529810bf40 con 0x7f529807be30 2023-11-16T16:40:02.988 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.643+0000 7f52957fa700 1 -- 172.21.15.2:0/3896446977 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f528801a6e0 con 0x7f529807be30 2023-11-16T16:40:02.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.643+0000 7f529e2e3700 1 -- 172.21.15.2:0/3896446977 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f529810bdb0 con 0x7f529807be30 2023-11-16T16:40:02.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.644+0000 7f52957fa700 1 -- 172.21.15.2:0/3896446977 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f528801a950 con 0x7f529807be30 2023-11-16T16:40:02.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.644+0000 7f52957fa700 1 -- 172.21.15.2:0/3896446977 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f528802ddf0 con 0x7f529807be30 2023-11-16T16:40:02.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.645+0000 7f527effd700 1 -- 172.21.15.2:0/3896446977 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52780052f0 con 0x7f529807be30 2023-11-16T16:40:02.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.647+0000 7f52957fa700 1 -- 172.21.15.2:0/3896446977 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f5288024a60 con 0x7f529807be30 2023-11-16T16:40:02.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.734+0000 7f527effd700 1 -- 172.21.15.2:0/3896446977 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5278005c90 con 0x7f529807be30 2023-11-16T16:40:02.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.735+0000 7f52957fa700 1 -- 172.21.15.2:0/3896446977 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f5288024c00 con 0x7f529807be30 2023-11-16T16:40:02.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.736+0000 7f527effd700 1 -- 172.21.15.2:0/3896446977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f529807be30 msgr2=0x7f529819b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:02.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.736+0000 7f527effd700 1 --2- 172.21.15.2:0/3896446977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f529807be30 0x7f529819b930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5288006af0 tx=0x7f5288011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:02.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.736+0000 7f527effd700 1 -- 172.21.15.2:0/3896446977 shutdown_connections 2023-11-16T16:40:02.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.736+0000 7f527effd700 1 --2- 172.21.15.2:0/3896446977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f529807be30 0x7f529819b930 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:02.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.736+0000 7f527effd700 1 -- 172.21.15.2:0/3896446977 >> 172.21.15.2:0/3896446977 conn(0x7f5298100fd0 msgr2=0x7f52981026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:02.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.736+0000 7f527effd700 1 -- 172.21.15.2:0/3896446977 shutdown_connections 2023-11-16T16:40:02.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:39:54.736+0000 7f527effd700 1 -- 172.21.15.2:0/3896446977 wait complete. 2023-11-16T16:40:02.990 INFO:teuthology.orchestra.run.smithi002.stdout:mgr not available, waiting (2/15)... 2023-11-16T16:40:03.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:02 smithi002 ceph-mon[128712]: mgrmap e4: x(active, since 2s) 2023-11-16T16:40:05.568 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T16:40:05.568 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2023-11-16T16:40:05.568 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsid": "875fcd24-849e-11ee-8dbe-212e2dc638e7", 2023-11-16T16:40:05.568 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "health": { 2023-11-16T16:40:05.568 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-11-16T16:40:05.568 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-11-16T16:40:05.569 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-11-16T16:40:05.569 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:05.569 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-11-16T16:40:05.569 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-11-16T16:40:05.569 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 0 2023-11-16T16:40:05.569 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T16:40:05.569 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-11-16T16:40:05.569 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "a" 2023-11-16T16:40:05.569 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T16:40:05.569 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2023-11-16T16:40:05.569 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "monmap": { 2023-11-16T16:40:05.569 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:40:05.570 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-11-16T16:40:05.570 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-11-16T16:40:05.570 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:05.570 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-11-16T16:40:05.570 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:40:05.570 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-11-16T16:40:05.570 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-11-16T16:40:05.570 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-11-16T16:40:05.570 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-11-16T16:40:05.570 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-11-16T16:40:05.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-11-16T16:40:05.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:05.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-11-16T16:40:05.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-11-16T16:40:05.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-11-16T16:40:05.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-11-16T16:40:05.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-11-16T16:40:05.578 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-11-16T16:40:05.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-11-16T16:40:05.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-11-16T16:40:05.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-11-16T16:40:05.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:05.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-11-16T16:40:05.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:40:05.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-11-16T16:40:05.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-11-16T16:40:05.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:05.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-11-16T16:40:05.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": true, 2023-11-16T16:40:05.579 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-11-16T16:40:05.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modules": [ 2023-11-16T16:40:05.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "iostat", 2023-11-16T16:40:05.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "nfs", 2023-11-16T16:40:05.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "restful" 2023-11-16T16:40:05.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2023-11-16T16:40:05.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T16:40:05.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:05.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-11-16T16:40:05.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-16T16:40:05.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modified": "2023-11-16T16:39:43.029922+0000", 2023-11-16T16:40:05.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-16T16:40:05.580 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2023-11-16T16:40:05.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-11-16T16:40:05.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2023-11-16T16:40:05.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.863+0000 7fcb8d53d700 1 Processor -- start 2023-11-16T16:40:05.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.863+0000 7fcb8d53d700 1 -- start start 2023-11-16T16:40:05.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.863+0000 7fcb8d53d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8807be70 0x7fcb8807a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:05.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.863+0000 7fcb8d53d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8807a8a0 con 0x7fcb8807be70 2023-11-16T16:40:05.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.864+0000 7fcb86ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8807be70 0x7fcb8807a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:05.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.864+0000 7fcb86ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8807be70 0x7fcb8807a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48442/0 (socket says 172.21.15.2:48442) 2023-11-16T16:40:05.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.864+0000 7fcb86ffd700 1 -- 172.21.15.2:0/3430514561 learned_addr learned my addr 172.21.15.2:0/3430514561 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:05.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.864+0000 7fcb86ffd700 1 -- 172.21.15.2:0/3430514561 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8807a9e0 con 0x7fcb8807be70 2023-11-16T16:40:05.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.864+0000 7fcb86ffd700 1 --2- 172.21.15.2:0/3430514561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8807be70 0x7fcb8807a2d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcb780098a0 tx=0x7fcb78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185094184313b32d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:05.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.865+0000 7fcb85ffb700 1 -- 172.21.15.2:0/3430514561 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcb78004160 con 0x7fcb8807be70 2023-11-16T16:40:05.581 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.865+0000 7fcb85ffb700 1 -- 172.21.15.2:0/3430514561 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb780042c0 con 0x7fcb8807be70 2023-11-16T16:40:05.582 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.865+0000 7fcb85ffb700 1 -- 172.21.15.2:0/3430514561 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcb78011a00 con 0x7fcb8807be70 2023-11-16T16:40:05.582 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.865+0000 7fcb8d53d700 1 -- 172.21.15.2:0/3430514561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8807be70 msgr2=0x7fcb8807a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:05.582 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.865+0000 7fcb8d53d700 1 --2- 172.21.15.2:0/3430514561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8807be70 0x7fcb8807a2d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcb780098a0 tx=0x7fcb78009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:05.582 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.865+0000 7fcb8d53d700 1 -- 172.21.15.2:0/3430514561 shutdown_connections 2023-11-16T16:40:05.582 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.865+0000 7fcb8d53d700 1 --2- 172.21.15.2:0/3430514561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8807be70 0x7fcb8807a2d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:05.582 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.865+0000 7fcb8d53d700 1 -- 172.21.15.2:0/3430514561 >> 172.21.15.2:0/3430514561 conn(0x7fcb88100fd0 msgr2=0x7fcb88103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:05.582 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.865+0000 7fcb8d53d700 1 -- 172.21.15.2:0/3430514561 shutdown_connections 2023-11-16T16:40:05.582 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.865+0000 7fcb8d53d700 1 -- 172.21.15.2:0/3430514561 wait complete. 2023-11-16T16:40:05.582 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.866+0000 7fcb8d53d700 1 Processor -- start 2023-11-16T16:40:05.582 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.867+0000 7fcb8d53d700 1 -- start start 2023-11-16T16:40:05.582 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.867+0000 7fcb8d53d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8807be70 0x7fcb8819b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:05.582 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.867+0000 7fcb8d53d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8819bef0 con 0x7fcb8807be70 2023-11-16T16:40:05.583 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.867+0000 7fcb86ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8807be70 0x7fcb8819b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:05.583 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.867+0000 7fcb86ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8807be70 0x7fcb8819b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48454/0 (socket says 172.21.15.2:48454) 2023-11-16T16:40:05.583 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.867+0000 7fcb86ffd700 1 -- 172.21.15.2:0/1771841812 learned_addr learned my addr 172.21.15.2:0/1771841812 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:05.583 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.867+0000 7fcb86ffd700 1 -- 172.21.15.2:0/1771841812 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb78009550 con 0x7fcb8807be70 2023-11-16T16:40:05.583 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.868+0000 7fcb86ffd700 1 --2- 172.21.15.2:0/1771841812 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8807be70 0x7fcb8819b9b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fcb78006af0 tx=0x7fcb78011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:05.583 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.868+0000 7fcb6ffff700 1 -- 172.21.15.2:0/1771841812 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcb78003a40 con 0x7fcb8807be70 2023-11-16T16:40:05.583 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.868+0000 7fcb6ffff700 1 -- 172.21.15.2:0/1771841812 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb78003ba0 con 0x7fcb8807be70 2023-11-16T16:40:05.583 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.868+0000 7fcb6ffff700 1 -- 172.21.15.2:0/1771841812 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcb7801a6e0 con 0x7fcb8807be70 2023-11-16T16:40:05.583 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.868+0000 7fcb8d53d700 1 -- 172.21.15.2:0/1771841812 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb8810bf60 con 0x7fcb8807be70 2023-11-16T16:40:05.583 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.868+0000 7fcb8d53d700 1 -- 172.21.15.2:0/1771841812 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb8810bdd0 con 0x7fcb8807be70 2023-11-16T16:40:05.583 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.869+0000 7fcb6ffff700 1 -- 172.21.15.2:0/1771841812 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7fcb78024a60 con 0x7fcb8807be70 2023-11-16T16:40:05.584 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.869+0000 7fcb8d53d700 1 -- 172.21.15.2:0/1771841812 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb88062120 con 0x7fcb8807be70 2023-11-16T16:40:05.584 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.870+0000 7fcb6ffff700 1 --2- 172.21.15.2:0/1771841812 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fcb70038710 0x7fcb7003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:05.584 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.870+0000 7fcb6ffff700 1 -- 172.21.15.2:0/1771841812 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcb7804c6d0 con 0x7fcb8807be70 2023-11-16T16:40:05.584 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.870+0000 7fcb867fc700 1 --2- 172.21.15.2:0/1771841812 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fcb70038710 0x7fcb7003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:05.584 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.871+0000 7fcb867fc700 1 --2- 172.21.15.2:0/1771841812 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fcb70038710 0x7fcb7003abc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c006fd0 tx=0x7fcb7c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:05.584 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:04.876+0000 7fcb6ffff700 1 -- 172.21.15.2:0/1771841812 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcb78024430 con 0x7fcb8807be70 2023-11-16T16:40:05.584 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.164+0000 7fcb8d53d700 1 -- 172.21.15.2:0/1771841812 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fcb8819c680 con 0x7fcb8807be70 2023-11-16T16:40:05.584 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.165+0000 7fcb6ffff700 1 -- 172.21.15.2:0/1771841812 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fcb78018350 con 0x7fcb8807be70 2023-11-16T16:40:05.584 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.167+0000 7fcb8d53d700 1 -- 172.21.15.2:0/1771841812 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fcb70038710 msgr2=0x7fcb7003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:05.584 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.167+0000 7fcb8d53d700 1 --2- 172.21.15.2:0/1771841812 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fcb70038710 0x7fcb7003abc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c006fd0 tx=0x7fcb7c006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:05.584 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.167+0000 7fcb8d53d700 1 -- 172.21.15.2:0/1771841812 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8807be70 msgr2=0x7fcb8819b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:05.584 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.167+0000 7fcb8d53d700 1 --2- 172.21.15.2:0/1771841812 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8807be70 0x7fcb8819b9b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fcb78006af0 tx=0x7fcb78011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:05.585 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.167+0000 7fcb8d53d700 1 -- 172.21.15.2:0/1771841812 shutdown_connections 2023-11-16T16:40:05.585 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.167+0000 7fcb8d53d700 1 --2- 172.21.15.2:0/1771841812 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fcb70038710 0x7fcb7003abc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:05.585 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.167+0000 7fcb8d53d700 1 --2- 172.21.15.2:0/1771841812 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8807be70 0x7fcb8819b9b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:05.585 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.167+0000 7fcb8d53d700 1 -- 172.21.15.2:0/1771841812 >> 172.21.15.2:0/1771841812 conn(0x7fcb88100fd0 msgr2=0x7fcb881026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:05.585 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.167+0000 7fcb8d53d700 1 -- 172.21.15.2:0/1771841812 shutdown_connections 2023-11-16T16:40:05.585 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:05.167+0000 7fcb8d53d700 1 -- 172.21.15.2:0/1771841812 wait complete. 2023-11-16T16:40:05.585 INFO:teuthology.orchestra.run.smithi002.stdout:mgr is available 2023-11-16T16:40:05.585 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:05 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1771841812' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-16T16:40:06.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:06 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/525701508' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-16T16:40:06.857 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T16:40:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [global] 2023-11-16T16:40:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout fsid = 875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:40:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-11-16T16:40:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-11-16T16:40:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-11-16T16:40:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-11-16T16:40:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T16:40:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [mgr] 2023-11-16T16:40:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-11-16T16:40:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2023-11-16T16:40:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [osd] 2023-11-16T16:40:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-11-16T16:40:06.859 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-11-16T16:40:06.859 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-11-16T16:40:06.859 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-11-16T16:40:06.859 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-11-16T16:40:06.859 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.119+0000 7f6e06495700 1 Processor -- start 2023-11-16T16:40:06.859 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.119+0000 7f6e06495700 1 -- start start 2023-11-16T16:40:06.859 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.120+0000 7f6e06495700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0007be10 0x7f6e0007a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:06.859 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.120+0000 7f6e06495700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0007a840 con 0x7f6e0007be10 2023-11-16T16:40:06.859 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.120+0000 7f6dfffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0007be10 0x7f6e0007a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:06.859 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.120+0000 7f6dfffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0007be10 0x7f6e0007a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48456/0 (socket says 172.21.15.2:48456) 2023-11-16T16:40:06.859 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.120+0000 7f6dfffff700 1 -- 172.21.15.2:0/3172494265 learned_addr learned my addr 172.21.15.2:0/3172494265 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:06.859 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.121+0000 7f6dfffff700 1 -- 172.21.15.2:0/3172494265 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0007a980 con 0x7f6e0007be10 2023-11-16T16:40:06.860 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.121+0000 7f6dfffff700 1 --2- 172.21.15.2:0/3172494265 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0007be10 0x7f6e0007a270 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6df00098a0 tx=0x7f6df0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78f9da55d2f4e332 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:06.860 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.121+0000 7f6dfeffd700 1 -- 172.21.15.2:0/3172494265 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6df0004160 con 0x7f6e0007be10 2023-11-16T16:40:06.860 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.121+0000 7f6dfeffd700 1 -- 172.21.15.2:0/3172494265 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6df00042c0 con 0x7f6e0007be10 2023-11-16T16:40:06.860 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.121+0000 7f6dfeffd700 1 -- 172.21.15.2:0/3172494265 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6df0011a00 con 0x7f6e0007be10 2023-11-16T16:40:06.860 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.122+0000 7f6e06495700 1 -- 172.21.15.2:0/3172494265 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0007be10 msgr2=0x7f6e0007a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:06.860 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.122+0000 7f6e06495700 1 --2- 172.21.15.2:0/3172494265 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0007be10 0x7f6e0007a270 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6df00098a0 tx=0x7f6df0009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:06.860 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.122+0000 7f6e06495700 1 -- 172.21.15.2:0/3172494265 shutdown_connections 2023-11-16T16:40:06.860 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.122+0000 7f6e06495700 1 --2- 172.21.15.2:0/3172494265 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0007be10 0x7f6e0007a270 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:06.860 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.122+0000 7f6e06495700 1 -- 172.21.15.2:0/3172494265 >> 172.21.15.2:0/3172494265 conn(0x7f6e00100fd0 msgr2=0x7f6e00103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:06.860 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.122+0000 7f6e06495700 1 -- 172.21.15.2:0/3172494265 shutdown_connections 2023-11-16T16:40:06.860 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.122+0000 7f6e06495700 1 -- 172.21.15.2:0/3172494265 wait complete. 2023-11-16T16:40:06.860 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.123+0000 7f6e06495700 1 Processor -- start 2023-11-16T16:40:06.860 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.123+0000 7f6e06495700 1 -- start start 2023-11-16T16:40:06.861 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.123+0000 7f6e06495700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0007be10 0x7f6e0019b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:06.861 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.123+0000 7f6e06495700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0019be90 con 0x7f6e0007be10 2023-11-16T16:40:06.861 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.124+0000 7f6dfffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0007be10 0x7f6e0019b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:06.861 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.124+0000 7f6dfffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0007be10 0x7f6e0019b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48466/0 (socket says 172.21.15.2:48466) 2023-11-16T16:40:06.861 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.124+0000 7f6dfffff700 1 -- 172.21.15.2:0/525701508 learned_addr learned my addr 172.21.15.2:0/525701508 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:06.861 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.124+0000 7f6dfffff700 1 -- 172.21.15.2:0/525701508 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df0009550 con 0x7f6e0007be10 2023-11-16T16:40:06.861 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.124+0000 7f6dfffff700 1 --2- 172.21.15.2:0/525701508 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0007be10 0x7f6e0019b950 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6df0006af0 tx=0x7f6df0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:06.861 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.124+0000 7f6dfd7fa700 1 -- 172.21.15.2:0/525701508 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6df0003a40 con 0x7f6e0007be10 2023-11-16T16:40:06.861 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.124+0000 7f6dfd7fa700 1 -- 172.21.15.2:0/525701508 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6df0003ba0 con 0x7f6e0007be10 2023-11-16T16:40:06.861 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.124+0000 7f6e06495700 1 -- 172.21.15.2:0/525701508 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0010bf60 con 0x7f6e0007be10 2023-11-16T16:40:06.861 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.125+0000 7f6dfd7fa700 1 -- 172.21.15.2:0/525701508 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6df001a6e0 con 0x7f6e0007be10 2023-11-16T16:40:06.861 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.125+0000 7f6e06495700 1 -- 172.21.15.2:0/525701508 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0010bdd0 con 0x7f6e0007be10 2023-11-16T16:40:06.861 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.126+0000 7f6dfd7fa700 1 -- 172.21.15.2:0/525701508 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f6df0024a60 con 0x7f6e0007be10 2023-11-16T16:40:06.861 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.126+0000 7f6e06495700 1 -- 172.21.15.2:0/525701508 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e00062120 con 0x7f6e0007be10 2023-11-16T16:40:06.862 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.126+0000 7f6dfd7fa700 1 --2- 172.21.15.2:0/525701508 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7f6de8038710 0x7f6de803abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:06.862 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.126+0000 7f6dfd7fa700 1 -- 172.21.15.2:0/525701508 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6df004c690 con 0x7f6e0007be10 2023-11-16T16:40:06.862 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.126+0000 7f6dff7fe700 1 --2- 172.21.15.2:0/525701508 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7f6de8038710 0x7f6de803abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:06.862 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.127+0000 7f6dff7fe700 1 --2- 172.21.15.2:0/525701508 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7f6de8038710 0x7f6de803abc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6df4006fd0 tx=0x7f6df4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:06.862 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.131+0000 7f6dfd7fa700 1 -- 172.21.15.2:0/525701508 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6df0024430 con 0x7f6e0007be10 2023-11-16T16:40:06.862 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.337+0000 7f6e06495700 1 -- 172.21.15.2:0/525701508 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f6e0019c620 con 0x7f6e0007be10 2023-11-16T16:40:06.862 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.338+0000 7f6dfd7fa700 1 -- 172.21.15.2:0/525701508 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f6df0018350 con 0x7f6e0007be10 2023-11-16T16:40:06.862 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.341+0000 7f6e06495700 1 -- 172.21.15.2:0/525701508 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7f6de8038710 msgr2=0x7f6de803abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:06.862 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.341+0000 7f6e06495700 1 --2- 172.21.15.2:0/525701508 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7f6de8038710 0x7f6de803abc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6df4006fd0 tx=0x7f6df4006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:06.862 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.341+0000 7f6e06495700 1 -- 172.21.15.2:0/525701508 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0007be10 msgr2=0x7f6e0019b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:06.862 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.341+0000 7f6e06495700 1 --2- 172.21.15.2:0/525701508 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0007be10 0x7f6e0019b950 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6df0006af0 tx=0x7f6df0011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:06.862 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.341+0000 7f6e06495700 1 -- 172.21.15.2:0/525701508 shutdown_connections 2023-11-16T16:40:06.862 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.341+0000 7f6e06495700 1 --2- 172.21.15.2:0/525701508 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7f6de8038710 0x7f6de803abc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:06.862 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.341+0000 7f6e06495700 1 --2- 172.21.15.2:0/525701508 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0007be10 0x7f6e0019b950 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:06.863 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.341+0000 7f6e06495700 1 -- 172.21.15.2:0/525701508 >> 172.21.15.2:0/525701508 conn(0x7f6e00100fd0 msgr2=0x7f6e001026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:06.863 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.341+0000 7f6e06495700 1 -- 172.21.15.2:0/525701508 shutdown_connections 2023-11-16T16:40:06.863 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:06.341+0000 7f6e06495700 1 -- 172.21.15.2:0/525701508 wait complete. 2023-11-16T16:40:06.863 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling cephadm module... 2023-11-16T16:40:08.028 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:07 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2926376240' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-11-16T16:40:08.029 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:07 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: ignoring --setuser ceph since I am not root 2023-11-16T16:40:08.029 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:07 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: ignoring --setgroup ceph since I am not root 2023-11-16T16:40:08.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.428+0000 7f1b76ff7700 1 Processor -- start 2023-11-16T16:40:08.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.429+0000 7f1b76ff7700 1 -- start start 2023-11-16T16:40:08.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.429+0000 7f1b76ff7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b7007be70 0x7f1b7007a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:08.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.429+0000 7f1b76ff7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7007a8a0 con 0x7f1b7007be70 2023-11-16T16:40:08.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.429+0000 7f1b74d93700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b7007be70 0x7f1b7007a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:08.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.429+0000 7f1b74d93700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b7007be70 0x7f1b7007a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48480/0 (socket says 172.21.15.2:48480) 2023-11-16T16:40:08.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.429+0000 7f1b74d93700 1 -- 172.21.15.2:0/2412349314 learned_addr learned my addr 172.21.15.2:0/2412349314 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:08.243 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.430+0000 7f1b74d93700 1 -- 172.21.15.2:0/2412349314 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7007a9e0 con 0x7f1b7007be70 2023-11-16T16:40:08.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.430+0000 7f1b74d93700 1 --2- 172.21.15.2:0/2412349314 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b7007be70 0x7f1b7007a2d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1b640098a0 tx=0x7f1b64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9c76fa54205be server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:08.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.430+0000 7f1b6f7fe700 1 -- 172.21.15.2:0/2412349314 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b64004160 con 0x7f1b7007be70 2023-11-16T16:40:08.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.430+0000 7f1b6f7fe700 1 -- 172.21.15.2:0/2412349314 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b640042c0 con 0x7f1b7007be70 2023-11-16T16:40:08.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.430+0000 7f1b6f7fe700 1 -- 172.21.15.2:0/2412349314 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b64011a00 con 0x7f1b7007be70 2023-11-16T16:40:08.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.431+0000 7f1b76ff7700 1 -- 172.21.15.2:0/2412349314 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b7007be70 msgr2=0x7f1b7007a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:08.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.431+0000 7f1b76ff7700 1 --2- 172.21.15.2:0/2412349314 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b7007be70 0x7f1b7007a2d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1b640098a0 tx=0x7f1b64009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:08.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.431+0000 7f1b76ff7700 1 -- 172.21.15.2:0/2412349314 shutdown_connections 2023-11-16T16:40:08.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.431+0000 7f1b76ff7700 1 --2- 172.21.15.2:0/2412349314 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b7007be70 0x7f1b7007a2d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:08.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.431+0000 7f1b76ff7700 1 -- 172.21.15.2:0/2412349314 >> 172.21.15.2:0/2412349314 conn(0x7f1b70100fd0 msgr2=0x7f1b70103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:08.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.431+0000 7f1b76ff7700 1 -- 172.21.15.2:0/2412349314 shutdown_connections 2023-11-16T16:40:08.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.431+0000 7f1b76ff7700 1 -- 172.21.15.2:0/2412349314 wait complete. 2023-11-16T16:40:08.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.432+0000 7f1b76ff7700 1 Processor -- start 2023-11-16T16:40:08.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.433+0000 7f1b76ff7700 1 -- start start 2023-11-16T16:40:08.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.433+0000 7f1b76ff7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b7007be70 0x7f1b7019b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:08.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.433+0000 7f1b76ff7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7019bef0 con 0x7f1b7007be70 2023-11-16T16:40:08.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.433+0000 7f1b74d93700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b7007be70 0x7f1b7019b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:08.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.433+0000 7f1b74d93700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b7007be70 0x7f1b7019b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48494/0 (socket says 172.21.15.2:48494) 2023-11-16T16:40:08.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.433+0000 7f1b74d93700 1 -- 172.21.15.2:0/2926376240 learned_addr learned my addr 172.21.15.2:0/2926376240 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:08.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.434+0000 7f1b74d93700 1 -- 172.21.15.2:0/2926376240 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b64009550 con 0x7f1b7007be70 2023-11-16T16:40:08.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.434+0000 7f1b74d93700 1 --2- 172.21.15.2:0/2926376240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b7007be70 0x7f1b7019b9b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1b64006af0 tx=0x7f1b64011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:08.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.434+0000 7f1b6dffb700 1 -- 172.21.15.2:0/2926376240 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b64003a40 con 0x7f1b7007be70 2023-11-16T16:40:08.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.434+0000 7f1b6dffb700 1 -- 172.21.15.2:0/2926376240 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b64003ba0 con 0x7f1b7007be70 2023-11-16T16:40:08.245 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.434+0000 7f1b76ff7700 1 -- 172.21.15.2:0/2926376240 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7010bf60 con 0x7f1b7007be70 2023-11-16T16:40:08.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.434+0000 7f1b6dffb700 1 -- 172.21.15.2:0/2926376240 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b6401a6e0 con 0x7f1b7007be70 2023-11-16T16:40:08.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.434+0000 7f1b76ff7700 1 -- 172.21.15.2:0/2926376240 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7010bdd0 con 0x7f1b7007be70 2023-11-16T16:40:08.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.435+0000 7f1b6dffb700 1 -- 172.21.15.2:0/2926376240 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f1b64024a60 con 0x7f1b7007be70 2023-11-16T16:40:08.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.435+0000 7f1b76ff7700 1 -- 172.21.15.2:0/2926376240 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b70062120 con 0x7f1b7007be70 2023-11-16T16:40:08.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.436+0000 7f1b6dffb700 1 --2- 172.21.15.2:0/2926376240 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7f1b600386c0 0x7f1b6003ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:08.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.436+0000 7f1b6dffb700 1 -- 172.21.15.2:0/2926376240 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1b6404c6d0 con 0x7f1b7007be70 2023-11-16T16:40:08.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.436+0000 7f1b6ffff700 1 --2- 172.21.15.2:0/2926376240 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7f1b600386c0 0x7f1b6003ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:08.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.437+0000 7f1b6ffff700 1 --2- 172.21.15.2:0/2926376240 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7f1b600386c0 0x7f1b6003ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c006fd0 tx=0x7f1b5c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:08.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.441+0000 7f1b6dffb700 1 -- 172.21.15.2:0/2926376240 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1b64024430 con 0x7f1b7007be70 2023-11-16T16:40:08.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.702+0000 7f1b76ff7700 1 -- 172.21.15.2:0/2926376240 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f1b7019c680 con 0x7f1b7007be70 2023-11-16T16:40:08.246 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.755+0000 7f1b6dffb700 1 -- 172.21.15.2:0/2926376240 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f1b6402d910 con 0x7f1b7007be70 2023-11-16T16:40:08.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.757+0000 7f1b6dffb700 1 -- 172.21.15.2:0/2926376240 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f1b6404c0f0 con 0x7f1b7007be70 2023-11-16T16:40:08.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.760+0000 7f1b76ff7700 1 -- 172.21.15.2:0/2926376240 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7f1b600386c0 msgr2=0x7f1b6003ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:08.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.760+0000 7f1b76ff7700 1 --2- 172.21.15.2:0/2926376240 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7f1b600386c0 0x7f1b6003ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c006fd0 tx=0x7f1b5c006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:08.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.760+0000 7f1b76ff7700 1 -- 172.21.15.2:0/2926376240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b7007be70 msgr2=0x7f1b7019b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:08.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.760+0000 7f1b76ff7700 1 --2- 172.21.15.2:0/2926376240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b7007be70 0x7f1b7019b9b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1b64006af0 tx=0x7f1b64011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:08.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.760+0000 7f1b76ff7700 1 -- 172.21.15.2:0/2926376240 shutdown_connections 2023-11-16T16:40:08.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.760+0000 7f1b76ff7700 1 --2- 172.21.15.2:0/2926376240 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7f1b600386c0 0x7f1b6003ab70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:08.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.760+0000 7f1b76ff7700 1 --2- 172.21.15.2:0/2926376240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b7007be70 0x7f1b7019b9b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:08.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.760+0000 7f1b76ff7700 1 -- 172.21.15.2:0/2926376240 >> 172.21.15.2:0/2926376240 conn(0x7f1b70100fd0 msgr2=0x7f1b701026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:08.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.760+0000 7f1b76ff7700 1 -- 172.21.15.2:0/2926376240 shutdown_connections 2023-11-16T16:40:08.247 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:07.760+0000 7f1b76ff7700 1 -- 172.21.15.2:0/2926376240 wait complete. 2023-11-16T16:40:08.384 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:08 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:08.027+0000 7f56f7a8b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T16:40:08.384 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:08 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:08.135+0000 7f56f7a8b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T16:40:08.384 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:08 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:08.253+0000 7f56f7a8b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T16:40:08.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:08 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:08.383+0000 7f56f7a8b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T16:40:08.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:08 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:08.631+0000 7f56f7a8b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T16:40:09.054 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:08 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2926376240' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-11-16T16:40:09.055 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:08 smithi002 ceph-mon[128712]: mgrmap e5: x(active, since 8s) 2023-11-16T16:40:09.360 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:09 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:09.053+0000 7f56f7a8b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T16:40:09.360 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:09 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:09.191+0000 7f56f7a8b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T16:40:09.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2023-11-16T16:40:09.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-11-16T16:40:09.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": true, 2023-11-16T16:40:09.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-11-16T16:40:09.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-11-16T16:40:09.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2023-11-16T16:40:09.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.759+0000 7efd27c3d700 1 Processor -- start 2023-11-16T16:40:09.528 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.760+0000 7efd27c3d700 1 -- start start 2023-11-16T16:40:09.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.760+0000 7efd27c3d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd20107c20 0x7efd20108030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:09.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.760+0000 7efd27c3d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd20108600 con 0x7efd20107c20 2023-11-16T16:40:09.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.760+0000 7efd259d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd20107c20 0x7efd20108030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:09.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.761+0000 7efd259d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd20107c20 0x7efd20108030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48532/0 (socket says 172.21.15.2:48532) 2023-11-16T16:40:09.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.761+0000 7efd259d9700 1 -- 172.21.15.2:0/381494864 learned_addr learned my addr 172.21.15.2:0/381494864 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:09.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.761+0000 7efd259d9700 1 -- 172.21.15.2:0/381494864 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd20108e20 con 0x7efd20107c20 2023-11-16T16:40:09.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.761+0000 7efd259d9700 1 --2- 172.21.15.2:0/381494864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd20107c20 0x7efd20108030 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efd1c0098a0 tx=0x7efd1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b719e4bd2d8be049 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:09.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.762+0000 7efd249d7700 1 -- 172.21.15.2:0/381494864 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd1c004160 con 0x7efd20107c20 2023-11-16T16:40:09.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.762+0000 7efd249d7700 1 -- 172.21.15.2:0/381494864 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd1c0042c0 con 0x7efd20107c20 2023-11-16T16:40:09.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.762+0000 7efd249d7700 1 -- 172.21.15.2:0/381494864 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd1c011a00 con 0x7efd20107c20 2023-11-16T16:40:09.529 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.762+0000 7efd27c3d700 1 -- 172.21.15.2:0/381494864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd20107c20 msgr2=0x7efd20108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:09.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.762+0000 7efd27c3d700 1 --2- 172.21.15.2:0/381494864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd20107c20 0x7efd20108030 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efd1c0098a0 tx=0x7efd1c009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:09.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.762+0000 7efd27c3d700 1 -- 172.21.15.2:0/381494864 shutdown_connections 2023-11-16T16:40:09.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.762+0000 7efd27c3d700 1 --2- 172.21.15.2:0/381494864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd20107c20 0x7efd20108030 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:09.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.762+0000 7efd27c3d700 1 -- 172.21.15.2:0/381494864 >> 172.21.15.2:0/381494864 conn(0x7efd2007b000 msgr2=0x7efd2007b420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:09.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.763+0000 7efd27c3d700 1 -- 172.21.15.2:0/381494864 shutdown_connections 2023-11-16T16:40:09.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.763+0000 7efd27c3d700 1 -- 172.21.15.2:0/381494864 wait complete. 2023-11-16T16:40:09.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.764+0000 7efd27c3d700 1 Processor -- start 2023-11-16T16:40:09.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.764+0000 7efd27c3d700 1 -- start start 2023-11-16T16:40:09.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.764+0000 7efd27c3d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd20107c20 0x7efd2019bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:09.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.764+0000 7efd27c3d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2019c0b0 con 0x7efd20107c20 2023-11-16T16:40:09.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.764+0000 7efd259d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd20107c20 0x7efd2019bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:09.530 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.764+0000 7efd259d9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd20107c20 0x7efd2019bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48542/0 (socket says 172.21.15.2:48542) 2023-11-16T16:40:09.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.764+0000 7efd259d9700 1 -- 172.21.15.2:0/2302998321 learned_addr learned my addr 172.21.15.2:0/2302998321 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:09.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.765+0000 7efd259d9700 1 -- 172.21.15.2:0/2302998321 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1c009550 con 0x7efd20107c20 2023-11-16T16:40:09.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.765+0000 7efd259d9700 1 --2- 172.21.15.2:0/2302998321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd20107c20 0x7efd2019bb70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efd1c006af0 tx=0x7efd1c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:09.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.765+0000 7efd16ffd700 1 -- 172.21.15.2:0/2302998321 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd1c003a40 con 0x7efd20107c20 2023-11-16T16:40:09.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.765+0000 7efd27c3d700 1 -- 172.21.15.2:0/2302998321 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd2010c050 con 0x7efd20107c20 2023-11-16T16:40:09.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.765+0000 7efd16ffd700 1 -- 172.21.15.2:0/2302998321 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd1c003ba0 con 0x7efd20107c20 2023-11-16T16:40:09.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.765+0000 7efd16ffd700 1 -- 172.21.15.2:0/2302998321 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd1c01a700 con 0x7efd20107c20 2023-11-16T16:40:09.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.765+0000 7efd27c3d700 1 -- 172.21.15.2:0/2302998321 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd2010c1e0 con 0x7efd20107c20 2023-11-16T16:40:09.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.766+0000 7efd16ffd700 1 -- 172.21.15.2:0/2302998321 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7efd1c01a930 con 0x7efd20107c20 2023-11-16T16:40:09.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.767+0000 7efd27c3d700 1 -- 172.21.15.2:0/2302998321 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd20109c90 con 0x7efd20107c20 2023-11-16T16:40:09.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.767+0000 7efd16ffd700 1 --2- 172.21.15.2:0/2302998321 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7efd0c038760 0x7efd0c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:09.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.767+0000 7efd16ffd700 1 -- 172.21.15.2:0/2302998321 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7efd1c04c6b0 con 0x7efd20107c20 2023-11-16T16:40:09.531 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.767+0000 7efd251d8700 1 -- 172.21.15.2:0/2302998321 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7efd0c038760 msgr2=0x7efd0c03ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3453602765 2023-11-16T16:40:09.532 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.767+0000 7efd251d8700 1 --2- 172.21.15.2:0/2302998321 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7efd0c038760 0x7efd0c03ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:40:09.532 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.777+0000 7efd16ffd700 1 -- 172.21.15.2:0/2302998321 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7efd20109c90 con 0x7efd20107c20 2023-11-16T16:40:09.532 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.968+0000 7efd251d8700 1 -- 172.21.15.2:0/2302998321 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7efd0c038760 msgr2=0x7efd0c03ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3453602765 2023-11-16T16:40:09.532 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:08.968+0000 7efd251d8700 1 --2- 172.21.15.2:0/2302998321 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7efd0c038760 0x7efd0c03ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T16:40:09.532 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.056+0000 7efd27c3d700 1 -- 172.21.15.2:0/2302998321 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7efd20109c90 con 0x7efd20107c20 2023-11-16T16:40:09.532 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.057+0000 7efd16ffd700 1 -- 172.21.15.2:0/2302998321 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7efd20109c90 con 0x7efd20107c20 2023-11-16T16:40:09.532 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.059+0000 7efd27c3d700 1 -- 172.21.15.2:0/2302998321 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7efd0c038760 msgr2=0x7efd0c03ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:40:09.532 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.059+0000 7efd27c3d700 1 --2- 172.21.15.2:0/2302998321 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7efd0c038760 0x7efd0c03ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:09.532 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.059+0000 7efd27c3d700 1 -- 172.21.15.2:0/2302998321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd20107c20 msgr2=0x7efd2019bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:09.532 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.059+0000 7efd27c3d700 1 --2- 172.21.15.2:0/2302998321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd20107c20 0x7efd2019bb70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efd1c006af0 tx=0x7efd1c011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:09.532 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.059+0000 7efd27c3d700 1 -- 172.21.15.2:0/2302998321 shutdown_connections 2023-11-16T16:40:09.533 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.059+0000 7efd27c3d700 1 --2- 172.21.15.2:0/2302998321 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7efd0c038760 0x7efd0c03ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:09.533 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.059+0000 7efd27c3d700 1 --2- 172.21.15.2:0/2302998321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd20107c20 0x7efd2019bb70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:09.533 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.059+0000 7efd27c3d700 1 -- 172.21.15.2:0/2302998321 >> 172.21.15.2:0/2302998321 conn(0x7efd2007b000 msgr2=0x7efd20106100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:09.533 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.059+0000 7efd27c3d700 1 -- 172.21.15.2:0/2302998321 shutdown_connections 2023-11-16T16:40:09.533 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:09.060+0000 7efd27c3d700 1 -- 172.21.15.2:0/2302998321 wait complete. 2023-11-16T16:40:09.533 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for the mgr to restart... 2023-11-16T16:40:09.533 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr epoch 5... 2023-11-16T16:40:09.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:09 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:09.359+0000 7f56f7a8b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T16:40:09.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:09 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:09.488+0000 7f56f7a8b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T16:40:10.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:09 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2302998321' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-16T16:40:10.405 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:10 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:10.074+0000 7f56f7a8b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T16:40:11.155 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:10 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:10.843+0000 7f56f7a8b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T16:40:11.155 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:11 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:11.057+0000 7f56f7a8b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T16:40:12.393 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:12 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:12.037+0000 7f56f7a8b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:40:12.393 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:12 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:12.161+0000 7f56f7a8b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T16:40:12.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:12 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:12.392+0000 7f56f7a8b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:40:13.050 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:12 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:12.765+0000 7f56f7a8b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T16:40:13.405 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:13 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:13.049+0000 7f56f7a8b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T16:40:13.406 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:13 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:13.248+0000 7f56f7a8b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T16:40:13.905 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:13 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:13.603+0000 7f56f7a8b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T16:40:14.905 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:14 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:14.562+0000 7f56f7a8b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T16:40:14.905 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:14 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:14.741+0000 7f56f7a8b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T16:40:15.405 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:14 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:14.996+0000 7f56f7a8b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T16:40:15.406 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:15 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:15.149+0000 7f56f7a8b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T16:40:15.905 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:15 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:15.436+0000 7f56f7a8b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T16:40:16.905 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:16 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:16.535+0000 7f56f7a8b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T16:40:16.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:16 smithi002 ceph-mon[128712]: Active manager daemon x restarted 2023-11-16T16:40:16.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:16 smithi002 ceph-mon[128712]: Activating manager daemon x 2023-11-16T16:40:16.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:16 smithi002 ceph-mon[128712]: osdmap e2: 0 total, 0 up, 0 in 2023-11-16T16:40:16.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:16 smithi002 ceph-mon[128712]: mgrmap e6: x(active, starting, since 0.00386401s) 2023-11-16T16:40:16.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:16 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:40:16.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:16 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T16:40:16.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:16 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T16:40:16.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:16 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T16:40:16.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:16 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T16:40:16.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:16 smithi002 ceph-mon[128712]: Manager daemon x is now available 2023-11-16T16:40:16.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:16 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' 2023-11-16T16:40:16.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:16 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' 2023-11-16T16:40:16.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:16 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:16.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:16 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:16.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:16 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:17.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:17 smithi002 ceph-mon[128712]: Found migration_current of "None". Setting to last migration. 2023-11-16T16:40:17.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:17 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T16:40:17.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:17 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T16:40:17.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:17 smithi002 ceph-mon[128712]: mgrmap e7: x(active, since 1.00723s) 2023-11-16T16:40:18.029 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2023-11-16T16:40:18.036 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-11-16T16:40:18.036 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "initialized": true 2023-11-16T16:40:18.036 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2023-11-16T16:40:18.036 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.070+0000 7fe771184700 1 Processor -- start 2023-11-16T16:40:18.036 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.070+0000 7fe771184700 1 -- start start 2023-11-16T16:40:18.037 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.070+0000 7fe771184700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe76c07be10 0x7fe76c07a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:18.037 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.070+0000 7fe771184700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c07a840 con 0x7fe76c07be10 2023-11-16T16:40:18.037 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.071+0000 7fe76ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe76c07be10 0x7fe76c07a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:18.037 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.071+0000 7fe76ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe76c07be10 0x7fe76c07a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48550/0 (socket says 172.21.15.2:48550) 2023-11-16T16:40:18.037 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.071+0000 7fe76ad9d700 1 -- 172.21.15.2:0/1726496642 learned_addr learned my addr 172.21.15.2:0/1726496642 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:18.037 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.071+0000 7fe76ad9d700 1 -- 172.21.15.2:0/1726496642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76c07a980 con 0x7fe76c07be10 2023-11-16T16:40:18.037 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.071+0000 7fe76ad9d700 1 --2- 172.21.15.2:0/1726496642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe76c07be10 0x7fe76c07a270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0098a0 tx=0x7fe75c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef3956c37dfce688 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:18.037 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.071+0000 7fe769d9b700 1 -- 172.21.15.2:0/1726496642 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe75c004160 con 0x7fe76c07be10 2023-11-16T16:40:18.037 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.072+0000 7fe769d9b700 1 -- 172.21.15.2:0/1726496642 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe75c0042c0 con 0x7fe76c07be10 2023-11-16T16:40:18.037 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.072+0000 7fe769d9b700 1 -- 172.21.15.2:0/1726496642 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe75c011a00 con 0x7fe76c07be10 2023-11-16T16:40:18.037 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.072+0000 7fe771184700 1 -- 172.21.15.2:0/1726496642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe76c07be10 msgr2=0x7fe76c07a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:18.037 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.072+0000 7fe771184700 1 --2- 172.21.15.2:0/1726496642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe76c07be10 0x7fe76c07a270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0098a0 tx=0x7fe75c009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:18.038 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.072+0000 7fe771184700 1 -- 172.21.15.2:0/1726496642 shutdown_connections 2023-11-16T16:40:18.038 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.072+0000 7fe771184700 1 --2- 172.21.15.2:0/1726496642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe76c07be10 0x7fe76c07a270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:18.038 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.072+0000 7fe771184700 1 -- 172.21.15.2:0/1726496642 >> 172.21.15.2:0/1726496642 conn(0x7fe76c100fd0 msgr2=0x7fe76c103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:18.038 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.072+0000 7fe771184700 1 -- 172.21.15.2:0/1726496642 shutdown_connections 2023-11-16T16:40:18.038 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.073+0000 7fe771184700 1 -- 172.21.15.2:0/1726496642 wait complete. 2023-11-16T16:40:18.038 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.073+0000 7fe771184700 1 Processor -- start 2023-11-16T16:40:18.038 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.074+0000 7fe771184700 1 -- start start 2023-11-16T16:40:18.038 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.074+0000 7fe771184700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe76c07be10 0x7fe76c19b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:18.038 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.074+0000 7fe771184700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c19be90 con 0x7fe76c07be10 2023-11-16T16:40:18.038 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.075+0000 7fe76ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe76c07be10 0x7fe76c19b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:18.038 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.075+0000 7fe76ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe76c07be10 0x7fe76c19b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48556/0 (socket says 172.21.15.2:48556) 2023-11-16T16:40:18.039 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.075+0000 7fe76ad9d700 1 -- 172.21.15.2:0/1865257970 learned_addr learned my addr 172.21.15.2:0/1865257970 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:18.039 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.075+0000 7fe76ad9d700 1 -- 172.21.15.2:0/1865257970 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75c009550 con 0x7fe76c07be10 2023-11-16T16:40:18.039 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.075+0000 7fe76ad9d700 1 --2- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe76c07be10 0x7fe76c19b950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe75c006af0 tx=0x7fe75c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:18.039 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.076+0000 7fe753fff700 1 -- 172.21.15.2:0/1865257970 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe75c003a40 con 0x7fe76c07be10 2023-11-16T16:40:18.039 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.076+0000 7fe753fff700 1 -- 172.21.15.2:0/1865257970 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe75c003ba0 con 0x7fe76c07be10 2023-11-16T16:40:18.039 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.076+0000 7fe753fff700 1 -- 172.21.15.2:0/1865257970 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe75c01a6e0 con 0x7fe76c07be10 2023-11-16T16:40:18.039 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.076+0000 7fe771184700 1 -- 172.21.15.2:0/1865257970 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76c10bf60 con 0x7fe76c07be10 2023-11-16T16:40:18.039 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.076+0000 7fe771184700 1 -- 172.21.15.2:0/1865257970 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76c10bdd0 con 0x7fe76c07be10 2023-11-16T16:40:18.039 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.077+0000 7fe753fff700 1 -- 172.21.15.2:0/1865257970 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fe75c01a860 con 0x7fe76c07be10 2023-11-16T16:40:18.039 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.078+0000 7fe753fff700 1 --2- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fe7540387b0 0x7fe75403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:18.039 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.078+0000 7fe753fff700 1 -- 172.21.15.2:0/1865257970 --> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe75403b370 con 0x7fe7540387b0 2023-11-16T16:40:18.039 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.078+0000 7fe753fff700 1 -- 172.21.15.2:0/1865257970 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe75c04c6e0 con 0x7fe76c07be10 2023-11-16T16:40:18.040 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.078+0000 7fe76a59c700 1 -- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fe7540387b0 msgr2=0x7fe75403ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3453602765 2023-11-16T16:40:18.040 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.078+0000 7fe76a59c700 1 --2- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fe7540387b0 0x7fe75403ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:40:18.040 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.279+0000 7fe76a59c700 1 -- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fe7540387b0 msgr2=0x7fe75403ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3453602765 2023-11-16T16:40:18.040 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.279+0000 7fe76a59c700 1 --2- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fe7540387b0 0x7fe75403ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T16:40:18.040 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.680+0000 7fe76a59c700 1 -- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fe7540387b0 msgr2=0x7fe75403ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3453602765 2023-11-16T16:40:18.040 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:10.680+0000 7fe76a59c700 1 --2- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fe7540387b0 0x7fe75403ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-16T16:40:18.040 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:11.481+0000 7fe76a59c700 1 -- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fe7540387b0 msgr2=0x7fe75403ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3453602765 2023-11-16T16:40:18.040 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:11.481+0000 7fe76a59c700 1 --2- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fe7540387b0 0x7fe75403ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-16T16:40:18.040 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:13.083+0000 7fe76a59c700 1 -- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fe7540387b0 msgr2=0x7fe75403ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3453602765 2023-11-16T16:40:18.040 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:13.083+0000 7fe76a59c700 1 --2- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fe7540387b0 0x7fe75403ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-16T16:40:18.040 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.285+0000 7fe76a59c700 1 -- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fe7540387b0 msgr2=0x7fe75403ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3453602765 2023-11-16T16:40:18.041 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.285+0000 7fe76a59c700 1 --2- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fe7540387b0 0x7fe75403ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-16T16:40:18.041 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.539+0000 7fe753fff700 1 -- 172.21.15.2:0/1865257970 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7fe75c01acf0 con 0x7fe76c07be10 2023-11-16T16:40:18.041 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.539+0000 7fe753fff700 1 -- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fe7540387b0 msgr2=0x7fe75403ac60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:40:18.041 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:16.539+0000 7fe753fff700 1 --2- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fe7540387b0 0x7fe75403ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:18.041 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.541+0000 7fe753fff700 1 -- 172.21.15.2:0/1865257970 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fe75c04e000 con 0x7fe76c07be10 2023-11-16T16:40:18.041 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.542+0000 7fe753fff700 1 --2- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe75403bfe0 0x7fe75403e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:18.041 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.542+0000 7fe753fff700 1 -- 172.21.15.2:0/1865257970 --> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe75403b370 con 0x7fe75403bfe0 2023-11-16T16:40:18.041 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.542+0000 7fe76a59c700 1 --2- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe75403bfe0 0x7fe75403e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:18.041 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.544+0000 7fe76a59c700 1 --2- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe75403bfe0 0x7fe75403e3c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe760003a10 tx=0x7fe7600092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:18.041 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.545+0000 7fe753fff700 1 -- 172.21.15.2:0/1865257970 <== mgr.14118 v2:172.21.15.2:6800/3359672074 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fe75403b370 con 0x7fe75403bfe0 2023-11-16T16:40:18.041 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.555+0000 7fe771184700 1 -- 172.21.15.2:0/1865257970 --> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fe76c19cc90 con 0x7fe75403bfe0 2023-11-16T16:40:18.041 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.555+0000 7fe753fff700 1 -- 172.21.15.2:0/1865257970 <== mgr.14118 v2:172.21.15.2:6800/3359672074 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fe76c19cc90 con 0x7fe75403bfe0 2023-11-16T16:40:18.042 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.556+0000 7fe771184700 1 -- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe75403bfe0 msgr2=0x7fe75403e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:18.042 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.556+0000 7fe771184700 1 --2- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe75403bfe0 0x7fe75403e3c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe760003a10 tx=0x7fe7600092b0 comp rx=0 tx=0).stop 2023-11-16T16:40:18.042 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.556+0000 7fe771184700 1 -- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe76c07be10 msgr2=0x7fe76c19b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:18.042 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.556+0000 7fe771184700 1 --2- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe76c07be10 0x7fe76c19b950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe75c006af0 tx=0x7fe75c011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:18.042 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.556+0000 7fe771184700 1 -- 172.21.15.2:0/1865257970 shutdown_connections 2023-11-16T16:40:18.042 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.556+0000 7fe771184700 1 --2- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe76c07be10 0x7fe76c19b950 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:18.042 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.556+0000 7fe771184700 1 --2- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe75403bfe0 0x7fe75403e3c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:18.042 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.556+0000 7fe771184700 1 --2- 172.21.15.2:0/1865257970 >> [v2:172.21.15.2:6800/3453602765,v1:172.21.15.2:6801/3453602765] conn(0x7fe7540387b0 0x7fe75403ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:18.042 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.556+0000 7fe771184700 1 -- 172.21.15.2:0/1865257970 >> 172.21.15.2:0/1865257970 conn(0x7fe76c100fd0 msgr2=0x7fe76c1026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:18.042 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.556+0000 7fe771184700 1 -- 172.21.15.2:0/1865257970 shutdown_connections 2023-11-16T16:40:18.042 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:17.556+0000 7fe771184700 1 -- 172.21.15.2:0/1865257970 wait complete. 2023-11-16T16:40:18.043 INFO:teuthology.orchestra.run.smithi002.stdout:mgr epoch 5 is available 2023-11-16T16:40:18.043 INFO:teuthology.orchestra.run.smithi002.stdout:Setting orchestrator backend to cephadm... 2023-11-16T16:40:18.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:18 smithi002 ceph-mon[128712]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-16T16:40:18.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:18 smithi002 ceph-mon[128712]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-16T16:40:18.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:18 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' 2023-11-16T16:40:18.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:18 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' 2023-11-16T16:40:19.422 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.721+0000 7fb1983a7700 1 Processor -- start 2023-11-16T16:40:19.422 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.721+0000 7fb1983a7700 1 -- start start 2023-11-16T16:40:19.422 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.721+0000 7fb1983a7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb19007be10 0x7fb19007a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:19.422 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.721+0000 7fb1983a7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19007a840 con 0x7fb19007be10 2023-11-16T16:40:19.422 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.722+0000 7fb196143700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb19007be10 0x7fb19007a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:19.423 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.722+0000 7fb196143700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb19007be10 0x7fb19007a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47866/0 (socket says 172.21.15.2:47866) 2023-11-16T16:40:19.423 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.722+0000 7fb196143700 1 -- 172.21.15.2:0/2267864448 learned_addr learned my addr 172.21.15.2:0/2267864448 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:19.423 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.722+0000 7fb196143700 1 -- 172.21.15.2:0/2267864448 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19007a980 con 0x7fb19007be10 2023-11-16T16:40:19.423 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.722+0000 7fb196143700 1 --2- 172.21.15.2:0/2267864448 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb19007be10 0x7fb19007a270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb18c0098a0 tx=0x7fb18c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d95c2bb9029fd9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:19.423 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.722+0000 7fb195141700 1 -- 172.21.15.2:0/2267864448 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb18c004160 con 0x7fb19007be10 2023-11-16T16:40:19.423 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.723+0000 7fb195141700 1 -- 172.21.15.2:0/2267864448 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb18c0042c0 con 0x7fb19007be10 2023-11-16T16:40:19.423 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.723+0000 7fb195141700 1 -- 172.21.15.2:0/2267864448 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb18c011a00 con 0x7fb19007be10 2023-11-16T16:40:19.423 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.723+0000 7fb1983a7700 1 -- 172.21.15.2:0/2267864448 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb19007be10 msgr2=0x7fb19007a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:19.423 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.723+0000 7fb1983a7700 1 --2- 172.21.15.2:0/2267864448 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb19007be10 0x7fb19007a270 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb18c0098a0 tx=0x7fb18c009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:19.423 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.723+0000 7fb1983a7700 1 -- 172.21.15.2:0/2267864448 shutdown_connections 2023-11-16T16:40:19.423 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.723+0000 7fb1983a7700 1 --2- 172.21.15.2:0/2267864448 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb19007be10 0x7fb19007a270 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:19.423 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.723+0000 7fb1983a7700 1 -- 172.21.15.2:0/2267864448 >> 172.21.15.2:0/2267864448 conn(0x7fb190100fd0 msgr2=0x7fb190103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:19.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.723+0000 7fb1983a7700 1 -- 172.21.15.2:0/2267864448 shutdown_connections 2023-11-16T16:40:19.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.723+0000 7fb1983a7700 1 -- 172.21.15.2:0/2267864448 wait complete. 2023-11-16T16:40:19.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.724+0000 7fb1983a7700 1 Processor -- start 2023-11-16T16:40:19.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.725+0000 7fb1983a7700 1 -- start start 2023-11-16T16:40:19.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.725+0000 7fb1983a7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb19007be10 0x7fb19019b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:19.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.725+0000 7fb1983a7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19019be90 con 0x7fb19007be10 2023-11-16T16:40:19.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.725+0000 7fb196143700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb19007be10 0x7fb19019b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:19.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.725+0000 7fb196143700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb19007be10 0x7fb19019b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47868/0 (socket says 172.21.15.2:47868) 2023-11-16T16:40:19.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.725+0000 7fb196143700 1 -- 172.21.15.2:0/3695174318 learned_addr learned my addr 172.21.15.2:0/3695174318 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:19.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.725+0000 7fb196143700 1 -- 172.21.15.2:0/3695174318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18c009550 con 0x7fb19007be10 2023-11-16T16:40:19.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.726+0000 7fb196143700 1 --2- 172.21.15.2:0/3695174318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb19007be10 0x7fb19019b950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb18c006af0 tx=0x7fb18c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:19.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.726+0000 7fb1877fe700 1 -- 172.21.15.2:0/3695174318 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb18c003a40 con 0x7fb19007be10 2023-11-16T16:40:19.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.726+0000 7fb1877fe700 1 -- 172.21.15.2:0/3695174318 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb18c003ba0 con 0x7fb19007be10 2023-11-16T16:40:19.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.726+0000 7fb1877fe700 1 -- 172.21.15.2:0/3695174318 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb18c01a6e0 con 0x7fb19007be10 2023-11-16T16:40:19.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.726+0000 7fb1983a7700 1 -- 172.21.15.2:0/3695174318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb19010bf60 con 0x7fb19007be10 2023-11-16T16:40:19.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.726+0000 7fb1983a7700 1 -- 172.21.15.2:0/3695174318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb19010bdd0 con 0x7fb19007be10 2023-11-16T16:40:19.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.727+0000 7fb1877fe700 1 -- 172.21.15.2:0/3695174318 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fb18c024a60 con 0x7fb19007be10 2023-11-16T16:40:19.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.727+0000 7fb1983a7700 1 -- 172.21.15.2:0/3695174318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb190062120 con 0x7fb19007be10 2023-11-16T16:40:19.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.728+0000 7fb1877fe700 1 --2- 172.21.15.2:0/3695174318 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fb17c038630 0x7fb17c03aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:19.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.728+0000 7fb1877fe700 1 -- 172.21.15.2:0/3695174318 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb18c04c5c0 con 0x7fb19007be10 2023-11-16T16:40:19.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.728+0000 7fb195942700 1 --2- 172.21.15.2:0/3695174318 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fb17c038630 0x7fb17c03aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:19.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.729+0000 7fb195942700 1 --2- 172.21.15.2:0/3695174318 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fb17c038630 0x7fb17c03aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb180006fd0 tx=0x7fb180006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:19.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.733+0000 7fb1877fe700 1 -- 172.21.15.2:0/3695174318 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb18c024430 con 0x7fb19007be10 2023-11-16T16:40:19.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.958+0000 7fb1983a7700 1 -- 172.21.15.2:0/3695174318 --> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1901032c0 con 0x7fb17c038630 2023-11-16T16:40:19.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.965+0000 7fb1877fe700 1 -- 172.21.15.2:0/3695174318 <== mgr.14118 v2:172.21.15.2:6800/3359672074 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb1901032c0 con 0x7fb17c038630 2023-11-16T16:40:19.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.967+0000 7fb1983a7700 1 -- 172.21.15.2:0/3695174318 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fb17c038630 msgr2=0x7fb17c03aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:19.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.967+0000 7fb1983a7700 1 --2- 172.21.15.2:0/3695174318 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fb17c038630 0x7fb17c03aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb180006fd0 tx=0x7fb180006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:19.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.967+0000 7fb1983a7700 1 -- 172.21.15.2:0/3695174318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb19007be10 msgr2=0x7fb19019b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:19.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.968+0000 7fb1983a7700 1 --2- 172.21.15.2:0/3695174318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb19007be10 0x7fb19019b950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb18c006af0 tx=0x7fb18c011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:19.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.968+0000 7fb1983a7700 1 -- 172.21.15.2:0/3695174318 shutdown_connections 2023-11-16T16:40:19.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.968+0000 7fb1983a7700 1 --2- 172.21.15.2:0/3695174318 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fb17c038630 0x7fb17c03aae0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:19.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.968+0000 7fb1983a7700 1 --2- 172.21.15.2:0/3695174318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb19007be10 0x7fb19019b950 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:19.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.968+0000 7fb1983a7700 1 -- 172.21.15.2:0/3695174318 >> 172.21.15.2:0/3695174318 conn(0x7fb190100fd0 msgr2=0x7fb1901026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:19.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.968+0000 7fb1983a7700 1 -- 172.21.15.2:0/3695174318 shutdown_connections 2023-11-16T16:40:19.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:18.968+0000 7fb1983a7700 1 -- 172.21.15.2:0/3695174318 wait complete. 2023-11-16T16:40:20.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:19 smithi002 ceph-mon[128712]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:40:20.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:19 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' 2023-11-16T16:40:20.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:19 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:20.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:19 smithi002 ceph-mon[128712]: [16/Nov/2023:16:40:19] ENGINE Bus STARTING 2023-11-16T16:40:20.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:19 smithi002 ceph-mon[128712]: mgrmap e8: x(active, since 2s) 2023-11-16T16:40:20.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:19 smithi002 ceph-mon[128712]: [16/Nov/2023:16:40:19] ENGINE Serving on https://172.21.15.2:7150 2023-11-16T16:40:20.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:19 smithi002 ceph-mon[128712]: [16/Nov/2023:16:40:19] ENGINE Serving on http://172.21.15.2:8765 2023-11-16T16:40:20.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:19 smithi002 ceph-mon[128712]: [16/Nov/2023:16:40:19] ENGINE Bus STARTED 2023-11-16T16:40:20.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:19 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:20.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout value unchanged 2023-11-16T16:40:20.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.009+0000 7f62bd725700 1 Processor -- start 2023-11-16T16:40:20.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.010+0000 7f62bd725700 1 -- start start 2023-11-16T16:40:20.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.010+0000 7f62bd725700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62b807be70 0x7f62b807a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:20.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.010+0000 7f62bd725700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b807a8a0 con 0x7f62b807be70 2023-11-16T16:40:20.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.010+0000 7f62b6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62b807be70 0x7f62b807a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:20.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.010+0000 7f62b6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62b807be70 0x7f62b807a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47870/0 (socket says 172.21.15.2:47870) 2023-11-16T16:40:20.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.010+0000 7f62b6ffd700 1 -- 172.21.15.2:0/3084320048 learned_addr learned my addr 172.21.15.2:0/3084320048 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:20.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.011+0000 7f62b6ffd700 1 -- 172.21.15.2:0/3084320048 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b807a9e0 con 0x7f62b807be70 2023-11-16T16:40:20.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.011+0000 7f62b6ffd700 1 --2- 172.21.15.2:0/3084320048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62b807be70 0x7f62b807a2d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f62a80098a0 tx=0x7f62a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3410cd15a76a1b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:20.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.011+0000 7f62b5ffb700 1 -- 172.21.15.2:0/3084320048 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f62a8004160 con 0x7f62b807be70 2023-11-16T16:40:20.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.011+0000 7f62b5ffb700 1 -- 172.21.15.2:0/3084320048 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62a80042c0 con 0x7f62b807be70 2023-11-16T16:40:20.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.011+0000 7f62b5ffb700 1 -- 172.21.15.2:0/3084320048 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f62a8011a00 con 0x7f62b807be70 2023-11-16T16:40:20.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.012+0000 7f62bd725700 1 -- 172.21.15.2:0/3084320048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62b807be70 msgr2=0x7f62b807a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:20.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.012+0000 7f62bd725700 1 --2- 172.21.15.2:0/3084320048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62b807be70 0x7f62b807a2d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f62a80098a0 tx=0x7f62a8009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:20.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.012+0000 7f62bd725700 1 -- 172.21.15.2:0/3084320048 shutdown_connections 2023-11-16T16:40:20.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.012+0000 7f62bd725700 1 --2- 172.21.15.2:0/3084320048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62b807be70 0x7f62b807a2d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:20.811 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.012+0000 7f62bd725700 1 -- 172.21.15.2:0/3084320048 >> 172.21.15.2:0/3084320048 conn(0x7f62b8100fd0 msgr2=0x7f62b8103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:20.811 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.012+0000 7f62bd725700 1 -- 172.21.15.2:0/3084320048 shutdown_connections 2023-11-16T16:40:20.811 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.012+0000 7f62bd725700 1 -- 172.21.15.2:0/3084320048 wait complete. 2023-11-16T16:40:20.811 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.013+0000 7f62bd725700 1 Processor -- start 2023-11-16T16:40:20.811 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.013+0000 7f62bd725700 1 -- start start 2023-11-16T16:40:20.811 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.013+0000 7f62bd725700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62b807be70 0x7f62b819b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:20.811 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.013+0000 7f62bd725700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b819bef0 con 0x7f62b807be70 2023-11-16T16:40:20.811 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.014+0000 7f62b6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62b807be70 0x7f62b819b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:20.811 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.014+0000 7f62b6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62b807be70 0x7f62b819b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47884/0 (socket says 172.21.15.2:47884) 2023-11-16T16:40:20.811 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.014+0000 7f62b6ffd700 1 -- 172.21.15.2:0/140847292 learned_addr learned my addr 172.21.15.2:0/140847292 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:20.811 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.014+0000 7f62b6ffd700 1 -- 172.21.15.2:0/140847292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a8009550 con 0x7f62b807be70 2023-11-16T16:40:20.811 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.014+0000 7f62b6ffd700 1 --2- 172.21.15.2:0/140847292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62b807be70 0x7f62b819b9b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f62a8006af0 tx=0x7f62a8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:20.812 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.014+0000 7f629ffff700 1 -- 172.21.15.2:0/140847292 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f62a8003a40 con 0x7f62b807be70 2023-11-16T16:40:20.812 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.014+0000 7f629ffff700 1 -- 172.21.15.2:0/140847292 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62a8003ba0 con 0x7f62b807be70 2023-11-16T16:40:20.812 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.014+0000 7f62bd725700 1 -- 172.21.15.2:0/140847292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b810bf60 con 0x7f62b807be70 2023-11-16T16:40:20.812 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.014+0000 7f629ffff700 1 -- 172.21.15.2:0/140847292 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f62a801a6e0 con 0x7f62b807be70 2023-11-16T16:40:20.812 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.015+0000 7f62bd725700 1 -- 172.21.15.2:0/140847292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b810bdd0 con 0x7f62b807be70 2023-11-16T16:40:20.812 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.016+0000 7f629ffff700 1 -- 172.21.15.2:0/140847292 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f62a8024430 con 0x7f62b807be70 2023-11-16T16:40:20.812 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.016+0000 7f62bd725700 1 -- 172.21.15.2:0/140847292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b8062120 con 0x7f62b807be70 2023-11-16T16:40:20.812 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.016+0000 7f629ffff700 1 --2- 172.21.15.2:0/140847292 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f62a0038760 0x7f62a003ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:20.812 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.016+0000 7f629ffff700 1 -- 172.21.15.2:0/140847292 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f62a804c7b0 con 0x7f62b807be70 2023-11-16T16:40:20.812 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.016+0000 7f62b67fc700 1 --2- 172.21.15.2:0/140847292 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f62a0038760 0x7f62a003ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:20.812 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.017+0000 7f62b67fc700 1 --2- 172.21.15.2:0/140847292 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f62a0038760 0x7f62a003ac10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f62ac006fd0 tx=0x7f62ac006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:20.812 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.021+0000 7f629ffff700 1 -- 172.21.15.2:0/140847292 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f62a80246e0 con 0x7f62b807be70 2023-11-16T16:40:20.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.227+0000 7f62bd725700 1 -- 172.21.15.2:0/140847292 --> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f62b81032c0 con 0x7f62a0038760 2023-11-16T16:40:20.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.228+0000 7f629ffff700 1 -- 172.21.15.2:0/140847292 <== mgr.14118 v2:172.21.15.2:6800/3359672074 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f62b81032c0 con 0x7f62a0038760 2023-11-16T16:40:20.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.230+0000 7f62bd725700 1 -- 172.21.15.2:0/140847292 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f62a0038760 msgr2=0x7f62a003ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:20.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.230+0000 7f62bd725700 1 --2- 172.21.15.2:0/140847292 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f62a0038760 0x7f62a003ac10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f62ac006fd0 tx=0x7f62ac006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:20.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.230+0000 7f62bd725700 1 -- 172.21.15.2:0/140847292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62b807be70 msgr2=0x7f62b819b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:20.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.230+0000 7f62bd725700 1 --2- 172.21.15.2:0/140847292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62b807be70 0x7f62b819b9b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f62a8006af0 tx=0x7f62a8011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:20.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.230+0000 7f62bd725700 1 -- 172.21.15.2:0/140847292 shutdown_connections 2023-11-16T16:40:20.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.230+0000 7f62bd725700 1 --2- 172.21.15.2:0/140847292 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f62a0038760 0x7f62a003ac10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:20.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.230+0000 7f62bd725700 1 --2- 172.21.15.2:0/140847292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62b807be70 0x7f62b819b9b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:20.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.230+0000 7f62bd725700 1 -- 172.21.15.2:0/140847292 >> 172.21.15.2:0/140847292 conn(0x7f62b8100fd0 msgr2=0x7f62b81026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:20.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.230+0000 7f62bd725700 1 -- 172.21.15.2:0/140847292 shutdown_connections 2023-11-16T16:40:20.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:20.230+0000 7f62bd725700 1 -- 172.21.15.2:0/140847292 wait complete. 2023-11-16T16:40:20.814 INFO:teuthology.orchestra.run.smithi002.stdout:Generating ssh key... 2023-11-16T16:40:21.058 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:20 smithi002 ceph-mon[128712]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:40:22.156 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:21 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: Generating public/private rsa key pair. 2023-11-16T16:40:22.156 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:21 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: Your identification has been saved in /tmp/tmp5wmf2uo3/key. 2023-11-16T16:40:22.156 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:21 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: Your public key has been saved in /tmp/tmp5wmf2uo3/key.pub. 2023-11-16T16:40:22.156 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:21 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: The key fingerprint is: 2023-11-16T16:40:22.156 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:21 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: SHA256:rpWCAOAGxGLE2gkpf2NU5Os5PvpFwrgyJR+H38M3fqI ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:40:22.156 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:21 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: The key's randomart image is: 2023-11-16T16:40:22.156 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:21 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: +---[RSA 3072]----+ 2023-11-16T16:40:22.156 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:21 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: |B+ oo | 2023-11-16T16:40:22.156 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:21 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: |O+ .. | 2023-11-16T16:40:22.157 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:21 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: |** o . | 2023-11-16T16:40:22.157 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:21 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: |o.+ ++ . | 2023-11-16T16:40:22.157 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:21 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: | oo=.= S | 2023-11-16T16:40:22.157 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:21 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: | = B B . | 2023-11-16T16:40:22.157 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:21 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: | o + * O o | 2023-11-16T16:40:22.157 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:21 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: | o ..* +... | 2023-11-16T16:40:22.157 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:21 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: | .o+.E..o | 2023-11-16T16:40:22.157 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:21 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: +----[SHA256]-----+ 2023-11-16T16:40:22.291 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.315+0000 7f99c15af700 1 Processor -- start 2023-11-16T16:40:22.291 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.315+0000 7f99c15af700 1 -- start start 2023-11-16T16:40:22.291 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.315+0000 7f99c15af700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99bc07be30 0x7f99bc07a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:22.291 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.315+0000 7f99c15af700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc07a860 con 0x7f99bc07be30 2023-11-16T16:40:22.291 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.315+0000 7f99baffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99bc07be30 0x7f99bc07a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:22.291 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.315+0000 7f99baffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99bc07be30 0x7f99bc07a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47886/0 (socket says 172.21.15.2:47886) 2023-11-16T16:40:22.291 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.315+0000 7f99baffd700 1 -- 172.21.15.2:0/459246690 learned_addr learned my addr 172.21.15.2:0/459246690 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:22.291 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.316+0000 7f99baffd700 1 -- 172.21.15.2:0/459246690 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99bc07a9a0 con 0x7f99bc07be30 2023-11-16T16:40:22.292 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.316+0000 7f99baffd700 1 --2- 172.21.15.2:0/459246690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99bc07be30 0x7f99bc07a290 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f99ac0098a0 tx=0x7f99ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6967053a2faf14c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:22.292 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.316+0000 7f99b9ffb700 1 -- 172.21.15.2:0/459246690 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f99ac004160 con 0x7f99bc07be30 2023-11-16T16:40:22.292 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.316+0000 7f99b9ffb700 1 -- 172.21.15.2:0/459246690 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99ac0042c0 con 0x7f99bc07be30 2023-11-16T16:40:22.292 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.316+0000 7f99b9ffb700 1 -- 172.21.15.2:0/459246690 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f99ac011a00 con 0x7f99bc07be30 2023-11-16T16:40:22.292 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.317+0000 7f99c15af700 1 -- 172.21.15.2:0/459246690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99bc07be30 msgr2=0x7f99bc07a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:22.292 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.317+0000 7f99c15af700 1 --2- 172.21.15.2:0/459246690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99bc07be30 0x7f99bc07a290 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f99ac0098a0 tx=0x7f99ac009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:22.292 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.317+0000 7f99c15af700 1 -- 172.21.15.2:0/459246690 shutdown_connections 2023-11-16T16:40:22.292 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.317+0000 7f99c15af700 1 --2- 172.21.15.2:0/459246690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99bc07be30 0x7f99bc07a290 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:22.292 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.317+0000 7f99c15af700 1 -- 172.21.15.2:0/459246690 >> 172.21.15.2:0/459246690 conn(0x7f99bc100fd0 msgr2=0x7f99bc103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:22.292 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.317+0000 7f99c15af700 1 -- 172.21.15.2:0/459246690 shutdown_connections 2023-11-16T16:40:22.292 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.317+0000 7f99c15af700 1 -- 172.21.15.2:0/459246690 wait complete. 2023-11-16T16:40:22.293 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.318+0000 7f99c15af700 1 Processor -- start 2023-11-16T16:40:22.293 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.318+0000 7f99c15af700 1 -- start start 2023-11-16T16:40:22.293 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.319+0000 7f99c15af700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99bc07be30 0x7f99bc19b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:22.293 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.319+0000 7f99c15af700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc19be70 con 0x7f99bc07be30 2023-11-16T16:40:22.293 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.319+0000 7f99baffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99bc07be30 0x7f99bc19b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:22.293 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.319+0000 7f99baffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99bc07be30 0x7f99bc19b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47892/0 (socket says 172.21.15.2:47892) 2023-11-16T16:40:22.293 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.319+0000 7f99baffd700 1 -- 172.21.15.2:0/3025476374 learned_addr learned my addr 172.21.15.2:0/3025476374 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:22.293 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.319+0000 7f99baffd700 1 -- 172.21.15.2:0/3025476374 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ac009550 con 0x7f99bc07be30 2023-11-16T16:40:22.293 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.319+0000 7f99baffd700 1 --2- 172.21.15.2:0/3025476374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99bc07be30 0x7f99bc19b930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f99ac006af0 tx=0x7f99ac011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:22.293 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.320+0000 7f99a3fff700 1 -- 172.21.15.2:0/3025476374 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f99ac003a40 con 0x7f99bc07be30 2023-11-16T16:40:22.293 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.320+0000 7f99a3fff700 1 -- 172.21.15.2:0/3025476374 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99ac003ba0 con 0x7f99bc07be30 2023-11-16T16:40:22.293 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.320+0000 7f99c15af700 1 -- 172.21.15.2:0/3025476374 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99bc10bf40 con 0x7f99bc07be30 2023-11-16T16:40:22.294 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.320+0000 7f99a3fff700 1 -- 172.21.15.2:0/3025476374 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f99ac01a6e0 con 0x7f99bc07be30 2023-11-16T16:40:22.294 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.320+0000 7f99c15af700 1 -- 172.21.15.2:0/3025476374 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99bc10bdb0 con 0x7f99bc07be30 2023-11-16T16:40:22.294 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.321+0000 7f99a3fff700 1 -- 172.21.15.2:0/3025476374 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f99ac024430 con 0x7f99bc07be30 2023-11-16T16:40:22.294 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.321+0000 7f99c15af700 1 -- 172.21.15.2:0/3025476374 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99bc062120 con 0x7f99bc07be30 2023-11-16T16:40:22.294 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.321+0000 7f99a3fff700 1 --2- 172.21.15.2:0/3025476374 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f99a40387b0 0x7f99a403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:22.294 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.321+0000 7f99a3fff700 1 -- 172.21.15.2:0/3025476374 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f99ac04c7b0 con 0x7f99bc07be30 2023-11-16T16:40:22.294 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.322+0000 7f99ba7fc700 1 --2- 172.21.15.2:0/3025476374 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f99a40387b0 0x7f99a403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:22.294 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.322+0000 7f99ba7fc700 1 --2- 172.21.15.2:0/3025476374 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f99a40387b0 0x7f99a403ac60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f99b0006fd0 tx=0x7f99b0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:22.294 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.327+0000 7f99a3fff700 1 -- 172.21.15.2:0/3025476374 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f99ac0246e0 con 0x7f99bc07be30 2023-11-16T16:40:22.294 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.533+0000 7f99c15af700 1 -- 172.21.15.2:0/3025476374 --> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f99bc103300 con 0x7f99a40387b0 2023-11-16T16:40:22.294 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.843+0000 7f99a3fff700 1 -- 172.21.15.2:0/3025476374 <== mgr.14118 v2:172.21.15.2:6800/3359672074 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f99bc103300 con 0x7f99a40387b0 2023-11-16T16:40:22.295 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.845+0000 7f99c15af700 1 -- 172.21.15.2:0/3025476374 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f99a40387b0 msgr2=0x7f99a403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:22.295 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.845+0000 7f99c15af700 1 --2- 172.21.15.2:0/3025476374 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f99a40387b0 0x7f99a403ac60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f99b0006fd0 tx=0x7f99b0006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:22.295 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.845+0000 7f99c15af700 1 -- 172.21.15.2:0/3025476374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99bc07be30 msgr2=0x7f99bc19b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:22.295 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.845+0000 7f99c15af700 1 --2- 172.21.15.2:0/3025476374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99bc07be30 0x7f99bc19b930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f99ac006af0 tx=0x7f99ac011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:22.295 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.845+0000 7f99c15af700 1 -- 172.21.15.2:0/3025476374 shutdown_connections 2023-11-16T16:40:22.295 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.845+0000 7f99c15af700 1 --2- 172.21.15.2:0/3025476374 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f99a40387b0 0x7f99a403ac60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:22.295 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.845+0000 7f99c15af700 1 --2- 172.21.15.2:0/3025476374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99bc07be30 0x7f99bc19b930 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:22.295 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.845+0000 7f99c15af700 1 -- 172.21.15.2:0/3025476374 >> 172.21.15.2:0/3025476374 conn(0x7f99bc100fd0 msgr2=0x7f99bc1026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:22.295 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.845+0000 7f99c15af700 1 -- 172.21.15.2:0/3025476374 shutdown_connections 2023-11-16T16:40:22.295 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:21.845+0000 7f99c15af700 1 -- 172.21.15.2:0/3025476374 wait complete. 2023-11-16T16:40:23.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:22 smithi002 ceph-mon[128712]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:40:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:22 smithi002 ceph-mon[128712]: Generating ssh key... 2023-11-16T16:40:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:22 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' 2023-11-16T16:40:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:22 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' 2023-11-16T16:40:23.775 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEM+zcML4+7BH9HF+gbIskc3Kc2Zw5Ad20Rg2zIyDbyMH8kdv842x+q+kSZ3cUA9IPdWUACpmLw222KVqJOA0oAkYuGonTieJ9Sdbrji5YlZ3FmJIA/yCQhHoBb1LGKQ2Nj19fls6loCtpzgJ21ZzSGShN1uBqTjrIE4pP4eHqWo9nzClTZ/RAY0TG778sExqR/P6V4opBXpxZeBB1oXB4bgGCpZmSDgzXqryAWkyUpMXBw/osnZTur8EfLUOU9erIU860MxqZ/scLw3MTMx7nuhQvxGdTBoEGoEqF771zmzFVSrQdK6PbY+aw+6pkqutHtA1DkrZG7+4Bn9AgPkWGa5MmGO+2v6DlYJuSzTyehZgJsbaWsV9KW2ArcfMgzuQ0N9y7muAaToA/DBdtn8msHA8F1de3iHRFOG9eHttq+KI9Pmd0thHj5eaL6MqClzddtkX5te+4nekAlT59iPeMGCTOEx2goIIsQ+bUn7pCSAu3cJFLRY9nIV8vocR9lX8= ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:40:23.775 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.005+0000 7f54d8219700 1 Processor -- start 2023-11-16T16:40:23.775 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.005+0000 7f54d8219700 1 -- start start 2023-11-16T16:40:23.775 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.006+0000 7f54d8219700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54d007be10 0x7f54d007a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:23.775 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.006+0000 7f54d8219700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d007a840 con 0x7f54d007be10 2023-11-16T16:40:23.775 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.006+0000 7f54d5fb5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54d007be10 0x7f54d007a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:23.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.006+0000 7f54d5fb5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54d007be10 0x7f54d007a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47908/0 (socket says 172.21.15.2:47908) 2023-11-16T16:40:23.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.006+0000 7f54d5fb5700 1 -- 172.21.15.2:0/885139408 learned_addr learned my addr 172.21.15.2:0/885139408 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:23.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.006+0000 7f54d5fb5700 1 -- 172.21.15.2:0/885139408 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d007a980 con 0x7f54d007be10 2023-11-16T16:40:23.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.007+0000 7f54d5fb5700 1 --2- 172.21.15.2:0/885139408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54d007be10 0x7f54d007a270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0098a0 tx=0x7f54cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7163a296c30556b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:23.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.007+0000 7f54d4fb3700 1 -- 172.21.15.2:0/885139408 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54cc004160 con 0x7f54d007be10 2023-11-16T16:40:23.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.007+0000 7f54d4fb3700 1 -- 172.21.15.2:0/885139408 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54cc0042c0 con 0x7f54d007be10 2023-11-16T16:40:23.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.007+0000 7f54d4fb3700 1 -- 172.21.15.2:0/885139408 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54cc011a00 con 0x7f54d007be10 2023-11-16T16:40:23.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.007+0000 7f54d8219700 1 -- 172.21.15.2:0/885139408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54d007be10 msgr2=0x7f54d007a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:23.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.007+0000 7f54d8219700 1 --2- 172.21.15.2:0/885139408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54d007be10 0x7f54d007a270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0098a0 tx=0x7f54cc009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:23.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.007+0000 7f54d8219700 1 -- 172.21.15.2:0/885139408 shutdown_connections 2023-11-16T16:40:23.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.007+0000 7f54d8219700 1 --2- 172.21.15.2:0/885139408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54d007be10 0x7f54d007a270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:23.776 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.008+0000 7f54d8219700 1 -- 172.21.15.2:0/885139408 >> 172.21.15.2:0/885139408 conn(0x7f54d0100fd0 msgr2=0x7f54d0103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:23.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.008+0000 7f54d8219700 1 -- 172.21.15.2:0/885139408 shutdown_connections 2023-11-16T16:40:23.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.008+0000 7f54d8219700 1 -- 172.21.15.2:0/885139408 wait complete. 2023-11-16T16:40:23.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.009+0000 7f54d8219700 1 Processor -- start 2023-11-16T16:40:23.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.009+0000 7f54d8219700 1 -- start start 2023-11-16T16:40:23.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.009+0000 7f54d8219700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54d007be10 0x7f54d019b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:23.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.009+0000 7f54d8219700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d019be90 con 0x7f54d007be10 2023-11-16T16:40:23.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.009+0000 7f54d5fb5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54d007be10 0x7f54d019b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:23.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.009+0000 7f54d5fb5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54d007be10 0x7f54d019b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47912/0 (socket says 172.21.15.2:47912) 2023-11-16T16:40:23.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.009+0000 7f54d5fb5700 1 -- 172.21.15.2:0/76727526 learned_addr learned my addr 172.21.15.2:0/76727526 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:23.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.010+0000 7f54d5fb5700 1 -- 172.21.15.2:0/76727526 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54cc009550 con 0x7f54d007be10 2023-11-16T16:40:23.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.010+0000 7f54d5fb5700 1 --2- 172.21.15.2:0/76727526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54d007be10 0x7f54d019b950 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f54cc006af0 tx=0x7f54cc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:23.777 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.010+0000 7f54c6ffd700 1 -- 172.21.15.2:0/76727526 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54cc003a40 con 0x7f54d007be10 2023-11-16T16:40:23.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.010+0000 7f54c6ffd700 1 -- 172.21.15.2:0/76727526 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54cc003ba0 con 0x7f54d007be10 2023-11-16T16:40:23.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.010+0000 7f54c6ffd700 1 -- 172.21.15.2:0/76727526 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54cc01a6e0 con 0x7f54d007be10 2023-11-16T16:40:23.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.010+0000 7f54d8219700 1 -- 172.21.15.2:0/76727526 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d010bf60 con 0x7f54d007be10 2023-11-16T16:40:23.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.010+0000 7f54d8219700 1 -- 172.21.15.2:0/76727526 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d010bdd0 con 0x7f54d007be10 2023-11-16T16:40:23.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.011+0000 7f54d8219700 1 -- 172.21.15.2:0/76727526 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d0062120 con 0x7f54d007be10 2023-11-16T16:40:23.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.012+0000 7f54c6ffd700 1 -- 172.21.15.2:0/76727526 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f54cc024a60 con 0x7f54d007be10 2023-11-16T16:40:23.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.012+0000 7f54c6ffd700 1 --2- 172.21.15.2:0/76727526 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f54bc038760 0x7f54bc03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:23.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.013+0000 7f54c6ffd700 1 -- 172.21.15.2:0/76727526 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f54cc04c720 con 0x7f54d007be10 2023-11-16T16:40:23.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.013+0000 7f54d57b4700 1 --2- 172.21.15.2:0/76727526 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f54bc038760 0x7f54bc03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:23.778 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.013+0000 7f54d57b4700 1 --2- 172.21.15.2:0/76727526 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f54bc038760 0x7f54bc03ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f54c0006fd0 tx=0x7f54c0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:23.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.017+0000 7f54c6ffd700 1 -- 172.21.15.2:0/76727526 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f54cc024430 con 0x7f54d007be10 2023-11-16T16:40:23.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.226+0000 7f54d8219700 1 -- 172.21.15.2:0/76727526 --> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f54d01032c0 con 0x7f54bc038760 2023-11-16T16:40:23.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.226+0000 7f54c6ffd700 1 -- 172.21.15.2:0/76727526 <== mgr.14118 v2:172.21.15.2:6800/3359672074 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f54d01032c0 con 0x7f54bc038760 2023-11-16T16:40:23.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.229+0000 7f54d8219700 1 -- 172.21.15.2:0/76727526 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f54bc038760 msgr2=0x7f54bc03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:23.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.229+0000 7f54d8219700 1 --2- 172.21.15.2:0/76727526 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f54bc038760 0x7f54bc03ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f54c0006fd0 tx=0x7f54c0006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:23.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.229+0000 7f54d8219700 1 -- 172.21.15.2:0/76727526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54d007be10 msgr2=0x7f54d019b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:23.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.229+0000 7f54d8219700 1 --2- 172.21.15.2:0/76727526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54d007be10 0x7f54d019b950 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f54cc006af0 tx=0x7f54cc011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:23.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.229+0000 7f54d8219700 1 -- 172.21.15.2:0/76727526 shutdown_connections 2023-11-16T16:40:23.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.229+0000 7f54d8219700 1 --2- 172.21.15.2:0/76727526 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f54bc038760 0x7f54bc03ac10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:23.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.229+0000 7f54d8219700 1 --2- 172.21.15.2:0/76727526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f54d007be10 0x7f54d019b950 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:23.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.229+0000 7f54d8219700 1 -- 172.21.15.2:0/76727526 >> 172.21.15.2:0/76727526 conn(0x7f54d0100fd0 msgr2=0x7f54d01026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:23.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.229+0000 7f54d8219700 1 -- 172.21.15.2:0/76727526 shutdown_connections 2023-11-16T16:40:23.780 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:23.229+0000 7f54d8219700 1 -- 172.21.15.2:0/76727526 wait complete. 2023-11-16T16:40:23.780 INFO:teuthology.orchestra.run.smithi002.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-11-16T16:40:23.780 INFO:teuthology.orchestra.run.smithi002.stdout:Adding key to root@localhost authorized_keys... 2023-11-16T16:40:23.780 INFO:teuthology.orchestra.run.smithi002.stdout:Adding host smithi002... 2023-11-16T16:40:24.709 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:24 smithi002 ceph-mon[128712]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:40:25.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:25 smithi002 ceph-mon[128712]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi002", "addr": "172.21.15.2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:40:26.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:26 smithi002 ceph-mon[128712]: Deploying cephadm binary to smithi002 2023-11-16T16:40:28.069 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Added host 'smithi002' with addr '172.21.15.2' 2023-11-16T16:40:28.069 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.360+0000 7fd0f48d3700 1 Processor -- start 2023-11-16T16:40:28.069 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.361+0000 7fd0f48d3700 1 -- start start 2023-11-16T16:40:28.069 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.361+0000 7fd0f48d3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0f007be10 0x7fd0f007a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:28.069 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.361+0000 7fd0f48d3700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f007a840 con 0x7fd0f007be10 2023-11-16T16:40:28.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.361+0000 7fd0ee59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0f007be10 0x7fd0f007a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:28.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.361+0000 7fd0ee59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0f007be10 0x7fd0f007a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55252/0 (socket says 172.21.15.2:55252) 2023-11-16T16:40:28.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.361+0000 7fd0ee59c700 1 -- 172.21.15.2:0/181198713 learned_addr learned my addr 172.21.15.2:0/181198713 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:28.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.361+0000 7fd0ee59c700 1 -- 172.21.15.2:0/181198713 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f007a980 con 0x7fd0f007be10 2023-11-16T16:40:28.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.362+0000 7fd0ee59c700 1 --2- 172.21.15.2:0/181198713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0f007be10 0x7fd0f007a270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd0e00098a0 tx=0x7fd0e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce0ff3d03a634dfb server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:28.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.362+0000 7fd0ed59a700 1 -- 172.21.15.2:0/181198713 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0e0004160 con 0x7fd0f007be10 2023-11-16T16:40:28.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.362+0000 7fd0ed59a700 1 -- 172.21.15.2:0/181198713 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd0e00042c0 con 0x7fd0f007be10 2023-11-16T16:40:28.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.362+0000 7fd0ed59a700 1 -- 172.21.15.2:0/181198713 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0e0011a00 con 0x7fd0f007be10 2023-11-16T16:40:28.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.363+0000 7fd0f48d3700 1 -- 172.21.15.2:0/181198713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0f007be10 msgr2=0x7fd0f007a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:28.070 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.363+0000 7fd0f48d3700 1 --2- 172.21.15.2:0/181198713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0f007be10 0x7fd0f007a270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd0e00098a0 tx=0x7fd0e0009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:28.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.363+0000 7fd0f48d3700 1 -- 172.21.15.2:0/181198713 shutdown_connections 2023-11-16T16:40:28.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.363+0000 7fd0f48d3700 1 --2- 172.21.15.2:0/181198713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0f007be10 0x7fd0f007a270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:28.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.363+0000 7fd0f48d3700 1 -- 172.21.15.2:0/181198713 >> 172.21.15.2:0/181198713 conn(0x7fd0f0100fd0 msgr2=0x7fd0f0103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:28.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.363+0000 7fd0f48d3700 1 -- 172.21.15.2:0/181198713 shutdown_connections 2023-11-16T16:40:28.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.363+0000 7fd0f48d3700 1 -- 172.21.15.2:0/181198713 wait complete. 2023-11-16T16:40:28.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.364+0000 7fd0f48d3700 1 Processor -- start 2023-11-16T16:40:28.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.364+0000 7fd0f48d3700 1 -- start start 2023-11-16T16:40:28.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.365+0000 7fd0f48d3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0f007be10 0x7fd0f019b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:28.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.365+0000 7fd0f48d3700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f019be90 con 0x7fd0f007be10 2023-11-16T16:40:28.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.365+0000 7fd0ee59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0f007be10 0x7fd0f019b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:28.071 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.365+0000 7fd0ee59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0f007be10 0x7fd0f019b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55258/0 (socket says 172.21.15.2:55258) 2023-11-16T16:40:28.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.365+0000 7fd0ee59c700 1 -- 172.21.15.2:0/2351110568 learned_addr learned my addr 172.21.15.2:0/2351110568 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:28.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.365+0000 7fd0ee59c700 1 -- 172.21.15.2:0/2351110568 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e0009550 con 0x7fd0f007be10 2023-11-16T16:40:28.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.365+0000 7fd0ee59c700 1 --2- 172.21.15.2:0/2351110568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0f007be10 0x7fd0f019b950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0006af0 tx=0x7fd0e0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:28.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.366+0000 7fd0df7fe700 1 -- 172.21.15.2:0/2351110568 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0e0003a40 con 0x7fd0f007be10 2023-11-16T16:40:28.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.366+0000 7fd0df7fe700 1 -- 172.21.15.2:0/2351110568 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd0e0003ba0 con 0x7fd0f007be10 2023-11-16T16:40:28.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.366+0000 7fd0df7fe700 1 -- 172.21.15.2:0/2351110568 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0e001a6e0 con 0x7fd0f007be10 2023-11-16T16:40:28.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.366+0000 7fd0f48d3700 1 -- 172.21.15.2:0/2351110568 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f010bf60 con 0x7fd0f007be10 2023-11-16T16:40:28.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.366+0000 7fd0f48d3700 1 -- 172.21.15.2:0/2351110568 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f010bdd0 con 0x7fd0f007be10 2023-11-16T16:40:28.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.367+0000 7fd0f48d3700 1 -- 172.21.15.2:0/2351110568 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f010c0f0 con 0x7fd0f007be10 2023-11-16T16:40:28.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.367+0000 7fd0df7fe700 1 -- 172.21.15.2:0/2351110568 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fd0e0024a60 con 0x7fd0f007be10 2023-11-16T16:40:28.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.367+0000 7fd0df7fe700 1 --2- 172.21.15.2:0/2351110568 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fd0d8038760 0x7fd0d803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:28.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.367+0000 7fd0df7fe700 1 -- 172.21.15.2:0/2351110568 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd0e004c7c0 con 0x7fd0f007be10 2023-11-16T16:40:28.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.368+0000 7fd0edd9b700 1 --2- 172.21.15.2:0/2351110568 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fd0d8038760 0x7fd0d803ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:28.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.368+0000 7fd0edd9b700 1 --2- 172.21.15.2:0/2351110568 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fd0d8038760 0x7fd0d803ac10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4006fd0 tx=0x7fd0e4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:28.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.373+0000 7fd0df7fe700 1 -- 172.21.15.2:0/2351110568 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd0e0024430 con 0x7fd0f007be10 2023-11-16T16:40:28.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:24.585+0000 7fd0f48d3700 1 -- 172.21.15.2:0/2351110568 --> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi002", "addr": "172.21.15.2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0f01032c0 con 0x7fd0d8038760 2023-11-16T16:40:28.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.678+0000 7fd0df7fe700 1 -- 172.21.15.2:0/2351110568 <== mgr.14118 v2:172.21.15.2:6800/3359672074 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fd0f01032c0 con 0x7fd0d8038760 2023-11-16T16:40:28.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.681+0000 7fd0f48d3700 1 -- 172.21.15.2:0/2351110568 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fd0d8038760 msgr2=0x7fd0d803ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:28.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.681+0000 7fd0f48d3700 1 --2- 172.21.15.2:0/2351110568 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fd0d8038760 0x7fd0d803ac10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4006fd0 tx=0x7fd0e4006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:28.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.681+0000 7fd0f48d3700 1 -- 172.21.15.2:0/2351110568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0f007be10 msgr2=0x7fd0f019b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:28.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.681+0000 7fd0f48d3700 1 --2- 172.21.15.2:0/2351110568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0f007be10 0x7fd0f019b950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0006af0 tx=0x7fd0e0011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:28.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.681+0000 7fd0f48d3700 1 -- 172.21.15.2:0/2351110568 shutdown_connections 2023-11-16T16:40:28.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.681+0000 7fd0f48d3700 1 --2- 172.21.15.2:0/2351110568 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fd0d8038760 0x7fd0d803ac10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:28.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.681+0000 7fd0f48d3700 1 --2- 172.21.15.2:0/2351110568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0f007be10 0x7fd0f019b950 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:28.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.681+0000 7fd0f48d3700 1 -- 172.21.15.2:0/2351110568 >> 172.21.15.2:0/2351110568 conn(0x7fd0f0100fd0 msgr2=0x7fd0f01026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:28.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.682+0000 7fd0f48d3700 1 -- 172.21.15.2:0/2351110568 shutdown_connections 2023-11-16T16:40:28.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:27.682+0000 7fd0f48d3700 1 -- 172.21.15.2:0/2351110568 wait complete. 2023-11-16T16:40:28.074 INFO:teuthology.orchestra.run.smithi002.stdout:Deploying unmanaged mon service... 2023-11-16T16:40:29.021 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:28 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' 2023-11-16T16:40:29.021 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:28 smithi002 ceph-mon[128712]: Added host smithi002 2023-11-16T16:40:29.021 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:28 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:29.265 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-11-16T16:40:29.265 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.634+0000 7f3bcb77a700 1 Processor -- start 2023-11-16T16:40:29.265 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.634+0000 7f3bcb77a700 1 -- start start 2023-11-16T16:40:29.265 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.634+0000 7f3bcb77a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bc407be30 0x7f3bc407a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:29.265 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.634+0000 7f3bcb77a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc407a860 con 0x7f3bc407be30 2023-11-16T16:40:29.265 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.634+0000 7f3bc9516700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bc407be30 0x7f3bc407a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:29.266 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.634+0000 7f3bc9516700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bc407be30 0x7f3bc407a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55262/0 (socket says 172.21.15.2:55262) 2023-11-16T16:40:29.266 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.634+0000 7f3bc9516700 1 -- 172.21.15.2:0/3849491066 learned_addr learned my addr 172.21.15.2:0/3849491066 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:29.266 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.635+0000 7f3bc9516700 1 -- 172.21.15.2:0/3849491066 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc407a9a0 con 0x7f3bc407be30 2023-11-16T16:40:29.266 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.635+0000 7f3bc9516700 1 --2- 172.21.15.2:0/3849491066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bc407be30 0x7f3bc407a290 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3bc00098a0 tx=0x7f3bc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f07f1a158f1a538a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:29.266 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.635+0000 7f3bbbfff700 1 -- 172.21.15.2:0/3849491066 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bc0004160 con 0x7f3bc407be30 2023-11-16T16:40:29.266 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.635+0000 7f3bbbfff700 1 -- 172.21.15.2:0/3849491066 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3bc00042c0 con 0x7f3bc407be30 2023-11-16T16:40:29.266 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.635+0000 7f3bbbfff700 1 -- 172.21.15.2:0/3849491066 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bc0011a00 con 0x7f3bc407be30 2023-11-16T16:40:29.266 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.636+0000 7f3bcb77a700 1 -- 172.21.15.2:0/3849491066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bc407be30 msgr2=0x7f3bc407a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:29.266 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.636+0000 7f3bcb77a700 1 --2- 172.21.15.2:0/3849491066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bc407be30 0x7f3bc407a290 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3bc00098a0 tx=0x7f3bc0009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:29.266 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.636+0000 7f3bcb77a700 1 -- 172.21.15.2:0/3849491066 shutdown_connections 2023-11-16T16:40:29.266 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.636+0000 7f3bcb77a700 1 --2- 172.21.15.2:0/3849491066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bc407be30 0x7f3bc407a290 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:29.266 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.636+0000 7f3bcb77a700 1 -- 172.21.15.2:0/3849491066 >> 172.21.15.2:0/3849491066 conn(0x7f3bc4100fd0 msgr2=0x7f3bc4103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:29.267 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.636+0000 7f3bcb77a700 1 -- 172.21.15.2:0/3849491066 shutdown_connections 2023-11-16T16:40:29.267 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.636+0000 7f3bcb77a700 1 -- 172.21.15.2:0/3849491066 wait complete. 2023-11-16T16:40:29.267 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.637+0000 7f3bcb77a700 1 Processor -- start 2023-11-16T16:40:29.267 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.637+0000 7f3bcb77a700 1 -- start start 2023-11-16T16:40:29.267 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.638+0000 7f3bcb77a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bc407be30 0x7f3bc419b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:29.267 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.638+0000 7f3bcb77a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc419be70 con 0x7f3bc407be30 2023-11-16T16:40:29.267 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.638+0000 7f3bc9516700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bc407be30 0x7f3bc419b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:29.267 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.638+0000 7f3bc9516700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bc407be30 0x7f3bc419b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55268/0 (socket says 172.21.15.2:55268) 2023-11-16T16:40:29.267 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.638+0000 7f3bc9516700 1 -- 172.21.15.2:0/1884958258 learned_addr learned my addr 172.21.15.2:0/1884958258 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:29.267 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.638+0000 7f3bc9516700 1 -- 172.21.15.2:0/1884958258 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc0009550 con 0x7f3bc407be30 2023-11-16T16:40:29.267 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.639+0000 7f3bc9516700 1 --2- 172.21.15.2:0/1884958258 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bc407be30 0x7f3bc419b930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0006af0 tx=0x7f3bc0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:29.268 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.639+0000 7f3bba7fc700 1 -- 172.21.15.2:0/1884958258 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bc0003a40 con 0x7f3bc407be30 2023-11-16T16:40:29.268 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.639+0000 7f3bba7fc700 1 -- 172.21.15.2:0/1884958258 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3bc0003ba0 con 0x7f3bc407be30 2023-11-16T16:40:29.268 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.639+0000 7f3bba7fc700 1 -- 172.21.15.2:0/1884958258 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bc001a6e0 con 0x7f3bc407be30 2023-11-16T16:40:29.268 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.639+0000 7f3bcb77a700 1 -- 172.21.15.2:0/1884958258 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc410bf40 con 0x7f3bc407be30 2023-11-16T16:40:29.268 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.639+0000 7f3bcb77a700 1 -- 172.21.15.2:0/1884958258 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bc410bdb0 con 0x7f3bc407be30 2023-11-16T16:40:29.268 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.640+0000 7f3bcb77a700 1 -- 172.21.15.2:0/1884958258 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bc4062120 con 0x7f3bc407be30 2023-11-16T16:40:29.268 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.640+0000 7f3bba7fc700 1 -- 172.21.15.2:0/1884958258 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f3bc0024a60 con 0x7f3bc407be30 2023-11-16T16:40:29.268 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.641+0000 7f3bba7fc700 1 --2- 172.21.15.2:0/1884958258 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f3bb0038760 0x7f3bb003ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:29.268 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.641+0000 7f3bba7fc700 1 -- 172.21.15.2:0/1884958258 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f3bc004c7c0 con 0x7f3bc407be30 2023-11-16T16:40:29.268 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.641+0000 7f3bc8d15700 1 --2- 172.21.15.2:0/1884958258 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f3bb0038760 0x7f3bb003ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:29.268 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.642+0000 7f3bc8d15700 1 --2- 172.21.15.2:0/1884958258 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f3bb0038760 0x7f3bb003ac10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4006fd0 tx=0x7f3bb4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:29.269 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.646+0000 7f3bba7fc700 1 -- 172.21.15.2:0/1884958258 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3bc0024430 con 0x7f3bc407be30 2023-11-16T16:40:29.269 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.867+0000 7f3bcb77a700 1 -- 172.21.15.2:0/1884958258 --> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f3bc4103300 con 0x7f3bb0038760 2023-11-16T16:40:29.269 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.873+0000 7f3bba7fc700 1 -- 172.21.15.2:0/1884958258 <== mgr.14118 v2:172.21.15.2:6800/3359672074 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f3bc4103300 con 0x7f3bb0038760 2023-11-16T16:40:29.269 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.875+0000 7f3bcb77a700 1 -- 172.21.15.2:0/1884958258 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f3bb0038760 msgr2=0x7f3bb003ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:29.269 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.875+0000 7f3bcb77a700 1 --2- 172.21.15.2:0/1884958258 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f3bb0038760 0x7f3bb003ac10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4006fd0 tx=0x7f3bb4006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:29.269 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.875+0000 7f3bcb77a700 1 -- 172.21.15.2:0/1884958258 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bc407be30 msgr2=0x7f3bc419b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:29.269 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.875+0000 7f3bcb77a700 1 --2- 172.21.15.2:0/1884958258 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bc407be30 0x7f3bc419b930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0006af0 tx=0x7f3bc0011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:29.269 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.875+0000 7f3bcb77a700 1 -- 172.21.15.2:0/1884958258 shutdown_connections 2023-11-16T16:40:29.269 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.875+0000 7f3bcb77a700 1 --2- 172.21.15.2:0/1884958258 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f3bb0038760 0x7f3bb003ac10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:29.269 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.875+0000 7f3bcb77a700 1 --2- 172.21.15.2:0/1884958258 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bc407be30 0x7f3bc419b930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:29.270 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.875+0000 7f3bcb77a700 1 -- 172.21.15.2:0/1884958258 >> 172.21.15.2:0/1884958258 conn(0x7f3bc4100fd0 msgr2=0x7f3bc41026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:29.270 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.875+0000 7f3bcb77a700 1 -- 172.21.15.2:0/1884958258 shutdown_connections 2023-11-16T16:40:29.270 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:28.875+0000 7f3bcb77a700 1 -- 172.21.15.2:0/1884958258 wait complete. 2023-11-16T16:40:29.270 INFO:teuthology.orchestra.run.smithi002.stdout:Deploying unmanaged mgr service... 2023-11-16T16:40:30.021 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:29 smithi002 ceph-mon[128712]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:40:30.021 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:29 smithi002 ceph-mon[128712]: Saving service mon spec with placement count:5 2023-11-16T16:40:30.021 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:29 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' 2023-11-16T16:40:30.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-11-16T16:40:30.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.100+0000 7f891a06f700 1 Processor -- start 2023-11-16T16:40:30.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.100+0000 7f891a06f700 1 -- start start 2023-11-16T16:40:30.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.101+0000 7f891a06f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f891407be70 0x7f891407a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:30.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.101+0000 7f891a06f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891407a8a0 con 0x7f891407be70 2023-11-16T16:40:30.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.101+0000 7f89137fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f891407be70 0x7f891407a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:30.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.101+0000 7f89137fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f891407be70 0x7f891407a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55282/0 (socket says 172.21.15.2:55282) 2023-11-16T16:40:30.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.101+0000 7f89137fe700 1 -- 172.21.15.2:0/719078163 learned_addr learned my addr 172.21.15.2:0/719078163 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:30.779 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.101+0000 7f89137fe700 1 -- 172.21.15.2:0/719078163 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891407a9e0 con 0x7f891407be70 2023-11-16T16:40:30.780 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.102+0000 7f89137fe700 1 --2- 172.21.15.2:0/719078163 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f891407be70 0x7f891407a2d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f89040098a0 tx=0x7f8904009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6756258e742078ef server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:30.780 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.102+0000 7f89127fc700 1 -- 172.21.15.2:0/719078163 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8904004160 con 0x7f891407be70 2023-11-16T16:40:30.780 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.102+0000 7f89127fc700 1 -- 172.21.15.2:0/719078163 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f89040042c0 con 0x7f891407be70 2023-11-16T16:40:30.780 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.102+0000 7f89127fc700 1 -- 172.21.15.2:0/719078163 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8904011a00 con 0x7f891407be70 2023-11-16T16:40:30.780 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.103+0000 7f891a06f700 1 -- 172.21.15.2:0/719078163 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f891407be70 msgr2=0x7f891407a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:30.780 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.103+0000 7f891a06f700 1 --2- 172.21.15.2:0/719078163 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f891407be70 0x7f891407a2d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f89040098a0 tx=0x7f8904009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:30.780 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.103+0000 7f891a06f700 1 -- 172.21.15.2:0/719078163 shutdown_connections 2023-11-16T16:40:30.780 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.103+0000 7f891a06f700 1 --2- 172.21.15.2:0/719078163 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f891407be70 0x7f891407a2d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:30.780 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.103+0000 7f891a06f700 1 -- 172.21.15.2:0/719078163 >> 172.21.15.2:0/719078163 conn(0x7f8914100fd0 msgr2=0x7f8914103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:30.780 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.103+0000 7f891a06f700 1 -- 172.21.15.2:0/719078163 shutdown_connections 2023-11-16T16:40:30.780 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.103+0000 7f891a06f700 1 -- 172.21.15.2:0/719078163 wait complete. 2023-11-16T16:40:30.780 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.104+0000 7f891a06f700 1 Processor -- start 2023-11-16T16:40:30.780 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.104+0000 7f891a06f700 1 -- start start 2023-11-16T16:40:30.781 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.104+0000 7f891a06f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f891407be70 0x7f891419b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:30.781 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.104+0000 7f891a06f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891419bef0 con 0x7f891407be70 2023-11-16T16:40:30.781 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.105+0000 7f89137fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f891407be70 0x7f891419b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:30.781 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.105+0000 7f89137fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f891407be70 0x7f891419b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55286/0 (socket says 172.21.15.2:55286) 2023-11-16T16:40:30.781 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.105+0000 7f89137fe700 1 -- 172.21.15.2:0/3228517657 learned_addr learned my addr 172.21.15.2:0/3228517657 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:30.781 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.105+0000 7f89137fe700 1 -- 172.21.15.2:0/3228517657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8904009550 con 0x7f891407be70 2023-11-16T16:40:30.781 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.105+0000 7f89137fe700 1 --2- 172.21.15.2:0/3228517657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f891407be70 0x7f891419b9b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8904009fb0 tx=0x7f8904011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:30.781 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.105+0000 7f8910ff9700 1 -- 172.21.15.2:0/3228517657 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8904003930 con 0x7f891407be70 2023-11-16T16:40:30.781 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.105+0000 7f8910ff9700 1 -- 172.21.15.2:0/3228517657 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8904003a90 con 0x7f891407be70 2023-11-16T16:40:30.781 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.105+0000 7f891a06f700 1 -- 172.21.15.2:0/3228517657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f891410bf60 con 0x7f891407be70 2023-11-16T16:40:30.781 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.105+0000 7f8910ff9700 1 -- 172.21.15.2:0/3228517657 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8904019610 con 0x7f891407be70 2023-11-16T16:40:30.781 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.105+0000 7f891a06f700 1 -- 172.21.15.2:0/3228517657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f891410bdd0 con 0x7f891407be70 2023-11-16T16:40:30.781 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.106+0000 7f8910ff9700 1 -- 172.21.15.2:0/3228517657 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f890401a9d0 con 0x7f891407be70 2023-11-16T16:40:30.781 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.106+0000 7f891a06f700 1 -- 172.21.15.2:0/3228517657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f891410c0f0 con 0x7f891407be70 2023-11-16T16:40:30.782 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.107+0000 7f8910ff9700 1 --2- 172.21.15.2:0/3228517657 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f88fc038710 0x7f88fc03abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:30.782 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.107+0000 7f8910ff9700 1 -- 172.21.15.2:0/3228517657 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f890404c860 con 0x7f891407be70 2023-11-16T16:40:30.782 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.107+0000 7f8912ffd700 1 --2- 172.21.15.2:0/3228517657 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f88fc038710 0x7f88fc03abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:30.782 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.108+0000 7f8912ffd700 1 --2- 172.21.15.2:0/3228517657 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f88fc038710 0x7f88fc03abc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8908006fd0 tx=0x7f8908006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:30.782 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.112+0000 7f8910ff9700 1 -- 172.21.15.2:0/3228517657 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8904018760 con 0x7f891407be70 2023-11-16T16:40:30.782 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.299+0000 7f8910ff9700 1 -- 172.21.15.2:0/3228517657 <== mon.0 v2:172.21.15.2:3300/0 7 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8904018760 con 0x7f891407be70 2023-11-16T16:40:30.782 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.343+0000 7f891a06f700 1 -- 172.21.15.2:0/3228517657 --> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f89141032c0 con 0x7f88fc038710 2023-11-16T16:40:30.782 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.348+0000 7f8910ff9700 1 -- 172.21.15.2:0/3228517657 <== mgr.14118 v2:172.21.15.2:6800/3359672074 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f89141032c0 con 0x7f88fc038710 2023-11-16T16:40:30.782 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.350+0000 7f891a06f700 1 -- 172.21.15.2:0/3228517657 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f88fc038710 msgr2=0x7f88fc03abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:30.782 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.350+0000 7f891a06f700 1 --2- 172.21.15.2:0/3228517657 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f88fc038710 0x7f88fc03abc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8908006fd0 tx=0x7f8908006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:30.782 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.350+0000 7f891a06f700 1 -- 172.21.15.2:0/3228517657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f891407be70 msgr2=0x7f891419b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:30.782 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.350+0000 7f891a06f700 1 --2- 172.21.15.2:0/3228517657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f891407be70 0x7f891419b9b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8904009fb0 tx=0x7f8904011ad0 comp rx=0 tx=0).stop 2023-11-16T16:40:30.783 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.351+0000 7f891a06f700 1 -- 172.21.15.2:0/3228517657 shutdown_connections 2023-11-16T16:40:30.783 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.351+0000 7f891a06f700 1 --2- 172.21.15.2:0/3228517657 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f88fc038710 0x7f88fc03abc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:30.783 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.351+0000 7f891a06f700 1 --2- 172.21.15.2:0/3228517657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f891407be70 0x7f891419b9b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:30.783 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.351+0000 7f891a06f700 1 -- 172.21.15.2:0/3228517657 >> 172.21.15.2:0/3228517657 conn(0x7f8914100fd0 msgr2=0x7f89141026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:30.783 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.351+0000 7f891a06f700 1 -- 172.21.15.2:0/3228517657 shutdown_connections 2023-11-16T16:40:30.783 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:30.351+0000 7f891a06f700 1 -- 172.21.15.2:0/3228517657 wait complete. 2023-11-16T16:40:31.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:31 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' 2023-11-16T16:40:31.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:31 smithi002 ceph-mon[128712]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:40:31.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:31 smithi002 ceph-mon[128712]: Saving service mgr spec with placement count:2 2023-11-16T16:40:31.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:31 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' 2023-11-16T16:40:31.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:31 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' 2023-11-16T16:40:32.064 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.308+0000 7fe164493700 1 Processor -- start 2023-11-16T16:40:32.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.309+0000 7fe164493700 1 -- start start 2023-11-16T16:40:32.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.309+0000 7fe164493700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe15c105a20 0x7fe15c105e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:32.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.309+0000 7fe164493700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c106400 con 0x7fe15c105a20 2023-11-16T16:40:32.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.309+0000 7fe16222f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe15c105a20 0x7fe15c105e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:32.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.310+0000 7fe16222f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe15c105a20 0x7fe15c105e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55298/0 (socket says 172.21.15.2:55298) 2023-11-16T16:40:32.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.310+0000 7fe16222f700 1 -- 172.21.15.2:0/3298994679 learned_addr learned my addr 172.21.15.2:0/3298994679 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:32.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.310+0000 7fe16222f700 1 -- 172.21.15.2:0/3298994679 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15c106c20 con 0x7fe15c105a20 2023-11-16T16:40:32.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.310+0000 7fe16222f700 1 --2- 172.21.15.2:0/3298994679 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe15c105a20 0x7fe15c105e30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe1580098a0 tx=0x7fe158009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c51b3e1a6aaa9c2f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:32.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.310+0000 7fe16122d700 1 -- 172.21.15.2:0/3298994679 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe158004160 con 0x7fe15c105a20 2023-11-16T16:40:32.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.311+0000 7fe16122d700 1 -- 172.21.15.2:0/3298994679 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe1580042c0 con 0x7fe15c105a20 2023-11-16T16:40:32.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.311+0000 7fe16122d700 1 -- 172.21.15.2:0/3298994679 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe1580119b0 con 0x7fe15c105a20 2023-11-16T16:40:32.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.311+0000 7fe164493700 1 -- 172.21.15.2:0/3298994679 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe15c105a20 msgr2=0x7fe15c105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:32.065 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.311+0000 7fe164493700 1 --2- 172.21.15.2:0/3298994679 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe15c105a20 0x7fe15c105e30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe1580098a0 tx=0x7fe158009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:32.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.311+0000 7fe164493700 1 -- 172.21.15.2:0/3298994679 shutdown_connections 2023-11-16T16:40:32.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.311+0000 7fe164493700 1 --2- 172.21.15.2:0/3298994679 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe15c105a20 0x7fe15c105e30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:32.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.311+0000 7fe164493700 1 -- 172.21.15.2:0/3298994679 >> 172.21.15.2:0/3298994679 conn(0x7fe15c100fd0 msgr2=0x7fe15c103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:32.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.312+0000 7fe164493700 1 -- 172.21.15.2:0/3298994679 shutdown_connections 2023-11-16T16:40:32.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.312+0000 7fe164493700 1 -- 172.21.15.2:0/3298994679 wait complete. 2023-11-16T16:40:32.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.313+0000 7fe164493700 1 Processor -- start 2023-11-16T16:40:32.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.313+0000 7fe164493700 1 -- start start 2023-11-16T16:40:32.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.313+0000 7fe164493700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe15c105a20 0x7fe15c19b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:32.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.313+0000 7fe164493700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c19bee0 con 0x7fe15c105a20 2023-11-16T16:40:32.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.313+0000 7fe16222f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe15c105a20 0x7fe15c19b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:32.066 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.313+0000 7fe16222f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe15c105a20 0x7fe15c19b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55300/0 (socket says 172.21.15.2:55300) 2023-11-16T16:40:32.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.313+0000 7fe16222f700 1 -- 172.21.15.2:0/120813058 learned_addr learned my addr 172.21.15.2:0/120813058 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:32.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.314+0000 7fe16222f700 1 -- 172.21.15.2:0/120813058 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe158009550 con 0x7fe15c105a20 2023-11-16T16:40:32.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.314+0000 7fe16222f700 1 --2- 172.21.15.2:0/120813058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe15c105a20 0x7fe15c19b9a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe158000c00 tx=0x7fe158011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:32.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.314+0000 7fe1537fe700 1 -- 172.21.15.2:0/120813058 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe1580039f0 con 0x7fe15c105a20 2023-11-16T16:40:32.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.314+0000 7fe1537fe700 1 -- 172.21.15.2:0/120813058 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe158003b50 con 0x7fe15c105a20 2023-11-16T16:40:32.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.314+0000 7fe164493700 1 -- 172.21.15.2:0/120813058 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15c109fe0 con 0x7fe15c105a20 2023-11-16T16:40:32.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.314+0000 7fe1537fe700 1 -- 172.21.15.2:0/120813058 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe15801a780 con 0x7fe15c105a20 2023-11-16T16:40:32.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.315+0000 7fe164493700 1 -- 172.21.15.2:0/120813058 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe15c109e50 con 0x7fe15c105a20 2023-11-16T16:40:32.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.316+0000 7fe1537fe700 1 -- 172.21.15.2:0/120813058 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fe158024440 con 0x7fe15c105a20 2023-11-16T16:40:32.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.316+0000 7fe164493700 1 -- 172.21.15.2:0/120813058 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15c062190 con 0x7fe15c105a20 2023-11-16T16:40:32.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.316+0000 7fe1537fe700 1 --2- 172.21.15.2:0/120813058 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe148038760 0x7fe14803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:32.067 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.316+0000 7fe1537fe700 1 -- 172.21.15.2:0/120813058 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe15804c870 con 0x7fe15c105a20 2023-11-16T16:40:32.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.317+0000 7fe161a2e700 1 --2- 172.21.15.2:0/120813058 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe148038760 0x7fe14803ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:32.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.317+0000 7fe161a2e700 1 --2- 172.21.15.2:0/120813058 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe148038760 0x7fe14803ac10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe14c006fd0 tx=0x7fe14c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:32.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.321+0000 7fe1537fe700 1 -- 172.21.15.2:0/120813058 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe15c062190 con 0x7fe15c105a20 2023-11-16T16:40:32.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.530+0000 7fe164493700 1 -- 172.21.15.2:0/120813058 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fe15c062190 con 0x7fe15c105a20 2023-11-16T16:40:32.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.534+0000 7fe1537fe700 1 -- 172.21.15.2:0/120813058 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fe15c062190 con 0x7fe15c105a20 2023-11-16T16:40:32.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.537+0000 7fe164493700 1 -- 172.21.15.2:0/120813058 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe148038760 msgr2=0x7fe14803ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:32.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.537+0000 7fe164493700 1 --2- 172.21.15.2:0/120813058 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe148038760 0x7fe14803ac10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe14c006fd0 tx=0x7fe14c006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:32.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.537+0000 7fe164493700 1 -- 172.21.15.2:0/120813058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe15c105a20 msgr2=0x7fe15c19b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:32.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.537+0000 7fe164493700 1 --2- 172.21.15.2:0/120813058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe15c105a20 0x7fe15c19b9a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe158000c00 tx=0x7fe158011a30 comp rx=0 tx=0).stop 2023-11-16T16:40:32.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.537+0000 7fe164493700 1 -- 172.21.15.2:0/120813058 shutdown_connections 2023-11-16T16:40:32.068 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.537+0000 7fe164493700 1 --2- 172.21.15.2:0/120813058 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe148038760 0x7fe14803ac10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:32.069 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.537+0000 7fe164493700 1 --2- 172.21.15.2:0/120813058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe15c105a20 0x7fe15c19b9a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:32.069 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.537+0000 7fe164493700 1 -- 172.21.15.2:0/120813058 >> 172.21.15.2:0/120813058 conn(0x7fe15c100fd0 msgr2=0x7fe15c102630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:32.069 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.537+0000 7fe164493700 1 -- 172.21.15.2:0/120813058 shutdown_connections 2023-11-16T16:40:32.069 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:31.537+0000 7fe164493700 1 -- 172.21.15.2:0/120813058 wait complete. 2023-11-16T16:40:32.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:32 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/120813058' entity='client.admin' 2023-11-16T16:40:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.659+0000 7ff54fa0b700 1 Processor -- start 2023-11-16T16:40:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.660+0000 7ff54fa0b700 1 -- start start 2023-11-16T16:40:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.660+0000 7ff54fa0b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff54807be10 0x7ff54807a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.660+0000 7ff54fa0b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54807a840 con 0x7ff54807be10 2023-11-16T16:40:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.660+0000 7ff54d7a7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff54807be10 0x7ff54807a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.660+0000 7ff54d7a7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff54807be10 0x7ff54807a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55306/0 (socket says 172.21.15.2:55306) 2023-11-16T16:40:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.660+0000 7ff54d7a7700 1 -- 172.21.15.2:0/499996628 learned_addr learned my addr 172.21.15.2:0/499996628 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.661+0000 7ff54d7a7700 1 -- 172.21.15.2:0/499996628 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54807a980 con 0x7ff54807be10 2023-11-16T16:40:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.661+0000 7ff54d7a7700 1 --2- 172.21.15.2:0/499996628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff54807be10 0x7ff54807a270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff5440098a0 tx=0x7ff544009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0a630f3bb1a16fc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.661+0000 7ff53ffff700 1 -- 172.21.15.2:0/499996628 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff544004160 con 0x7ff54807be10 2023-11-16T16:40:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.661+0000 7ff53ffff700 1 -- 172.21.15.2:0/499996628 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff5440042c0 con 0x7ff54807be10 2023-11-16T16:40:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.661+0000 7ff53ffff700 1 -- 172.21.15.2:0/499996628 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5440119b0 con 0x7ff54807be10 2023-11-16T16:40:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.662+0000 7ff54fa0b700 1 -- 172.21.15.2:0/499996628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff54807be10 msgr2=0x7ff54807a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.662+0000 7ff54fa0b700 1 --2- 172.21.15.2:0/499996628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff54807be10 0x7ff54807a270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff5440098a0 tx=0x7ff544009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.662+0000 7ff54fa0b700 1 -- 172.21.15.2:0/499996628 shutdown_connections 2023-11-16T16:40:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.662+0000 7ff54fa0b700 1 --2- 172.21.15.2:0/499996628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff54807be10 0x7ff54807a270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.662+0000 7ff54fa0b700 1 -- 172.21.15.2:0/499996628 >> 172.21.15.2:0/499996628 conn(0x7ff548100fd0 msgr2=0x7ff548103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.662+0000 7ff54fa0b700 1 -- 172.21.15.2:0/499996628 shutdown_connections 2023-11-16T16:40:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.662+0000 7ff54fa0b700 1 -- 172.21.15.2:0/499996628 wait complete. 2023-11-16T16:40:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.663+0000 7ff54fa0b700 1 Processor -- start 2023-11-16T16:40:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.664+0000 7ff54fa0b700 1 -- start start 2023-11-16T16:40:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.664+0000 7ff54fa0b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff54807be10 0x7ff54819b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.664+0000 7ff54fa0b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54819beb0 con 0x7ff54807be10 2023-11-16T16:40:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.664+0000 7ff54d7a7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff54807be10 0x7ff54819b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.664+0000 7ff54d7a7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff54807be10 0x7ff54819b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55322/0 (socket says 172.21.15.2:55322) 2023-11-16T16:40:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.664+0000 7ff54d7a7700 1 -- 172.21.15.2:0/898385970 learned_addr learned my addr 172.21.15.2:0/898385970 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.665+0000 7ff54d7a7700 1 -- 172.21.15.2:0/898385970 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff544009550 con 0x7ff54807be10 2023-11-16T16:40:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.665+0000 7ff54d7a7700 1 --2- 172.21.15.2:0/898385970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff54807be10 0x7ff54819b970 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff544003e60 tx=0x7ff544011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.665+0000 7ff53e7fc700 1 -- 172.21.15.2:0/898385970 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff544003970 con 0x7ff54807be10 2023-11-16T16:40:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.665+0000 7ff53e7fc700 1 -- 172.21.15.2:0/898385970 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff544003ad0 con 0x7ff54807be10 2023-11-16T16:40:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.665+0000 7ff54fa0b700 1 -- 172.21.15.2:0/898385970 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff54810bfb0 con 0x7ff54807be10 2023-11-16T16:40:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.665+0000 7ff53e7fc700 1 -- 172.21.15.2:0/898385970 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5440195e0 con 0x7ff54807be10 2023-11-16T16:40:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.665+0000 7ff54fa0b700 1 -- 172.21.15.2:0/898385970 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff54810be20 con 0x7ff54807be10 2023-11-16T16:40:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.666+0000 7ff53e7fc700 1 -- 172.21.15.2:0/898385970 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7ff54401a430 con 0x7ff54807be10 2023-11-16T16:40:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.666+0000 7ff54fa0b700 1 -- 172.21.15.2:0/898385970 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff54810c140 con 0x7ff54807be10 2023-11-16T16:40:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.667+0000 7ff53e7fc700 1 --2- 172.21.15.2:0/898385970 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7ff5340386c0 0x7ff53403ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.667+0000 7ff53e7fc700 1 -- 172.21.15.2:0/898385970 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff54404c8f0 con 0x7ff54807be10 2023-11-16T16:40:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.667+0000 7ff54cfa6700 1 --2- 172.21.15.2:0/898385970 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7ff5340386c0 0x7ff53403ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.668+0000 7ff54cfa6700 1 --2- 172.21.15.2:0/898385970 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7ff5340386c0 0x7ff53403ab70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff538006fd0 tx=0x7ff538006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.672+0000 7ff53e7fc700 1 -- 172.21.15.2:0/898385970 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff544003c40 con 0x7ff54807be10 2023-11-16T16:40:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.880+0000 7ff54fa0b700 1 -- 172.21.15.2:0/898385970 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7ff54819f5f0 con 0x7ff54807be10 2023-11-16T16:40:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.884+0000 7ff53e7fc700 1 -- 172.21.15.2:0/898385970 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7ff544023b80 con 0x7ff54807be10 2023-11-16T16:40:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.887+0000 7ff54fa0b700 1 -- 172.21.15.2:0/898385970 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7ff5340386c0 msgr2=0x7ff53403ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.887+0000 7ff54fa0b700 1 --2- 172.21.15.2:0/898385970 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7ff5340386c0 0x7ff53403ab70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff538006fd0 tx=0x7ff538006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.887+0000 7ff54fa0b700 1 -- 172.21.15.2:0/898385970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff54807be10 msgr2=0x7ff54819b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.887+0000 7ff54fa0b700 1 --2- 172.21.15.2:0/898385970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff54807be10 0x7ff54819b970 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff544003e60 tx=0x7ff544011aa0 comp rx=0 tx=0).stop 2023-11-16T16:40:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.888+0000 7ff54fa0b700 1 -- 172.21.15.2:0/898385970 shutdown_connections 2023-11-16T16:40:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.888+0000 7ff54fa0b700 1 --2- 172.21.15.2:0/898385970 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7ff5340386c0 0x7ff53403ab70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.888+0000 7ff54fa0b700 1 --2- 172.21.15.2:0/898385970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff54807be10 0x7ff54819b970 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:33.451 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.888+0000 7ff54fa0b700 1 -- 172.21.15.2:0/898385970 >> 172.21.15.2:0/898385970 conn(0x7ff548100fd0 msgr2=0x7ff5481026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:33.451 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.888+0000 7ff54fa0b700 1 -- 172.21.15.2:0/898385970 shutdown_connections 2023-11-16T16:40:33.451 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:32.888+0000 7ff54fa0b700 1 -- 172.21.15.2:0/898385970 wait complete. 2023-11-16T16:40:33.451 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling the dashboard module... 2023-11-16T16:40:34.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:33 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/898385970' entity='client.admin' 2023-11-16T16:40:35.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:34 smithi002 ceph-mon[128712]: from='mgr.14118 172.21.15.2:0/188696927' entity='mgr.x' 2023-11-16T16:40:35.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:34 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2558635635' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-11-16T16:40:35.156 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:35 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: ignoring --setuser ceph since I am not root 2023-11-16T16:40:35.156 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:35 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: ignoring --setgroup ceph since I am not root 2023-11-16T16:40:35.328 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.028+0000 7f42a0c8c700 1 Processor -- start 2023-11-16T16:40:35.333 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.029+0000 7f42a0c8c700 1 -- start start 2023-11-16T16:40:35.333 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.029+0000 7f42a0c8c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f429c1058d0 0x7f429c105ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:35.334 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.029+0000 7f42a0c8c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429c106220 con 0x7f429c1058d0 2023-11-16T16:40:35.334 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.029+0000 7f429b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f429c1058d0 0x7f429c105ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:35.334 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.029+0000 7f429b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f429c1058d0 0x7f429c105ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55336/0 (socket says 172.21.15.2:55336) 2023-11-16T16:40:35.334 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.029+0000 7f429b7fe700 1 -- 172.21.15.2:0/3300656252 learned_addr learned my addr 172.21.15.2:0/3300656252 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:35.334 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.030+0000 7f429b7fe700 1 -- 172.21.15.2:0/3300656252 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429c106360 con 0x7f429c1058d0 2023-11-16T16:40:35.334 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.030+0000 7f429b7fe700 1 --2- 172.21.15.2:0/3300656252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f429c1058d0 0x7f429c105ce0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f428c0098a0 tx=0x7f428c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87e62ce59184c1c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:35.334 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.030+0000 7f429a7fc700 1 -- 172.21.15.2:0/3300656252 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f428c004160 con 0x7f429c1058d0 2023-11-16T16:40:35.334 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.030+0000 7f429a7fc700 1 -- 172.21.15.2:0/3300656252 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f428c0042c0 con 0x7f429c1058d0 2023-11-16T16:40:35.334 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.030+0000 7f429a7fc700 1 -- 172.21.15.2:0/3300656252 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f428c0119b0 con 0x7f429c1058d0 2023-11-16T16:40:35.334 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.031+0000 7f42a0c8c700 1 -- 172.21.15.2:0/3300656252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f429c1058d0 msgr2=0x7f429c105ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:35.334 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.031+0000 7f42a0c8c700 1 --2- 172.21.15.2:0/3300656252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f429c1058d0 0x7f429c105ce0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f428c0098a0 tx=0x7f428c009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:35.334 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.031+0000 7f42a0c8c700 1 -- 172.21.15.2:0/3300656252 shutdown_connections 2023-11-16T16:40:35.335 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.031+0000 7f42a0c8c700 1 --2- 172.21.15.2:0/3300656252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f429c1058d0 0x7f429c105ce0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:35.335 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.031+0000 7f42a0c8c700 1 -- 172.21.15.2:0/3300656252 >> 172.21.15.2:0/3300656252 conn(0x7f429c100f40 msgr2=0x7f429c103350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:35.335 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.031+0000 7f42a0c8c700 1 -- 172.21.15.2:0/3300656252 shutdown_connections 2023-11-16T16:40:35.335 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.031+0000 7f42a0c8c700 1 -- 172.21.15.2:0/3300656252 wait complete. 2023-11-16T16:40:35.335 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.032+0000 7f42a0c8c700 1 Processor -- start 2023-11-16T16:40:35.335 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.032+0000 7f42a0c8c700 1 -- start start 2023-11-16T16:40:35.335 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.033+0000 7f42a0c8c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f429c1058d0 0x7f429c19bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:35.335 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.033+0000 7f42a0c8c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429c19c020 con 0x7f429c1058d0 2023-11-16T16:40:35.335 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.033+0000 7f429b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f429c1058d0 0x7f429c19bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:35.335 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.033+0000 7f429b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f429c1058d0 0x7f429c19bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55338/0 (socket says 172.21.15.2:55338) 2023-11-16T16:40:35.335 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.033+0000 7f429b7fe700 1 -- 172.21.15.2:0/2558635635 learned_addr learned my addr 172.21.15.2:0/2558635635 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:35.335 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.033+0000 7f429b7fe700 1 -- 172.21.15.2:0/2558635635 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428c009550 con 0x7f429c1058d0 2023-11-16T16:40:35.336 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.034+0000 7f429b7fe700 1 --2- 172.21.15.2:0/2558635635 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f429c1058d0 0x7f429c19bae0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f428c003f00 tx=0x7f428c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:35.336 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.034+0000 7f4298ff9700 1 -- 172.21.15.2:0/2558635635 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f428c003b20 con 0x7f429c1058d0 2023-11-16T16:40:35.336 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.034+0000 7f4298ff9700 1 -- 172.21.15.2:0/2558635635 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f428c003c80 con 0x7f429c1058d0 2023-11-16T16:40:35.336 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.034+0000 7f4298ff9700 1 -- 172.21.15.2:0/2558635635 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f428c019770 con 0x7f429c1058d0 2023-11-16T16:40:35.336 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.034+0000 7f42a0c8c700 1 -- 172.21.15.2:0/2558635635 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f429c107960 con 0x7f429c1058d0 2023-11-16T16:40:35.336 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.034+0000 7f42a0c8c700 1 -- 172.21.15.2:0/2558635635 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f429c1077d0 con 0x7f429c1058d0 2023-11-16T16:40:35.336 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.036+0000 7f4298ff9700 1 -- 172.21.15.2:0/2558635635 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f428c01ab60 con 0x7f429c1058d0 2023-11-16T16:40:35.336 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.036+0000 7f42a0c8c700 1 -- 172.21.15.2:0/2558635635 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f429c109da0 con 0x7f429c1058d0 2023-11-16T16:40:35.336 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.036+0000 7f4298ff9700 1 --2- 172.21.15.2:0/2558635635 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f4284038760 0x7f428403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:35.336 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.036+0000 7f4298ff9700 1 -- 172.21.15.2:0/2558635635 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f428c04c970 con 0x7f429c1058d0 2023-11-16T16:40:35.336 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.037+0000 7f429affd700 1 --2- 172.21.15.2:0/2558635635 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f4284038760 0x7f428403ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:35.336 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.037+0000 7f429affd700 1 --2- 172.21.15.2:0/2558635635 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f4284038760 0x7f428403ac10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4290006fd0 tx=0x7f4290006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:35.336 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.042+0000 7f4298ff9700 1 -- 172.21.15.2:0/2558635635 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f428c01a430 con 0x7f429c1058d0 2023-11-16T16:40:35.336 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.309+0000 7f42a0c8c700 1 -- 172.21.15.2:0/2558635635 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f429c195a90 con 0x7f429c1058d0 2023-11-16T16:40:35.337 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.900+0000 7f4298ff9700 1 -- 172.21.15.2:0/2558635635 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f428c02d350 con 0x7f429c1058d0 2023-11-16T16:40:35.337 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.902+0000 7f4298ff9700 1 -- 172.21.15.2:0/2558635635 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f428c018580 con 0x7f429c1058d0 2023-11-16T16:40:35.337 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.904+0000 7f42a0c8c700 1 -- 172.21.15.2:0/2558635635 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f4284038760 msgr2=0x7f428403ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:35.337 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.904+0000 7f42a0c8c700 1 --2- 172.21.15.2:0/2558635635 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f4284038760 0x7f428403ac10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4290006fd0 tx=0x7f4290006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:35.337 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.904+0000 7f42a0c8c700 1 -- 172.21.15.2:0/2558635635 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f429c1058d0 msgr2=0x7f429c19bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:35.337 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.904+0000 7f42a0c8c700 1 --2- 172.21.15.2:0/2558635635 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f429c1058d0 0x7f429c19bae0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f428c003f00 tx=0x7f428c011a30 comp rx=0 tx=0).stop 2023-11-16T16:40:35.337 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.904+0000 7f42a0c8c700 1 -- 172.21.15.2:0/2558635635 shutdown_connections 2023-11-16T16:40:35.337 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.904+0000 7f42a0c8c700 1 --2- 172.21.15.2:0/2558635635 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f4284038760 0x7f428403ac10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:35.337 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.904+0000 7f42a0c8c700 1 --2- 172.21.15.2:0/2558635635 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f429c1058d0 0x7f429c19bae0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:35.337 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.904+0000 7f42a0c8c700 1 -- 172.21.15.2:0/2558635635 >> 172.21.15.2:0/2558635635 conn(0x7f429c100f40 msgr2=0x7f429c103230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:35.337 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.905+0000 7f42a0c8c700 1 -- 172.21.15.2:0/2558635635 shutdown_connections 2023-11-16T16:40:35.337 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:34.905+0000 7f42a0c8c700 1 -- 172.21.15.2:0/2558635635 wait complete. 2023-11-16T16:40:35.564 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:35 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:35.203+0000 7f90e4c33200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T16:40:35.564 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:35 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:35.309+0000 7f90e4c33200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T16:40:35.564 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:35 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:35.433+0000 7f90e4c33200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T16:40:35.901 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:35 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:35.561+0000 7f90e4c33200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T16:40:35.901 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:35 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:35.807+0000 7f90e4c33200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T16:40:36.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:35 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2558635635' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-11-16T16:40:36.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:35 smithi002 ceph-mon[128712]: mgrmap e9: x(active, since 18s) 2023-11-16T16:40:36.519 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:36 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:36.218+0000 7f90e4c33200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T16:40:36.519 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:36 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:36.346+0000 7f90e4c33200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T16:40:36.520 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:36 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:36.517+0000 7f90e4c33200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T16:40:36.622 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2023-11-16T16:40:36.622 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-11-16T16:40:36.623 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": true, 2023-11-16T16:40:36.623 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-11-16T16:40:36.623 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-11-16T16:40:36.623 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2023-11-16T16:40:36.623 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.901+0000 7fe56ec5a700 1 Processor -- start 2023-11-16T16:40:36.623 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.901+0000 7fe56ec5a700 1 -- start start 2023-11-16T16:40:36.623 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.901+0000 7fe56ec5a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe568105a20 0x7fe568105e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:36.623 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.901+0000 7fe56ec5a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe568106400 con 0x7fe568105a20 2023-11-16T16:40:36.623 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.902+0000 7fe56c9f6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe568105a20 0x7fe568105e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:36.623 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.902+0000 7fe56c9f6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe568105a20 0x7fe568105e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43504/0 (socket says 172.21.15.2:43504) 2023-11-16T16:40:36.624 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.902+0000 7fe56c9f6700 1 -- 172.21.15.2:0/3593244788 learned_addr learned my addr 172.21.15.2:0/3593244788 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:36.624 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.903+0000 7fe56c9f6700 1 -- 172.21.15.2:0/3593244788 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe568106c20 con 0x7fe568105a20 2023-11-16T16:40:36.624 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.903+0000 7fe56c9f6700 1 --2- 172.21.15.2:0/3593244788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe568105a20 0x7fe568105e30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe55c0098a0 tx=0x7fe55c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f644b0b6565fc45 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:36.624 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.903+0000 7fe5677fe700 1 -- 172.21.15.2:0/3593244788 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe55c004160 con 0x7fe568105a20 2023-11-16T16:40:36.624 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.903+0000 7fe5677fe700 1 -- 172.21.15.2:0/3593244788 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe55c0042c0 con 0x7fe568105a20 2023-11-16T16:40:36.624 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.903+0000 7fe5677fe700 1 -- 172.21.15.2:0/3593244788 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe55c0119b0 con 0x7fe568105a20 2023-11-16T16:40:36.624 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.904+0000 7fe56ec5a700 1 -- 172.21.15.2:0/3593244788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe568105a20 msgr2=0x7fe568105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:36.624 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.904+0000 7fe56ec5a700 1 --2- 172.21.15.2:0/3593244788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe568105a20 0x7fe568105e30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe55c0098a0 tx=0x7fe55c009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:36.625 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.904+0000 7fe56ec5a700 1 -- 172.21.15.2:0/3593244788 shutdown_connections 2023-11-16T16:40:36.625 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.904+0000 7fe56ec5a700 1 --2- 172.21.15.2:0/3593244788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe568105a20 0x7fe568105e30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:36.625 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.904+0000 7fe56ec5a700 1 -- 172.21.15.2:0/3593244788 >> 172.21.15.2:0/3593244788 conn(0x7fe568100fd0 msgr2=0x7fe568103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:36.625 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.904+0000 7fe56ec5a700 1 -- 172.21.15.2:0/3593244788 shutdown_connections 2023-11-16T16:40:36.625 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.904+0000 7fe56ec5a700 1 -- 172.21.15.2:0/3593244788 wait complete. 2023-11-16T16:40:36.625 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.906+0000 7fe56ec5a700 1 Processor -- start 2023-11-16T16:40:36.625 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.906+0000 7fe56ec5a700 1 -- start start 2023-11-16T16:40:36.625 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.907+0000 7fe56ec5a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe568105a20 0x7fe56807bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:36.625 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.907+0000 7fe56ec5a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56807c140 con 0x7fe568105a20 2023-11-16T16:40:36.625 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.907+0000 7fe56c9f6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe568105a20 0x7fe56807bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:36.626 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.907+0000 7fe56c9f6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe568105a20 0x7fe56807bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43518/0 (socket says 172.21.15.2:43518) 2023-11-16T16:40:36.626 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.907+0000 7fe56c9f6700 1 -- 172.21.15.2:0/1589014585 learned_addr learned my addr 172.21.15.2:0/1589014585 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:36.626 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.907+0000 7fe56c9f6700 1 -- 172.21.15.2:0/1589014585 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55c009550 con 0x7fe568105a20 2023-11-16T16:40:36.626 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.907+0000 7fe56c9f6700 1 --2- 172.21.15.2:0/1589014585 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe568105a20 0x7fe56807bc00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe55c000c00 tx=0x7fe55c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:36.626 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.908+0000 7fe565ffb700 1 -- 172.21.15.2:0/1589014585 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe55c0039f0 con 0x7fe568105a20 2023-11-16T16:40:36.626 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.908+0000 7fe565ffb700 1 -- 172.21.15.2:0/1589014585 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe55c003b50 con 0x7fe568105a20 2023-11-16T16:40:36.626 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.908+0000 7fe565ffb700 1 -- 172.21.15.2:0/1589014585 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe55c01a780 con 0x7fe568105a20 2023-11-16T16:40:36.626 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.908+0000 7fe56ec5a700 1 -- 172.21.15.2:0/1589014585 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe568109fe0 con 0x7fe568105a20 2023-11-16T16:40:36.626 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.908+0000 7fe56ec5a700 1 -- 172.21.15.2:0/1589014585 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe568109e50 con 0x7fe568105a20 2023-11-16T16:40:36.626 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.909+0000 7fe56ec5a700 1 -- 172.21.15.2:0/1589014585 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe568062190 con 0x7fe568105a20 2023-11-16T16:40:36.627 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.910+0000 7fe565ffb700 1 -- 172.21.15.2:0/1589014585 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fe55c024440 con 0x7fe568105a20 2023-11-16T16:40:36.627 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.910+0000 7fe565ffb700 1 --2- 172.21.15.2:0/1589014585 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe5580387b0 0x7fe55803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:36.627 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.910+0000 7fe565ffb700 1 -- 172.21.15.2:0/1589014585 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe55c04c820 con 0x7fe568105a20 2023-11-16T16:40:36.627 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.910+0000 7fe567fff700 1 -- 172.21.15.2:0/1589014585 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe5580387b0 msgr2=0x7fe55803ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3359672074 2023-11-16T16:40:36.627 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.910+0000 7fe567fff700 1 --2- 172.21.15.2:0/1589014585 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe5580387b0 0x7fe55803ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:40:36.627 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:35.915+0000 7fe565ffb700 1 -- 172.21.15.2:0/1589014585 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe568062190 con 0x7fe568105a20 2023-11-16T16:40:36.627 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:36.111+0000 7fe567fff700 1 -- 172.21.15.2:0/1589014585 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe5580387b0 msgr2=0x7fe55803ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3359672074 2023-11-16T16:40:36.627 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:36.111+0000 7fe567fff700 1 --2- 172.21.15.2:0/1589014585 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe5580387b0 0x7fe55803ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T16:40:36.627 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:36.181+0000 7fe56ec5a700 1 -- 172.21.15.2:0/1589014585 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fe568062120 con 0x7fe568105a20 2023-11-16T16:40:36.628 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:36.181+0000 7fe565ffb700 1 -- 172.21.15.2:0/1589014585 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fe55c0189a0 con 0x7fe568105a20 2023-11-16T16:40:36.628 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:36.183+0000 7fe56ec5a700 1 -- 172.21.15.2:0/1589014585 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe5580387b0 msgr2=0x7fe55803ac60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:40:36.628 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:36.183+0000 7fe56ec5a700 1 --2- 172.21.15.2:0/1589014585 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe5580387b0 0x7fe55803ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:36.628 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:36.183+0000 7fe56ec5a700 1 -- 172.21.15.2:0/1589014585 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe568105a20 msgr2=0x7fe56807bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:36.628 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:36.183+0000 7fe56ec5a700 1 --2- 172.21.15.2:0/1589014585 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe568105a20 0x7fe56807bc00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe55c000c00 tx=0x7fe55c011a30 comp rx=0 tx=0).stop 2023-11-16T16:40:36.628 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:36.184+0000 7fe56ec5a700 1 -- 172.21.15.2:0/1589014585 shutdown_connections 2023-11-16T16:40:36.628 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:36.184+0000 7fe56ec5a700 1 --2- 172.21.15.2:0/1589014585 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7fe5580387b0 0x7fe55803ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:36.628 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:36.184+0000 7fe56ec5a700 1 --2- 172.21.15.2:0/1589014585 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe568105a20 0x7fe56807bc00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:36.628 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:36.184+0000 7fe56ec5a700 1 -- 172.21.15.2:0/1589014585 >> 172.21.15.2:0/1589014585 conn(0x7fe568100fd0 msgr2=0x7fe568102680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:36.628 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:36.184+0000 7fe56ec5a700 1 -- 172.21.15.2:0/1589014585 shutdown_connections 2023-11-16T16:40:36.629 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:36.184+0000 7fe56ec5a700 1 -- 172.21.15.2:0/1589014585 wait complete. 2023-11-16T16:40:36.629 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for the mgr to restart... 2023-11-16T16:40:36.629 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr epoch 9... 2023-11-16T16:40:36.902 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:36 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:36.650+0000 7f90e4c33200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T16:40:37.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:36 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1589014585' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-16T16:40:37.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:37 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:37.220+0000 7f90e4c33200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T16:40:38.405 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:37 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:37.990+0000 7f90e4c33200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T16:40:38.405 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:38 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:38.204+0000 7f90e4c33200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T16:40:39.544 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:39 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:39.185+0000 7f90e4c33200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:40:39.545 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:39 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:39.309+0000 7f90e4c33200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T16:40:39.545 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:39 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:39.542+0000 7f90e4c33200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:40:40.203 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:39 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:39.918+0000 7f90e4c33200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T16:40:40.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:40 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:40.201+0000 7f90e4c33200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T16:40:40.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:40 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:40.400+0000 7f90e4c33200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T16:40:41.155 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:40 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:40.757+0000 7f90e4c33200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T16:40:42.155 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:41 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:41.724+0000 7f90e4c33200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T16:40:42.155 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:41 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:41.902+0000 7f90e4c33200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T16:40:42.600 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:42 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:42.158+0000 7f90e4c33200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T16:40:42.600 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:42 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:42.311+0000 7f90e4c33200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T16:40:42.905 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:42 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:42.598+0000 7f90e4c33200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T16:40:44.155 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:40:43 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:40:43.701+0000 7f90e4c33200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T16:40:44.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:44 smithi002 ceph-mon[128712]: Active manager daemon x restarted 2023-11-16T16:40:44.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:44 smithi002 ceph-mon[128712]: Activating manager daemon x 2023-11-16T16:40:44.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:44 smithi002 ceph-mon[128712]: osdmap e3: 0 total, 0 up, 0 in 2023-11-16T16:40:44.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:44 smithi002 ceph-mon[128712]: mgrmap e10: x(active, starting, since 0.00360827s) 2023-11-16T16:40:44.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:40:44.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T16:40:44.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T16:40:44.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T16:40:44.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T16:40:44.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:44 smithi002 ceph-mon[128712]: Manager daemon x is now available 2023-11-16T16:40:44.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:44.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:44.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T16:40:44.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T16:40:45.125 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2023-11-16T16:40:45.125 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-11-16T16:40:45.126 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "initialized": true 2023-11-16T16:40:45.126 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2023-11-16T16:40:45.126 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.248+0000 7f1c00483700 1 Processor -- start 2023-11-16T16:40:45.126 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.248+0000 7f1c00483700 1 -- start start 2023-11-16T16:40:45.126 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.249+0000 7f1c00483700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf807be10 0x7f1bf807a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:45.126 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.249+0000 7f1c00483700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf807a840 con 0x7f1bf807be10 2023-11-16T16:40:45.126 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.249+0000 7f1bfe21f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf807be10 0x7f1bf807a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:45.126 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.249+0000 7f1bfe21f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf807be10 0x7f1bf807a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43526/0 (socket says 172.21.15.2:43526) 2023-11-16T16:40:45.126 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.249+0000 7f1bfe21f700 1 -- 172.21.15.2:0/2017917756 learned_addr learned my addr 172.21.15.2:0/2017917756 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:45.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.249+0000 7f1bfe21f700 1 -- 172.21.15.2:0/2017917756 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf807a980 con 0x7f1bf807be10 2023-11-16T16:40:45.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.250+0000 7f1bfe21f700 1 --2- 172.21.15.2:0/2017917756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf807be10 0x7f1bf807a270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40098a0 tx=0x7f1bf4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70f3a1bd5232757 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:45.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.250+0000 7f1bfd21d700 1 -- 172.21.15.2:0/2017917756 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bf4004160 con 0x7f1bf807be10 2023-11-16T16:40:45.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.250+0000 7f1bfd21d700 1 -- 172.21.15.2:0/2017917756 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1bf40042c0 con 0x7f1bf807be10 2023-11-16T16:40:45.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.250+0000 7f1bfd21d700 1 -- 172.21.15.2:0/2017917756 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bf40119b0 con 0x7f1bf807be10 2023-11-16T16:40:45.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.251+0000 7f1c00483700 1 -- 172.21.15.2:0/2017917756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf807be10 msgr2=0x7f1bf807a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:45.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.251+0000 7f1c00483700 1 --2- 172.21.15.2:0/2017917756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf807be10 0x7f1bf807a270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40098a0 tx=0x7f1bf4009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:45.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.251+0000 7f1c00483700 1 -- 172.21.15.2:0/2017917756 shutdown_connections 2023-11-16T16:40:45.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.251+0000 7f1c00483700 1 --2- 172.21.15.2:0/2017917756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf807be10 0x7f1bf807a270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:45.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.251+0000 7f1c00483700 1 -- 172.21.15.2:0/2017917756 >> 172.21.15.2:0/2017917756 conn(0x7f1bf8100fd0 msgr2=0x7f1bf8103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:45.127 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.251+0000 7f1c00483700 1 -- 172.21.15.2:0/2017917756 shutdown_connections 2023-11-16T16:40:45.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.251+0000 7f1c00483700 1 -- 172.21.15.2:0/2017917756 wait complete. 2023-11-16T16:40:45.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.252+0000 7f1c00483700 1 Processor -- start 2023-11-16T16:40:45.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.253+0000 7f1c00483700 1 -- start start 2023-11-16T16:40:45.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.253+0000 7f1c00483700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf807be10 0x7f1bf819b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:45.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.253+0000 7f1c00483700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf819beb0 con 0x7f1bf807be10 2023-11-16T16:40:45.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.253+0000 7f1bfe21f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf807be10 0x7f1bf819b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:45.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.253+0000 7f1bfe21f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf807be10 0x7f1bf819b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43530/0 (socket says 172.21.15.2:43530) 2023-11-16T16:40:45.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.253+0000 7f1bfe21f700 1 -- 172.21.15.2:0/520299244 learned_addr learned my addr 172.21.15.2:0/520299244 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:45.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.254+0000 7f1bfe21f700 1 -- 172.21.15.2:0/520299244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf4009550 con 0x7f1bf807be10 2023-11-16T16:40:45.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.254+0000 7f1bfe21f700 1 --2- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf807be10 0x7f1bf819b970 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4011980 tx=0x7f1bf4011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:45.128 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.254+0000 7f1bef7fe700 1 -- 172.21.15.2:0/520299244 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bf4003a60 con 0x7f1bf807be10 2023-11-16T16:40:45.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.254+0000 7f1bef7fe700 1 -- 172.21.15.2:0/520299244 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1bf4003bc0 con 0x7f1bf807be10 2023-11-16T16:40:45.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.254+0000 7f1c00483700 1 -- 172.21.15.2:0/520299244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf810bfb0 con 0x7f1bf807be10 2023-11-16T16:40:45.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.254+0000 7f1bef7fe700 1 -- 172.21.15.2:0/520299244 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1bf401a670 con 0x7f1bf807be10 2023-11-16T16:40:45.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.254+0000 7f1c00483700 1 -- 172.21.15.2:0/520299244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf810be20 con 0x7f1bf807be10 2023-11-16T16:40:45.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.256+0000 7f1bef7fe700 1 -- 172.21.15.2:0/520299244 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f1bf4024a00 con 0x7f1bf807be10 2023-11-16T16:40:45.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.256+0000 7f1bef7fe700 1 --2- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f1be40387b0 0x7f1be403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:45.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.256+0000 7f1bef7fe700 1 -- 172.21.15.2:0/520299244 --> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1be403b370 con 0x7f1be40387b0 2023-11-16T16:40:45.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.256+0000 7f1bef7fe700 1 -- 172.21.15.2:0/520299244 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1bf404c930 con 0x7f1bf807be10 2023-11-16T16:40:45.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.256+0000 7f1bfda1e700 1 -- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f1be40387b0 msgr2=0x7f1be403ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3359672074 2023-11-16T16:40:45.129 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.256+0000 7f1bfda1e700 1 --2- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f1be40387b0 0x7f1be403ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:40:45.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.457+0000 7f1bfda1e700 1 -- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f1be40387b0 msgr2=0x7f1be403ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3359672074 2023-11-16T16:40:45.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.457+0000 7f1bfda1e700 1 --2- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f1be40387b0 0x7f1be403ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T16:40:45.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.858+0000 7f1bfda1e700 1 -- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f1be40387b0 msgr2=0x7f1be403ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3359672074 2023-11-16T16:40:45.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:37.858+0000 7f1bfda1e700 1 --2- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f1be40387b0 0x7f1be403ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-16T16:40:45.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:38.659+0000 7f1bfda1e700 1 -- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f1be40387b0 msgr2=0x7f1be403ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3359672074 2023-11-16T16:40:45.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:38.659+0000 7f1bfda1e700 1 --2- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f1be40387b0 0x7f1be403ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-16T16:40:45.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:40.260+0000 7f1bfda1e700 1 -- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f1be40387b0 msgr2=0x7f1be403ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3359672074 2023-11-16T16:40:45.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:40.261+0000 7f1bfda1e700 1 --2- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f1be40387b0 0x7f1be403ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-16T16:40:45.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.462+0000 7f1bfda1e700 1 -- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f1be40387b0 msgr2=0x7f1be403ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/3359672074 2023-11-16T16:40:45.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.462+0000 7f1bfda1e700 1 --2- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f1be40387b0 0x7f1be403ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-16T16:40:45.130 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.706+0000 7f1bef7fe700 1 -- 172.21.15.2:0/520299244 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7f1bf4024620 con 0x7f1bf807be10 2023-11-16T16:40:45.131 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.706+0000 7f1bef7fe700 1 -- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f1be40387b0 msgr2=0x7f1be403ac60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:40:45.131 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:43.706+0000 7f1bef7fe700 1 --2- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f1be40387b0 0x7f1be403ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:45.131 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.708+0000 7f1bef7fe700 1 -- 172.21.15.2:0/520299244 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f1bf404df00 con 0x7f1bf807be10 2023-11-16T16:40:45.131 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.709+0000 7f1bef7fe700 1 --2- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f1be403bff0 0x7f1be403e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:45.131 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.709+0000 7f1bef7fe700 1 -- 172.21.15.2:0/520299244 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1be403b370 con 0x7f1be403bff0 2023-11-16T16:40:45.131 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.709+0000 7f1bfda1e700 1 --2- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f1be403bff0 0x7f1be403e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:45.131 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.711+0000 7f1bfda1e700 1 --2- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f1be403bff0 0x7f1be403e3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1be8003a10 tx=0x7f1be8009270 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:45.131 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.712+0000 7f1bef7fe700 1 -- 172.21.15.2:0/520299244 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f1be403b370 con 0x7f1be403bff0 2023-11-16T16:40:45.131 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.719+0000 7f1c00483700 1 -- 172.21.15.2:0/520299244 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f1bf819ccb0 con 0x7f1be403bff0 2023-11-16T16:40:45.131 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.719+0000 7f1bef7fe700 1 -- 172.21.15.2:0/520299244 <== mgr.14150 v2:172.21.15.2:6800/1793067160 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f1bf819ccb0 con 0x7f1be403bff0 2023-11-16T16:40:45.131 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.720+0000 7f1c00483700 1 -- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f1be403bff0 msgr2=0x7f1be403e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:45.132 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.720+0000 7f1c00483700 1 --2- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f1be403bff0 0x7f1be403e3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1be8003a10 tx=0x7f1be8009270 comp rx=0 tx=0).stop 2023-11-16T16:40:45.132 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.720+0000 7f1c00483700 1 -- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf807be10 msgr2=0x7f1bf819b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:45.132 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.720+0000 7f1c00483700 1 --2- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf807be10 0x7f1bf819b970 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4011980 tx=0x7f1bf4011a60 comp rx=0 tx=0).stop 2023-11-16T16:40:45.132 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.720+0000 7f1c00483700 1 -- 172.21.15.2:0/520299244 shutdown_connections 2023-11-16T16:40:45.132 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.720+0000 7f1c00483700 1 --2- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf807be10 0x7f1bf819b970 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:45.132 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.720+0000 7f1c00483700 1 --2- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f1be403bff0 0x7f1be403e3d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:45.132 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.720+0000 7f1c00483700 1 --2- 172.21.15.2:0/520299244 >> [v2:172.21.15.2:6800/3359672074,v1:172.21.15.2:6801/3359672074] conn(0x7f1be40387b0 0x7f1be403ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:45.132 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.720+0000 7f1c00483700 1 -- 172.21.15.2:0/520299244 >> 172.21.15.2:0/520299244 conn(0x7f1bf8100fd0 msgr2=0x7f1bf81026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:45.132 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.720+0000 7f1c00483700 1 -- 172.21.15.2:0/520299244 shutdown_connections 2023-11-16T16:40:45.132 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:44.720+0000 7f1c00483700 1 -- 172.21.15.2:0/520299244 wait complete. 2023-11-16T16:40:45.132 INFO:teuthology.orchestra.run.smithi002.stdout:mgr epoch 9 is available 2023-11-16T16:40:45.133 INFO:teuthology.orchestra.run.smithi002.stdout:Generating a dashboard self-signed certificate... 2023-11-16T16:40:45.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:45 smithi002 ceph-mon[128712]: [16/Nov/2023:16:40:44] ENGINE Bus STARTING 2023-11-16T16:40:45.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:45 smithi002 ceph-mon[128712]: [16/Nov/2023:16:40:44] ENGINE Serving on https://172.21.15.2:7150 2023-11-16T16:40:45.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:45 smithi002 ceph-mon[128712]: [16/Nov/2023:16:40:44] ENGINE Serving on http://172.21.15.2:8765 2023-11-16T16:40:45.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:45 smithi002 ceph-mon[128712]: [16/Nov/2023:16:40:44] ENGINE Bus STARTED 2023-11-16T16:40:45.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:45 smithi002 ceph-mon[128712]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-16T16:40:45.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:45 smithi002 ceph-mon[128712]: mgrmap e11: x(active, since 1.00776s) 2023-11-16T16:40:45.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:45 smithi002 ceph-mon[128712]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-16T16:40:46.616 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-11-16T16:40:46.616 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.854+0000 7f44a9599700 1 Processor -- start 2023-11-16T16:40:46.616 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.855+0000 7f44a9599700 1 -- start start 2023-11-16T16:40:46.616 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.855+0000 7f44a9599700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44a407be50 0x7f44a407a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:46.616 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.855+0000 7f44a9599700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a407a880 con 0x7f44a407be50 2023-11-16T16:40:46.616 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.855+0000 7f44a2ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44a407be50 0x7f44a407a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:46.617 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.855+0000 7f44a2ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44a407be50 0x7f44a407a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44284/0 (socket says 172.21.15.2:44284) 2023-11-16T16:40:46.617 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.855+0000 7f44a2ffd700 1 -- 172.21.15.2:0/228037507 learned_addr learned my addr 172.21.15.2:0/228037507 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:46.617 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.856+0000 7f44a2ffd700 1 -- 172.21.15.2:0/228037507 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a407a9c0 con 0x7f44a407be50 2023-11-16T16:40:46.617 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.856+0000 7f44a2ffd700 1 --2- 172.21.15.2:0/228037507 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44a407be50 0x7f44a407a2b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f44940098a0 tx=0x7f4494009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a58a7dd6386f98da server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:46.617 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.856+0000 7f44a1ffb700 1 -- 172.21.15.2:0/228037507 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4494004160 con 0x7f44a407be50 2023-11-16T16:40:46.617 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.856+0000 7f44a1ffb700 1 -- 172.21.15.2:0/228037507 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f44940042c0 con 0x7f44a407be50 2023-11-16T16:40:46.617 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.856+0000 7f44a1ffb700 1 -- 172.21.15.2:0/228037507 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f44940119b0 con 0x7f44a407be50 2023-11-16T16:40:46.618 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.857+0000 7f44a9599700 1 -- 172.21.15.2:0/228037507 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44a407be50 msgr2=0x7f44a407a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:46.618 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.857+0000 7f44a9599700 1 --2- 172.21.15.2:0/228037507 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44a407be50 0x7f44a407a2b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f44940098a0 tx=0x7f4494009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:46.618 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.857+0000 7f44a9599700 1 -- 172.21.15.2:0/228037507 shutdown_connections 2023-11-16T16:40:46.618 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.857+0000 7f44a9599700 1 --2- 172.21.15.2:0/228037507 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44a407be50 0x7f44a407a2b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:46.618 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.857+0000 7f44a9599700 1 -- 172.21.15.2:0/228037507 >> 172.21.15.2:0/228037507 conn(0x7f44a4100fd0 msgr2=0x7f44a4103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:46.618 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.857+0000 7f44a9599700 1 -- 172.21.15.2:0/228037507 shutdown_connections 2023-11-16T16:40:46.618 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.857+0000 7f44a9599700 1 -- 172.21.15.2:0/228037507 wait complete. 2023-11-16T16:40:46.618 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.858+0000 7f44a9599700 1 Processor -- start 2023-11-16T16:40:46.619 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.858+0000 7f44a9599700 1 -- start start 2023-11-16T16:40:46.619 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.858+0000 7f44a9599700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44a407be50 0x7f44a419b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:46.619 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.858+0000 7f44a9599700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a419bee0 con 0x7f44a407be50 2023-11-16T16:40:46.619 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.859+0000 7f44a2ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44a407be50 0x7f44a419b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:46.619 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.859+0000 7f44a2ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44a407be50 0x7f44a419b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44294/0 (socket says 172.21.15.2:44294) 2023-11-16T16:40:46.619 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.859+0000 7f44a2ffd700 1 -- 172.21.15.2:0/3088245783 learned_addr learned my addr 172.21.15.2:0/3088245783 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:46.619 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.859+0000 7f44a2ffd700 1 -- 172.21.15.2:0/3088245783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4494009550 con 0x7f44a407be50 2023-11-16T16:40:46.619 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.859+0000 7f44a2ffd700 1 --2- 172.21.15.2:0/3088245783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44a407be50 0x7f44a419b9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4494011980 tx=0x7f4494011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:46.620 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.860+0000 7f448bfff700 1 -- 172.21.15.2:0/3088245783 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4494003a60 con 0x7f44a407be50 2023-11-16T16:40:46.620 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.860+0000 7f448bfff700 1 -- 172.21.15.2:0/3088245783 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4494003bc0 con 0x7f44a407be50 2023-11-16T16:40:46.620 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.860+0000 7f448bfff700 1 -- 172.21.15.2:0/3088245783 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f449401a670 con 0x7f44a407be50 2023-11-16T16:40:46.620 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.860+0000 7f44a9599700 1 -- 172.21.15.2:0/3088245783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a410bfd0 con 0x7f44a407be50 2023-11-16T16:40:46.620 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.860+0000 7f44a9599700 1 -- 172.21.15.2:0/3088245783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a410be40 con 0x7f44a407be50 2023-11-16T16:40:46.620 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.861+0000 7f448bfff700 1 -- 172.21.15.2:0/3088245783 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f4494024a00 con 0x7f44a407be50 2023-11-16T16:40:46.620 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.861+0000 7f44a9599700 1 -- 172.21.15.2:0/3088245783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a410c160 con 0x7f44a407be50 2023-11-16T16:40:46.620 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.861+0000 7f448bfff700 1 --2- 172.21.15.2:0/3088245783 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f448c0387b0 0x7f448c03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:46.621 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.861+0000 7f448bfff700 1 -- 172.21.15.2:0/3088245783 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f449404ca10 con 0x7f44a407be50 2023-11-16T16:40:46.621 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.862+0000 7f44a27fc700 1 --2- 172.21.15.2:0/3088245783 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f448c0387b0 0x7f448c03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:46.621 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.862+0000 7f44a27fc700 1 --2- 172.21.15.2:0/3088245783 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f448c0387b0 0x7f448c03ac60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4498006fd0 tx=0x7f4498006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:46.621 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:45.867+0000 7f448bfff700 1 -- 172.21.15.2:0/3088245783 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4494024440 con 0x7f44a407be50 2023-11-16T16:40:46.621 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.083+0000 7f44a9599700 1 -- 172.21.15.2:0/3088245783 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f44a41032d0 con 0x7f448c0387b0 2023-11-16T16:40:46.621 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.121+0000 7f448bfff700 1 -- 172.21.15.2:0/3088245783 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f44a41032d0 con 0x7f448c0387b0 2023-11-16T16:40:46.621 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.123+0000 7f44a9599700 1 -- 172.21.15.2:0/3088245783 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f448c0387b0 msgr2=0x7f448c03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:46.621 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.123+0000 7f44a9599700 1 --2- 172.21.15.2:0/3088245783 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f448c0387b0 0x7f448c03ac60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4498006fd0 tx=0x7f4498006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:46.622 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.123+0000 7f44a9599700 1 -- 172.21.15.2:0/3088245783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44a407be50 msgr2=0x7f44a419b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:46.622 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.123+0000 7f44a9599700 1 --2- 172.21.15.2:0/3088245783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44a407be50 0x7f44a419b9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4494011980 tx=0x7f4494011a60 comp rx=0 tx=0).stop 2023-11-16T16:40:46.622 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.123+0000 7f44a9599700 1 -- 172.21.15.2:0/3088245783 shutdown_connections 2023-11-16T16:40:46.622 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.123+0000 7f44a9599700 1 --2- 172.21.15.2:0/3088245783 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f448c0387b0 0x7f448c03ac60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:46.622 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.123+0000 7f44a9599700 1 --2- 172.21.15.2:0/3088245783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44a407be50 0x7f44a419b9a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:46.622 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.123+0000 7f44a9599700 1 -- 172.21.15.2:0/3088245783 >> 172.21.15.2:0/3088245783 conn(0x7f44a4100fd0 msgr2=0x7f44a41026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:46.622 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.123+0000 7f44a9599700 1 -- 172.21.15.2:0/3088245783 shutdown_connections 2023-11-16T16:40:46.623 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:46.123+0000 7f44a9599700 1 -- 172.21.15.2:0/3088245783 wait complete. 2023-11-16T16:40:46.623 INFO:teuthology.orchestra.run.smithi002.stdout:Creating initial admin user... 2023-11-16T16:40:46.900 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:46 smithi002 ceph-mon[128712]: mgrmap e12: x(active, since 2s) 2023-11-16T16:40:46.901 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:46 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:40:46.901 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:46 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:40:46.901 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:46 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:40:48.099 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$zGauHaFk1pNOtaKVBbOvf..Sx02Nd4FKnSb.iI5RTwtT4yEtfmNrG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1700152847, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-11-16T16:40:48.099 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.181+0000 7f6d61e57700 1 Processor -- start 2023-11-16T16:40:48.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.181+0000 7f6d61e57700 1 -- start start 2023-11-16T16:40:48.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.182+0000 7f6d61e57700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d5c07be30 0x7f6d5c07a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:48.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.182+0000 7f6d61e57700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c07a860 con 0x7f6d5c07be30 2023-11-16T16:40:48.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.182+0000 7f6d5b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d5c07be30 0x7f6d5c07a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:48.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.182+0000 7f6d5b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d5c07be30 0x7f6d5c07a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44310/0 (socket says 172.21.15.2:44310) 2023-11-16T16:40:48.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.182+0000 7f6d5b7fe700 1 -- 172.21.15.2:0/2784322030 learned_addr learned my addr 172.21.15.2:0/2784322030 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:48.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.183+0000 7f6d5b7fe700 1 -- 172.21.15.2:0/2784322030 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d5c07a9a0 con 0x7f6d5c07be30 2023-11-16T16:40:48.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.183+0000 7f6d5b7fe700 1 --2- 172.21.15.2:0/2784322030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d5c07be30 0x7f6d5c07a290 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0098a0 tx=0x7f6d4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9334d6a5171edd3c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:48.101 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.183+0000 7f6d5a7fc700 1 -- 172.21.15.2:0/2784322030 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6d4c004160 con 0x7f6d5c07be30 2023-11-16T16:40:48.101 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.183+0000 7f6d5a7fc700 1 -- 172.21.15.2:0/2784322030 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d4c0042c0 con 0x7f6d5c07be30 2023-11-16T16:40:48.101 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.183+0000 7f6d5a7fc700 1 -- 172.21.15.2:0/2784322030 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6d4c0119b0 con 0x7f6d5c07be30 2023-11-16T16:40:48.101 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.184+0000 7f6d61e57700 1 -- 172.21.15.2:0/2784322030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d5c07be30 msgr2=0x7f6d5c07a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:48.101 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.184+0000 7f6d61e57700 1 --2- 172.21.15.2:0/2784322030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d5c07be30 0x7f6d5c07a290 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c0098a0 tx=0x7f6d4c009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:48.101 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.184+0000 7f6d61e57700 1 -- 172.21.15.2:0/2784322030 shutdown_connections 2023-11-16T16:40:48.101 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.184+0000 7f6d61e57700 1 --2- 172.21.15.2:0/2784322030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d5c07be30 0x7f6d5c07a290 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:48.101 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.184+0000 7f6d61e57700 1 -- 172.21.15.2:0/2784322030 >> 172.21.15.2:0/2784322030 conn(0x7f6d5c100fd0 msgr2=0x7f6d5c103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:48.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.184+0000 7f6d61e57700 1 -- 172.21.15.2:0/2784322030 shutdown_connections 2023-11-16T16:40:48.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.184+0000 7f6d61e57700 1 -- 172.21.15.2:0/2784322030 wait complete. 2023-11-16T16:40:48.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.185+0000 7f6d61e57700 1 Processor -- start 2023-11-16T16:40:48.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.185+0000 7f6d61e57700 1 -- start start 2023-11-16T16:40:48.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.186+0000 7f6d61e57700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d5c07be30 0x7f6d5c19b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:48.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.186+0000 7f6d61e57700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c19bed0 con 0x7f6d5c07be30 2023-11-16T16:40:48.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.186+0000 7f6d5b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d5c07be30 0x7f6d5c19b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:48.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.186+0000 7f6d5b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d5c07be30 0x7f6d5c19b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44314/0 (socket says 172.21.15.2:44314) 2023-11-16T16:40:48.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.186+0000 7f6d5b7fe700 1 -- 172.21.15.2:0/2708450256 learned_addr learned my addr 172.21.15.2:0/2708450256 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:48.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.186+0000 7f6d5b7fe700 1 -- 172.21.15.2:0/2708450256 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4c009550 con 0x7f6d5c07be30 2023-11-16T16:40:48.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.187+0000 7f6d5b7fe700 1 --2- 172.21.15.2:0/2708450256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d5c07be30 0x7f6d5c19b990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c003e60 tx=0x7f6d4c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:48.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.187+0000 7f6d58ff9700 1 -- 172.21.15.2:0/2708450256 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6d4c003970 con 0x7f6d5c07be30 2023-11-16T16:40:48.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.187+0000 7f6d58ff9700 1 -- 172.21.15.2:0/2708450256 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d4c003ad0 con 0x7f6d5c07be30 2023-11-16T16:40:48.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.187+0000 7f6d61e57700 1 -- 172.21.15.2:0/2708450256 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d5c10bf90 con 0x7f6d5c07be30 2023-11-16T16:40:48.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.187+0000 7f6d58ff9700 1 -- 172.21.15.2:0/2708450256 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6d4c0195e0 con 0x7f6d5c07be30 2023-11-16T16:40:48.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.187+0000 7f6d61e57700 1 -- 172.21.15.2:0/2708450256 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d5c10be00 con 0x7f6d5c07be30 2023-11-16T16:40:48.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.188+0000 7f6d58ff9700 1 -- 172.21.15.2:0/2708450256 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f6d4c0198a0 con 0x7f6d5c07be30 2023-11-16T16:40:48.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.189+0000 7f6d61e57700 1 -- 172.21.15.2:0/2708450256 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d5c10c120 con 0x7f6d5c07be30 2023-11-16T16:40:48.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.189+0000 7f6d58ff9700 1 --2- 172.21.15.2:0/2708450256 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6d44038710 0x7f6d4403abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:48.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.189+0000 7f6d58ff9700 1 -- 172.21.15.2:0/2708450256 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6d4c04c9e0 con 0x7f6d5c07be30 2023-11-16T16:40:48.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.189+0000 7f6d5affd700 1 --2- 172.21.15.2:0/2708450256 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6d44038710 0x7f6d4403abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:48.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.190+0000 7f6d5affd700 1 --2- 172.21.15.2:0/2708450256 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6d44038710 0x7f6d4403abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6d50006fd0 tx=0x7f6d50006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:48.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.194+0000 7f6d58ff9700 1 -- 172.21.15.2:0/2708450256 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6d4c02b340 con 0x7f6d5c07be30 2023-11-16T16:40:48.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.417+0000 7f6d61e57700 1 -- 172.21.15.2:0/2708450256 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f6d5c061f40 con 0x7f6d44038710 2023-11-16T16:40:48.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.664+0000 7f6d58ff9700 1 -- 172.21.15.2:0/2708450256 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f6d5c061f40 con 0x7f6d44038710 2023-11-16T16:40:48.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.666+0000 7f6d61e57700 1 -- 172.21.15.2:0/2708450256 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6d44038710 msgr2=0x7f6d4403abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:48.105 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.666+0000 7f6d61e57700 1 --2- 172.21.15.2:0/2708450256 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6d44038710 0x7f6d4403abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6d50006fd0 tx=0x7f6d50006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:48.105 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.666+0000 7f6d61e57700 1 -- 172.21.15.2:0/2708450256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d5c07be30 msgr2=0x7f6d5c19b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:48.105 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.666+0000 7f6d61e57700 1 --2- 172.21.15.2:0/2708450256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d5c07be30 0x7f6d5c19b990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c003e60 tx=0x7f6d4c011aa0 comp rx=0 tx=0).stop 2023-11-16T16:40:48.105 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.667+0000 7f6d61e57700 1 -- 172.21.15.2:0/2708450256 shutdown_connections 2023-11-16T16:40:48.105 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.667+0000 7f6d61e57700 1 --2- 172.21.15.2:0/2708450256 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6d44038710 0x7f6d4403abc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:48.105 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.667+0000 7f6d61e57700 1 --2- 172.21.15.2:0/2708450256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d5c07be30 0x7f6d5c19b990 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:48.105 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.667+0000 7f6d61e57700 1 -- 172.21.15.2:0/2708450256 >> 172.21.15.2:0/2708450256 conn(0x7f6d5c100fd0 msgr2=0x7f6d5c102710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:48.105 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.667+0000 7f6d61e57700 1 -- 172.21.15.2:0/2708450256 shutdown_connections 2023-11-16T16:40:48.105 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:47.667+0000 7f6d61e57700 1 -- 172.21.15.2:0/2708450256 wait complete. 2023-11-16T16:40:48.105 INFO:teuthology.orchestra.run.smithi002.stdout:Fetching dashboard port number... 2023-11-16T16:40:48.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:47 smithi002 ceph-mon[128712]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:40:48.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:47 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:40:48.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:47 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:40:49.076 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:48 smithi002 ceph-mon[128712]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:40:49.482 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 8443 2023-11-16T16:40:49.482 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.631+0000 7fb577da8700 1 Processor -- start 2023-11-16T16:40:49.482 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.631+0000 7fb577da8700 1 -- start start 2023-11-16T16:40:49.482 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.632+0000 7fb577da8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb57007be30 0x7fb57007a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:49.483 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.632+0000 7fb577da8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57007a860 con 0x7fb57007be30 2023-11-16T16:40:49.483 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.632+0000 7fb575b44700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb57007be30 0x7fb57007a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:49.483 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.632+0000 7fb575b44700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb57007be30 0x7fb57007a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44324/0 (socket says 172.21.15.2:44324) 2023-11-16T16:40:49.483 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.632+0000 7fb575b44700 1 -- 172.21.15.2:0/3844063779 learned_addr learned my addr 172.21.15.2:0/3844063779 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:49.483 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.633+0000 7fb575b44700 1 -- 172.21.15.2:0/3844063779 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57007a9a0 con 0x7fb57007be30 2023-11-16T16:40:49.483 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.633+0000 7fb575b44700 1 --2- 172.21.15.2:0/3844063779 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb57007be30 0x7fb57007a290 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb56c0098a0 tx=0x7fb56c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b169de1e7546937d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:49.484 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.633+0000 7fb574b42700 1 -- 172.21.15.2:0/3844063779 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb56c004160 con 0x7fb57007be30 2023-11-16T16:40:49.484 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.633+0000 7fb574b42700 1 -- 172.21.15.2:0/3844063779 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb56c0042c0 con 0x7fb57007be30 2023-11-16T16:40:49.484 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.633+0000 7fb574b42700 1 -- 172.21.15.2:0/3844063779 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb56c0119b0 con 0x7fb57007be30 2023-11-16T16:40:49.484 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.634+0000 7fb577da8700 1 -- 172.21.15.2:0/3844063779 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb57007be30 msgr2=0x7fb57007a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:49.484 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.634+0000 7fb577da8700 1 --2- 172.21.15.2:0/3844063779 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb57007be30 0x7fb57007a290 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb56c0098a0 tx=0x7fb56c009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:49.488 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.634+0000 7fb577da8700 1 -- 172.21.15.2:0/3844063779 shutdown_connections 2023-11-16T16:40:49.488 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.634+0000 7fb577da8700 1 --2- 172.21.15.2:0/3844063779 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb57007be30 0x7fb57007a290 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:49.488 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.634+0000 7fb577da8700 1 -- 172.21.15.2:0/3844063779 >> 172.21.15.2:0/3844063779 conn(0x7fb570100fd0 msgr2=0x7fb570103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:49.489 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.634+0000 7fb577da8700 1 -- 172.21.15.2:0/3844063779 shutdown_connections 2023-11-16T16:40:49.489 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.634+0000 7fb577da8700 1 -- 172.21.15.2:0/3844063779 wait complete. 2023-11-16T16:40:49.489 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.636+0000 7fb577da8700 1 Processor -- start 2023-11-16T16:40:49.489 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.636+0000 7fb577da8700 1 -- start start 2023-11-16T16:40:49.489 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.636+0000 7fb577da8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb57007be30 0x7fb57019b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:49.489 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.636+0000 7fb577da8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57019be90 con 0x7fb57007be30 2023-11-16T16:40:49.490 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.637+0000 7fb575b44700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb57007be30 0x7fb57019b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:49.490 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.637+0000 7fb575b44700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb57007be30 0x7fb57019b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44328/0 (socket says 172.21.15.2:44328) 2023-11-16T16:40:49.490 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.637+0000 7fb575b44700 1 -- 172.21.15.2:0/4252693546 learned_addr learned my addr 172.21.15.2:0/4252693546 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:49.490 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.637+0000 7fb575b44700 1 -- 172.21.15.2:0/4252693546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56c009550 con 0x7fb57007be30 2023-11-16T16:40:49.490 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.637+0000 7fb575b44700 1 --2- 172.21.15.2:0/4252693546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb57007be30 0x7fb57019b950 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb56c011980 tx=0x7fb56c011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:49.491 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.637+0000 7fb566ffd700 1 -- 172.21.15.2:0/4252693546 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb56c003a60 con 0x7fb57007be30 2023-11-16T16:40:49.491 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.638+0000 7fb566ffd700 1 -- 172.21.15.2:0/4252693546 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb56c003bc0 con 0x7fb57007be30 2023-11-16T16:40:49.491 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.638+0000 7fb577da8700 1 -- 172.21.15.2:0/4252693546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57010bf90 con 0x7fb57007be30 2023-11-16T16:40:49.491 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.638+0000 7fb566ffd700 1 -- 172.21.15.2:0/4252693546 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb56c01a670 con 0x7fb57007be30 2023-11-16T16:40:49.491 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.638+0000 7fb577da8700 1 -- 172.21.15.2:0/4252693546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57010be00 con 0x7fb57007be30 2023-11-16T16:40:49.491 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.639+0000 7fb577da8700 1 -- 172.21.15.2:0/4252693546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb57010c120 con 0x7fb57007be30 2023-11-16T16:40:49.492 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.639+0000 7fb566ffd700 1 -- 172.21.15.2:0/4252693546 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fb56c01a930 con 0x7fb57007be30 2023-11-16T16:40:49.492 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.639+0000 7fb566ffd700 1 --2- 172.21.15.2:0/4252693546 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb55c038760 0x7fb55c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:49.492 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.639+0000 7fb566ffd700 1 -- 172.21.15.2:0/4252693546 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb56c04c910 con 0x7fb57007be30 2023-11-16T16:40:49.492 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.640+0000 7fb575343700 1 --2- 172.21.15.2:0/4252693546 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb55c038760 0x7fb55c03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:49.492 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.640+0000 7fb575343700 1 --2- 172.21.15.2:0/4252693546 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb55c038760 0x7fb55c03ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb560006fd0 tx=0x7fb560006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:49.492 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.645+0000 7fb566ffd700 1 -- 172.21.15.2:0/4252693546 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb56c018b80 con 0x7fb57007be30 2023-11-16T16:40:49.493 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.872+0000 7fb577da8700 1 -- 172.21.15.2:0/4252693546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fb5700621a0 con 0x7fb57007be30 2023-11-16T16:40:49.493 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.874+0000 7fb566ffd700 1 -- 172.21.15.2:0/4252693546 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fb56c018350 con 0x7fb57007be30 2023-11-16T16:40:49.493 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.876+0000 7fb577da8700 1 -- 172.21.15.2:0/4252693546 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb55c038760 msgr2=0x7fb55c03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:49.493 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.876+0000 7fb577da8700 1 --2- 172.21.15.2:0/4252693546 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb55c038760 0x7fb55c03ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb560006fd0 tx=0x7fb560006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:49.493 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.876+0000 7fb577da8700 1 -- 172.21.15.2:0/4252693546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb57007be30 msgr2=0x7fb57019b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:49.493 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.876+0000 7fb577da8700 1 --2- 172.21.15.2:0/4252693546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb57007be30 0x7fb57019b950 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb56c011980 tx=0x7fb56c011a60 comp rx=0 tx=0).stop 2023-11-16T16:40:49.494 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.877+0000 7fb577da8700 1 -- 172.21.15.2:0/4252693546 shutdown_connections 2023-11-16T16:40:49.494 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.877+0000 7fb577da8700 1 --2- 172.21.15.2:0/4252693546 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb55c038760 0x7fb55c03ac10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:49.494 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.877+0000 7fb577da8700 1 --2- 172.21.15.2:0/4252693546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb57007be30 0x7fb57019b950 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:49.494 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.877+0000 7fb577da8700 1 -- 172.21.15.2:0/4252693546 >> 172.21.15.2:0/4252693546 conn(0x7fb570100fd0 msgr2=0x7fb5701026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:49.494 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.877+0000 7fb577da8700 1 -- 172.21.15.2:0/4252693546 shutdown_connections 2023-11-16T16:40:49.495 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:48.877+0000 7fb577da8700 1 -- 172.21.15.2:0/4252693546 wait complete. 2023-11-16T16:40:49.507 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-16T16:40:49.507 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2023-11-16T16:40:49.523 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-16T16:40:49.523 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2023-11-16T16:40:49.523 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2023-11-16T16:40:49.524 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-11-16T16:40:49.525 INFO:teuthology.orchestra.run.smithi002.stdout:Ceph Dashboard is now available at: 2023-11-16T16:40:49.526 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:40:49.526 INFO:teuthology.orchestra.run.smithi002.stdout: URL: https://smithi002.front.sepia.ceph.com:8443/ 2023-11-16T16:40:49.526 INFO:teuthology.orchestra.run.smithi002.stdout: User: admin 2023-11-16T16:40:49.526 INFO:teuthology.orchestra.run.smithi002.stdout: Password: xot09wnq2d 2023-11-16T16:40:49.526 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:40:49.527 INFO:teuthology.orchestra.run.smithi002.stdout:Saving cluster configuration to /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config directory 2023-11-16T16:40:49.527 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling autotune for osd_memory_target 2023-11-16T16:40:49.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:49 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/4252693546' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-11-16T16:40:50.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:50 smithi002 ceph-mon[128712]: mgrmap e13: x(active, since 6s) 2023-11-16T16:40:50.966 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.340+0000 7fb487e66700 1 Processor -- start 2023-11-16T16:40:50.966 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.340+0000 7fb487e66700 1 -- start start 2023-11-16T16:40:50.966 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.340+0000 7fb487e66700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb480106510 0x7fb4801088f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:50.966 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.340+0000 7fb487e66700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb480079de0 con 0x7fb480106510 2023-11-16T16:40:50.966 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.341+0000 7fb485c02700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb480106510 0x7fb4801088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:50.967 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.341+0000 7fb485c02700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb480106510 0x7fb4801088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44344/0 (socket says 172.21.15.2:44344) 2023-11-16T16:40:50.967 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.341+0000 7fb485c02700 1 -- 172.21.15.2:0/2175266450 learned_addr learned my addr 172.21.15.2:0/2175266450 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:50.967 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.341+0000 7fb485c02700 1 -- 172.21.15.2:0/2175266450 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb480108e30 con 0x7fb480106510 2023-11-16T16:40:50.967 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.341+0000 7fb485c02700 1 --2- 172.21.15.2:0/2175266450 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb480106510 0x7fb4801088f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb47c0098a0 tx=0x7fb47c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2455babc5f8ebe5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:50.967 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.342+0000 7fb484c00700 1 -- 172.21.15.2:0/2175266450 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb47c004160 con 0x7fb480106510 2023-11-16T16:40:50.967 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.342+0000 7fb484c00700 1 -- 172.21.15.2:0/2175266450 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb47c0042c0 con 0x7fb480106510 2023-11-16T16:40:50.967 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.342+0000 7fb484c00700 1 -- 172.21.15.2:0/2175266450 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb47c0119b0 con 0x7fb480106510 2023-11-16T16:40:50.967 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.342+0000 7fb487e66700 1 -- 172.21.15.2:0/2175266450 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb480106510 msgr2=0x7fb4801088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:50.967 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.342+0000 7fb487e66700 1 --2- 172.21.15.2:0/2175266450 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb480106510 0x7fb4801088f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb47c0098a0 tx=0x7fb47c009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:50.967 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.342+0000 7fb487e66700 1 -- 172.21.15.2:0/2175266450 shutdown_connections 2023-11-16T16:40:50.967 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.342+0000 7fb487e66700 1 --2- 172.21.15.2:0/2175266450 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb480106510 0x7fb4801088f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:50.968 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.342+0000 7fb487e66700 1 -- 172.21.15.2:0/2175266450 >> 172.21.15.2:0/2175266450 conn(0x7fb4800ffed0 msgr2=0x7fb480102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:50.968 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.343+0000 7fb487e66700 1 -- 172.21.15.2:0/2175266450 shutdown_connections 2023-11-16T16:40:50.968 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.343+0000 7fb487e66700 1 -- 172.21.15.2:0/2175266450 wait complete. 2023-11-16T16:40:50.968 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.343+0000 7fb487e66700 1 Processor -- start 2023-11-16T16:40:50.968 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.344+0000 7fb487e66700 1 -- start start 2023-11-16T16:40:50.968 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.344+0000 7fb487e66700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb480106510 0x7fb48019b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:50.968 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.344+0000 7fb487e66700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48019bef0 con 0x7fb480106510 2023-11-16T16:40:50.968 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.344+0000 7fb485c02700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb480106510 0x7fb48019b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:50.968 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.344+0000 7fb485c02700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb480106510 0x7fb48019b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44348/0 (socket says 172.21.15.2:44348) 2023-11-16T16:40:50.968 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.344+0000 7fb485c02700 1 -- 172.21.15.2:0/2649611766 learned_addr learned my addr 172.21.15.2:0/2649611766 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:50.968 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.345+0000 7fb485c02700 1 -- 172.21.15.2:0/2649611766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c009550 con 0x7fb480106510 2023-11-16T16:40:50.968 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.345+0000 7fb485c02700 1 --2- 172.21.15.2:0/2649611766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb480106510 0x7fb48019b9b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb47c011980 tx=0x7fb47c011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:50.968 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.345+0000 7fb476ffd700 1 -- 172.21.15.2:0/2649611766 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb47c003a60 con 0x7fb480106510 2023-11-16T16:40:50.969 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.345+0000 7fb476ffd700 1 -- 172.21.15.2:0/2649611766 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb47c003bc0 con 0x7fb480106510 2023-11-16T16:40:50.969 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.345+0000 7fb476ffd700 1 -- 172.21.15.2:0/2649611766 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb47c01a670 con 0x7fb480106510 2023-11-16T16:40:50.969 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.345+0000 7fb487e66700 1 -- 172.21.15.2:0/2649611766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb480109d50 con 0x7fb480106510 2023-11-16T16:40:50.969 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.345+0000 7fb487e66700 1 -- 172.21.15.2:0/2649611766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb480109bc0 con 0x7fb480106510 2023-11-16T16:40:50.969 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.346+0000 7fb487e66700 1 -- 172.21.15.2:0/2649611766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb480109ee0 con 0x7fb480106510 2023-11-16T16:40:50.969 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.347+0000 7fb476ffd700 1 -- 172.21.15.2:0/2649611766 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7fb47c024a00 con 0x7fb480106510 2023-11-16T16:40:50.969 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.347+0000 7fb476ffd700 1 --2- 172.21.15.2:0/2649611766 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb46c038850 0x7fb46c03ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:50.969 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.348+0000 7fb476ffd700 1 -- 172.21.15.2:0/2649611766 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb47c04cdf0 con 0x7fb480106510 2023-11-16T16:40:50.969 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.348+0000 7fb485401700 1 --2- 172.21.15.2:0/2649611766 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb46c038850 0x7fb46c03ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:50.969 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.348+0000 7fb485401700 1 --2- 172.21.15.2:0/2649611766 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb46c038850 0x7fb46c03ad00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb470006fd0 tx=0x7fb470006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:50.969 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.352+0000 7fb476ffd700 1 -- 172.21.15.2:0/2649611766 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb47c024440 con 0x7fb480106510 2023-11-16T16:40:50.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.559+0000 7fb487e66700 1 -- 172.21.15.2:0/2649611766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fb48019f630 con 0x7fb480106510 2023-11-16T16:40:50.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.559+0000 7fb476ffd700 1 -- 172.21.15.2:0/2649611766 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fb47c018350 con 0x7fb480106510 2023-11-16T16:40:50.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.561+0000 7fb487e66700 1 -- 172.21.15.2:0/2649611766 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb46c038850 msgr2=0x7fb46c03ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:50.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.561+0000 7fb487e66700 1 --2- 172.21.15.2:0/2649611766 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb46c038850 0x7fb46c03ad00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb470006fd0 tx=0x7fb470006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:50.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.561+0000 7fb487e66700 1 -- 172.21.15.2:0/2649611766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb480106510 msgr2=0x7fb48019b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:50.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.561+0000 7fb487e66700 1 --2- 172.21.15.2:0/2649611766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb480106510 0x7fb48019b9b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb47c011980 tx=0x7fb47c011a60 comp rx=0 tx=0).stop 2023-11-16T16:40:50.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.561+0000 7fb487e66700 1 -- 172.21.15.2:0/2649611766 shutdown_connections 2023-11-16T16:40:50.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.562+0000 7fb487e66700 1 --2- 172.21.15.2:0/2649611766 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb46c038850 0x7fb46c03ad00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:50.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.562+0000 7fb487e66700 1 --2- 172.21.15.2:0/2649611766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb480106510 0x7fb48019b9b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:50.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.562+0000 7fb487e66700 1 -- 172.21.15.2:0/2649611766 >> 172.21.15.2:0/2649611766 conn(0x7fb4800ffed0 msgr2=0x7fb480100c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:50.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.562+0000 7fb487e66700 1 -- 172.21.15.2:0/2649611766 shutdown_connections 2023-11-16T16:40:50.970 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:50.562+0000 7fb487e66700 1 -- 172.21.15.2:0/2649611766 wait complete. 2023-11-16T16:40:52.347 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.505+0000 7fb6ce60d700 1 Processor -- start 2023-11-16T16:40:52.347 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.505+0000 7fb6ce60d700 1 -- start start 2023-11-16T16:40:52.347 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.505+0000 7fb6ce60d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6c8105a20 0x7fb6c8105e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:52.347 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.505+0000 7fb6ce60d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c8106400 con 0x7fb6c8105a20 2023-11-16T16:40:52.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.505+0000 7fb6c7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6c8105a20 0x7fb6c8105e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:52.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.506+0000 7fb6c7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6c8105a20 0x7fb6c8105e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44350/0 (socket says 172.21.15.2:44350) 2023-11-16T16:40:52.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.506+0000 7fb6c7fff700 1 -- 172.21.15.2:0/1205869249 learned_addr learned my addr 172.21.15.2:0/1205869249 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:52.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.506+0000 7fb6c7fff700 1 -- 172.21.15.2:0/1205869249 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c8106c20 con 0x7fb6c8105a20 2023-11-16T16:40:52.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.506+0000 7fb6c7fff700 1 --2- 172.21.15.2:0/1205869249 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6c8105a20 0x7fb6c8105e30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb6b80098a0 tx=0x7fb6b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8176dd8dbfbdc3be server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:52.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.506+0000 7fb6c6ffd700 1 -- 172.21.15.2:0/1205869249 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb6b8004160 con 0x7fb6c8105a20 2023-11-16T16:40:52.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.506+0000 7fb6c6ffd700 1 -- 172.21.15.2:0/1205869249 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb6b80042c0 con 0x7fb6c8105a20 2023-11-16T16:40:52.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.506+0000 7fb6c6ffd700 1 -- 172.21.15.2:0/1205869249 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb6b80119b0 con 0x7fb6c8105a20 2023-11-16T16:40:52.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.507+0000 7fb6ce60d700 1 -- 172.21.15.2:0/1205869249 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6c8105a20 msgr2=0x7fb6c8105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:52.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.507+0000 7fb6ce60d700 1 --2- 172.21.15.2:0/1205869249 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6c8105a20 0x7fb6c8105e30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb6b80098a0 tx=0x7fb6b8009bb0 comp rx=0 tx=0).stop 2023-11-16T16:40:52.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.507+0000 7fb6ce60d700 1 -- 172.21.15.2:0/1205869249 shutdown_connections 2023-11-16T16:40:52.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.507+0000 7fb6ce60d700 1 --2- 172.21.15.2:0/1205869249 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6c8105a20 0x7fb6c8105e30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:52.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.507+0000 7fb6ce60d700 1 -- 172.21.15.2:0/1205869249 >> 172.21.15.2:0/1205869249 conn(0x7fb6c8100fd0 msgr2=0x7fb6c8103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:52.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.507+0000 7fb6ce60d700 1 -- 172.21.15.2:0/1205869249 shutdown_connections 2023-11-16T16:40:52.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.507+0000 7fb6ce60d700 1 -- 172.21.15.2:0/1205869249 wait complete. 2023-11-16T16:40:52.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.508+0000 7fb6ce60d700 1 Processor -- start 2023-11-16T16:40:52.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.508+0000 7fb6ce60d700 1 -- start start 2023-11-16T16:40:52.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.509+0000 7fb6ce60d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6c8105a20 0x7fb6c819b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:52.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.509+0000 7fb6ce60d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c819bf10 con 0x7fb6c8105a20 2023-11-16T16:40:52.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.509+0000 7fb6c7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6c8105a20 0x7fb6c819b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:52.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.509+0000 7fb6c7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6c8105a20 0x7fb6c819b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44358/0 (socket says 172.21.15.2:44358) 2023-11-16T16:40:52.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.509+0000 7fb6c7fff700 1 -- 172.21.15.2:0/1846136992 learned_addr learned my addr 172.21.15.2:0/1846136992 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:52.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.509+0000 7fb6c7fff700 1 -- 172.21.15.2:0/1846136992 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b8009550 con 0x7fb6c8105a20 2023-11-16T16:40:52.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.509+0000 7fb6c7fff700 1 --2- 172.21.15.2:0/1846136992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6c8105a20 0x7fb6c819b9d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8000c00 tx=0x7fb6b8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:52.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.510+0000 7fb6c57fa700 1 -- 172.21.15.2:0/1846136992 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb6b80039f0 con 0x7fb6c8105a20 2023-11-16T16:40:52.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.510+0000 7fb6c57fa700 1 -- 172.21.15.2:0/1846136992 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb6b8003b50 con 0x7fb6c8105a20 2023-11-16T16:40:52.351 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.510+0000 7fb6c57fa700 1 -- 172.21.15.2:0/1846136992 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb6b801a780 con 0x7fb6c8105a20 2023-11-16T16:40:52.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.510+0000 7fb6ce60d700 1 -- 172.21.15.2:0/1846136992 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c8109fe0 con 0x7fb6c8105a20 2023-11-16T16:40:52.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.510+0000 7fb6ce60d700 1 -- 172.21.15.2:0/1846136992 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c8109e50 con 0x7fb6c8105a20 2023-11-16T16:40:52.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.511+0000 7fb6ce60d700 1 -- 172.21.15.2:0/1846136992 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c8062120 con 0x7fb6c8105a20 2023-11-16T16:40:52.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.511+0000 7fb6c57fa700 1 -- 172.21.15.2:0/1846136992 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7fb6b8024440 con 0x7fb6c8105a20 2023-11-16T16:40:52.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.511+0000 7fb6c57fa700 1 --2- 172.21.15.2:0/1846136992 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb6b0038850 0x7fb6b003ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:52.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.512+0000 7fb6c57fa700 1 -- 172.21.15.2:0/1846136992 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb6b804c990 con 0x7fb6c8105a20 2023-11-16T16:40:52.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.512+0000 7fb6c77fe700 1 --2- 172.21.15.2:0/1846136992 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb6b0038850 0x7fb6b003ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:52.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.512+0000 7fb6c77fe700 1 --2- 172.21.15.2:0/1846136992 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb6b0038850 0x7fb6b003ad00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc006fd0 tx=0x7fb6bc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:52.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.516+0000 7fb6c57fa700 1 -- 172.21.15.2:0/1846136992 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb6b8018db0 con 0x7fb6c8105a20 2023-11-16T16:40:52.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.838+0000 7fb6ce60d700 1 -- 172.21.15.2:0/1846136992 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fb6c810a170 con 0x7fb6c8105a20 2023-11-16T16:40:52.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.840+0000 7fb6c57fa700 1 -- 172.21.15.2:0/1846136992 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) v1 ==== 153+0+0 (secure 0 0 0) 0x7fb6b8018350 con 0x7fb6c8105a20 2023-11-16T16:40:52.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.843+0000 7fb6ce60d700 1 -- 172.21.15.2:0/1846136992 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb6b0038850 msgr2=0x7fb6b003ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:52.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.843+0000 7fb6ce60d700 1 --2- 172.21.15.2:0/1846136992 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb6b0038850 0x7fb6b003ad00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc006fd0 tx=0x7fb6bc006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:52.354 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.843+0000 7fb6ce60d700 1 -- 172.21.15.2:0/1846136992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6c8105a20 msgr2=0x7fb6c819b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:52.354 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.843+0000 7fb6ce60d700 1 --2- 172.21.15.2:0/1846136992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6c8105a20 0x7fb6c819b9d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8000c00 tx=0x7fb6b8011a30 comp rx=0 tx=0).stop 2023-11-16T16:40:52.354 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.843+0000 7fb6ce60d700 1 -- 172.21.15.2:0/1846136992 shutdown_connections 2023-11-16T16:40:52.354 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.843+0000 7fb6ce60d700 1 --2- 172.21.15.2:0/1846136992 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb6b0038850 0x7fb6b003ad00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:52.354 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.843+0000 7fb6ce60d700 1 --2- 172.21.15.2:0/1846136992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6c8105a20 0x7fb6c819b9d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:52.354 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.843+0000 7fb6ce60d700 1 -- 172.21.15.2:0/1846136992 >> 172.21.15.2:0/1846136992 conn(0x7fb6c8100fd0 msgr2=0x7fb6c8102640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:52.355 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.843+0000 7fb6ce60d700 1 -- 172.21.15.2:0/1846136992 shutdown_connections 2023-11-16T16:40:52.355 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2023-11-16T16:40:51.843+0000 7fb6ce60d700 1 -- 172.21.15.2:0/1846136992 wait complete. 2023-11-16T16:40:52.355 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-11-16T16:40:52.355 INFO:teuthology.orchestra.run.smithi002.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-11-16T16:40:52.355 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:40:52.355 INFO:teuthology.orchestra.run.smithi002.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-11-16T16:40:52.355 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:40:52.356 INFO:teuthology.orchestra.run.smithi002.stdout:Or, if you are only running a single cluster on this host: 2023-11-16T16:40:52.356 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:40:52.356 INFO:teuthology.orchestra.run.smithi002.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-11-16T16:40:52.356 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:40:52.356 INFO:teuthology.orchestra.run.smithi002.stdout:Please consider enabling telemetry to help improve Ceph: 2023-11-16T16:40:52.356 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:40:52.356 INFO:teuthology.orchestra.run.smithi002.stdout: ceph telemetry on 2023-11-16T16:40:52.356 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:40:52.357 INFO:teuthology.orchestra.run.smithi002.stdout:For more information see: 2023-11-16T16:40:52.357 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:40:52.357 INFO:teuthology.orchestra.run.smithi002.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-11-16T16:40:52.357 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:40:52.357 INFO:teuthology.orchestra.run.smithi002.stdout:Bootstrap complete. 2023-11-16T16:40:52.412 INFO:tasks.cephadm:Fetching config... 2023-11-16T16:40:52.412 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T16:40:52.412 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-11-16T16:40:52.456 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-11-16T16:40:52.456 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T16:40:52.456 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-11-16T16:40:52.543 INFO:tasks.cephadm:Fetching mon keyring... 2023-11-16T16:40:52.543 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T16:40:52.543 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/keyring of=/dev/stdout 2023-11-16T16:40:52.657 INFO:tasks.cephadm:Fetching pub ssh key... 2023-11-16T16:40:52.657 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T16:40:52.657 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-11-16T16:40:52.740 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-11-16T16:40:52.740 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEM+zcML4+7BH9HF+gbIskc3Kc2Zw5Ad20Rg2zIyDbyMH8kdv842x+q+kSZ3cUA9IPdWUACpmLw222KVqJOA0oAkYuGonTieJ9Sdbrji5YlZ3FmJIA/yCQhHoBb1LGKQ2Nj19fls6loCtpzgJ21ZzSGShN1uBqTjrIE4pP4eHqWo9nzClTZ/RAY0TG778sExqR/P6V4opBXpxZeBB1oXB4bgGCpZmSDgzXqryAWkyUpMXBw/osnZTur8EfLUOU9erIU860MxqZ/scLw3MTMx7nuhQvxGdTBoEGoEqF771zmzFVSrQdK6PbY+aw+6pkqutHtA1DkrZG7+4Bn9AgPkWGa5MmGO+2v6DlYJuSzTyehZgJsbaWsV9KW2ArcfMgzuQ0N9y7muAaToA/DBdtn8msHA8F1de3iHRFOG9eHttq+KI9Pmd0thHj5eaL6MqClzddtkX5te+4nekAlT59iPeMGCTOEx2goIIsQ+bUn7pCSAu3cJFLRY9nIV8vocR9lX8= ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-16T16:40:52.894 INFO:teuthology.orchestra.run.smithi002.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEM+zcML4+7BH9HF+gbIskc3Kc2Zw5Ad20Rg2zIyDbyMH8kdv842x+q+kSZ3cUA9IPdWUACpmLw222KVqJOA0oAkYuGonTieJ9Sdbrji5YlZ3FmJIA/yCQhHoBb1LGKQ2Nj19fls6loCtpzgJ21ZzSGShN1uBqTjrIE4pP4eHqWo9nzClTZ/RAY0TG778sExqR/P6V4opBXpxZeBB1oXB4bgGCpZmSDgzXqryAWkyUpMXBw/osnZTur8EfLUOU9erIU860MxqZ/scLw3MTMx7nuhQvxGdTBoEGoEqF771zmzFVSrQdK6PbY+aw+6pkqutHtA1DkrZG7+4Bn9AgPkWGa5MmGO+2v6DlYJuSzTyehZgJsbaWsV9KW2ArcfMgzuQ0N9y7muAaToA/DBdtn8msHA8F1de3iHRFOG9eHttq+KI9Pmd0thHj5eaL6MqClzddtkX5te+4nekAlT59iPeMGCTOEx2goIIsQ+bUn7pCSAu3cJFLRY9nIV8vocR9lX8= ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:40:52.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:52 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1846136992' entity='client.admin' 2023-11-16T16:40:52.924 DEBUG:teuthology.orchestra.run.smithi141:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEM+zcML4+7BH9HF+gbIskc3Kc2Zw5Ad20Rg2zIyDbyMH8kdv842x+q+kSZ3cUA9IPdWUACpmLw222KVqJOA0oAkYuGonTieJ9Sdbrji5YlZ3FmJIA/yCQhHoBb1LGKQ2Nj19fls6loCtpzgJ21ZzSGShN1uBqTjrIE4pP4eHqWo9nzClTZ/RAY0TG778sExqR/P6V4opBXpxZeBB1oXB4bgGCpZmSDgzXqryAWkyUpMXBw/osnZTur8EfLUOU9erIU860MxqZ/scLw3MTMx7nuhQvxGdTBoEGoEqF771zmzFVSrQdK6PbY+aw+6pkqutHtA1DkrZG7+4Bn9AgPkWGa5MmGO+2v6DlYJuSzTyehZgJsbaWsV9KW2ArcfMgzuQ0N9y7muAaToA/DBdtn8msHA8F1de3iHRFOG9eHttq+KI9Pmd0thHj5eaL6MqClzddtkX5te+4nekAlT59iPeMGCTOEx2goIIsQ+bUn7pCSAu3cJFLRY9nIV8vocR9lX8= ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-16T16:40:52.971 INFO:teuthology.orchestra.run.smithi141.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEM+zcML4+7BH9HF+gbIskc3Kc2Zw5Ad20Rg2zIyDbyMH8kdv842x+q+kSZ3cUA9IPdWUACpmLw222KVqJOA0oAkYuGonTieJ9Sdbrji5YlZ3FmJIA/yCQhHoBb1LGKQ2Nj19fls6loCtpzgJ21ZzSGShN1uBqTjrIE4pP4eHqWo9nzClTZ/RAY0TG778sExqR/P6V4opBXpxZeBB1oXB4bgGCpZmSDgzXqryAWkyUpMXBw/osnZTur8EfLUOU9erIU860MxqZ/scLw3MTMx7nuhQvxGdTBoEGoEqF771zmzFVSrQdK6PbY+aw+6pkqutHtA1DkrZG7+4Bn9AgPkWGa5MmGO+2v6DlYJuSzTyehZgJsbaWsV9KW2ArcfMgzuQ0N9y7muAaToA/DBdtn8msHA8F1de3iHRFOG9eHttq+KI9Pmd0thHj5eaL6MqClzddtkX5te+4nekAlT59iPeMGCTOEx2goIIsQ+bUn7pCSAu3cJFLRY9nIV8vocR9lX8= ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:40:52.983 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEM+zcML4+7BH9HF+gbIskc3Kc2Zw5Ad20Rg2zIyDbyMH8kdv842x+q+kSZ3cUA9IPdWUACpmLw222KVqJOA0oAkYuGonTieJ9Sdbrji5YlZ3FmJIA/yCQhHoBb1LGKQ2Nj19fls6loCtpzgJ21ZzSGShN1uBqTjrIE4pP4eHqWo9nzClTZ/RAY0TG778sExqR/P6V4opBXpxZeBB1oXB4bgGCpZmSDgzXqryAWkyUpMXBw/osnZTur8EfLUOU9erIU860MxqZ/scLw3MTMx7nuhQvxGdTBoEGoEqF771zmzFVSrQdK6PbY+aw+6pkqutHtA1DkrZG7+4Bn9AgPkWGa5MmGO+2v6DlYJuSzTyehZgJsbaWsV9KW2ArcfMgzuQ0N9y7muAaToA/DBdtn8msHA8F1de3iHRFOG9eHttq+KI9Pmd0thHj5eaL6MqClzddtkX5te+4nekAlT59iPeMGCTOEx2goIIsQ+bUn7pCSAu3cJFLRY9nIV8vocR9lX8= ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-16T16:40:53.029 INFO:teuthology.orchestra.run.smithi155.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEM+zcML4+7BH9HF+gbIskc3Kc2Zw5Ad20Rg2zIyDbyMH8kdv842x+q+kSZ3cUA9IPdWUACpmLw222KVqJOA0oAkYuGonTieJ9Sdbrji5YlZ3FmJIA/yCQhHoBb1LGKQ2Nj19fls6loCtpzgJ21ZzSGShN1uBqTjrIE4pP4eHqWo9nzClTZ/RAY0TG778sExqR/P6V4opBXpxZeBB1oXB4bgGCpZmSDgzXqryAWkyUpMXBw/osnZTur8EfLUOU9erIU860MxqZ/scLw3MTMx7nuhQvxGdTBoEGoEqF771zmzFVSrQdK6PbY+aw+6pkqutHtA1DkrZG7+4Bn9AgPkWGa5MmGO+2v6DlYJuSzTyehZgJsbaWsV9KW2ArcfMgzuQ0N9y7muAaToA/DBdtn8msHA8F1de3iHRFOG9eHttq+KI9Pmd0thHj5eaL6MqClzddtkX5te+4nekAlT59iPeMGCTOEx2goIIsQ+bUn7pCSAu3cJFLRY9nIV8vocR9lX8= ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:40:53.042 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-11-16T16:40:54.056 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:40:55.318 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:55 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:40:55.318 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:55 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:40:55.318 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:55 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi002", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:40:55.318 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:55 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:40:55.318 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:55 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:55.319 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:55 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:40:55.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.601+0000 7f69e6e86700 1 -- 172.21.15.2:0/2166619056 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e00ffd00 msgr2=0x7f69e0100110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:55.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.601+0000 7f69e6e86700 1 --2- 172.21.15.2:0/2166619056 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e00ffd00 0x7f69e0100110 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009910 tx=0x7f69d4009c20 comp rx=0 tx=0).stop 2023-11-16T16:40:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.601+0000 7f69e6e86700 1 -- 172.21.15.2:0/2166619056 shutdown_connections 2023-11-16T16:40:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.601+0000 7f69e6e86700 1 --2- 172.21.15.2:0/2166619056 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e00ffd00 0x7f69e0100110 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.601+0000 7f69e6e86700 1 -- 172.21.15.2:0/2166619056 >> 172.21.15.2:0/2166619056 conn(0x7f69e00fb2b0 msgr2=0x7f69e00fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.601+0000 7f69e6e86700 1 -- 172.21.15.2:0/2166619056 shutdown_connections 2023-11-16T16:40:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.601+0000 7f69e6e86700 1 -- 172.21.15.2:0/2166619056 wait complete. 2023-11-16T16:40:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.602+0000 7f69e6e86700 1 Processor -- start 2023-11-16T16:40:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.602+0000 7f69e6e86700 1 -- start start 2023-11-16T16:40:55.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.603+0000 7f69e6e86700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e00ffd00 0x7f69e0194e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:55.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.603+0000 7f69e6e86700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e0195340 con 0x7f69e00ffd00 2023-11-16T16:40:55.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.603+0000 7f69e4c22700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e00ffd00 0x7f69e0194e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:55.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.603+0000 7f69e4c22700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e00ffd00 0x7f69e0194e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57474/0 (socket says 172.21.15.2:57474) 2023-11-16T16:40:55.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.603+0000 7f69e4c22700 1 -- 172.21.15.2:0/3470253634 learned_addr learned my addr 172.21.15.2:0/3470253634 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:55.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.603+0000 7f69e4c22700 1 -- 172.21.15.2:0/3470253634 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d40095f0 con 0x7f69e00ffd00 2023-11-16T16:40:55.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.603+0000 7f69e4c22700 1 --2- 172.21.15.2:0/3470253634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e00ffd00 0x7f69e0194e00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f69d4004f40 tx=0x7f69d40057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:55.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.604+0000 7f69ddffb700 1 -- 172.21.15.2:0/3470253634 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69d401d070 con 0x7f69e00ffd00 2023-11-16T16:40:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.604+0000 7f69ddffb700 1 -- 172.21.15.2:0/3470253634 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f69d400bc10 con 0x7f69e00ffd00 2023-11-16T16:40:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.604+0000 7f69ddffb700 1 -- 172.21.15.2:0/3470253634 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69d4003a70 con 0x7f69e00ffd00 2023-11-16T16:40:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.604+0000 7f69e6e86700 1 -- 172.21.15.2:0/3470253634 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e0105bd0 con 0x7f69e00ffd00 2023-11-16T16:40:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.604+0000 7f69e6e86700 1 -- 172.21.15.2:0/3470253634 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e0105a40 con 0x7f69e00ffd00 2023-11-16T16:40:55.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.605+0000 7f69ddffb700 1 -- 172.21.15.2:0/3470253634 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f69d4003ce0 con 0x7f69e00ffd00 2023-11-16T16:40:55.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.605+0000 7f69e6e86700 1 -- 172.21.15.2:0/3470253634 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e0105d60 con 0x7f69e00ffd00 2023-11-16T16:40:55.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.605+0000 7f69ddffb700 1 --2- 172.21.15.2:0/3470253634 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f69d00388c0 0x7f69d003ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:55.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.605+0000 7f69ddffb700 1 -- 172.21.15.2:0/3470253634 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f69d404c530 con 0x7f69e00ffd00 2023-11-16T16:40:55.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.606+0000 7f69dffff700 1 --2- 172.21.15.2:0/3470253634 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f69d00388c0 0x7f69d003ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:55.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.606+0000 7f69dffff700 1 --2- 172.21.15.2:0/3470253634 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f69d00388c0 0x7f69d003ad70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f69cc006fd0 tx=0x7f69cc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:55.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.610+0000 7f69ddffb700 1 -- 172.21.15.2:0/3470253634 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f69d402aba0 con 0x7f69e00ffd00 2023-11-16T16:40:55.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.819+0000 7f69e6e86700 1 -- 172.21.15.2:0/3470253634 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f69e0195930 con 0x7f69e00ffd00 2023-11-16T16:40:55.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.823+0000 7f69ddffb700 1 -- 172.21.15.2:0/3470253634 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f69d4026070 con 0x7f69e00ffd00 2023-11-16T16:40:55.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.825+0000 7f69e6e86700 1 -- 172.21.15.2:0/3470253634 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f69d00388c0 msgr2=0x7f69d003ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:55.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.825+0000 7f69e6e86700 1 --2- 172.21.15.2:0/3470253634 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f69d00388c0 0x7f69d003ad70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f69cc006fd0 tx=0x7f69cc006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:55.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.825+0000 7f69e6e86700 1 -- 172.21.15.2:0/3470253634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e00ffd00 msgr2=0x7f69e0194e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:55.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.825+0000 7f69e6e86700 1 --2- 172.21.15.2:0/3470253634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e00ffd00 0x7f69e0194e00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f69d4004f40 tx=0x7f69d40057c0 comp rx=0 tx=0).stop 2023-11-16T16:40:55.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.825+0000 7f69e6e86700 1 -- 172.21.15.2:0/3470253634 shutdown_connections 2023-11-16T16:40:55.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.825+0000 7f69e6e86700 1 --2- 172.21.15.2:0/3470253634 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f69d00388c0 0x7f69d003ad70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:55.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.825+0000 7f69e6e86700 1 --2- 172.21.15.2:0/3470253634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e00ffd00 0x7f69e0194e00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:55.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.825+0000 7f69e6e86700 1 -- 172.21.15.2:0/3470253634 >> 172.21.15.2:0/3470253634 conn(0x7f69e00fb2b0 msgr2=0x7f69e00fbf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:55.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.825+0000 7f69e6e86700 1 -- 172.21.15.2:0/3470253634 shutdown_connections 2023-11-16T16:40:55.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:55.826+0000 7f69e6e86700 1 -- 172.21.15.2:0/3470253634 wait complete. 2023-11-16T16:40:56.250 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-11-16T16:40:56.250 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-11-16T16:40:57.109 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:40:57.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:56 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3470253634' entity='client.admin' 2023-11-16T16:40:57.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:56 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:40:57.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:56 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:40:58.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.653+0000 7f3e78650700 1 -- 172.21.15.2:0/547313329 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e700ffd00 msgr2=0x7f3e70100110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:58.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.653+0000 7f3e78650700 1 --2- 172.21.15.2:0/547313329 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e700ffd00 0x7f3e70100110 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c009910 tx=0x7f3e6c009c20 comp rx=0 tx=0).stop 2023-11-16T16:40:58.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.653+0000 7f3e78650700 1 -- 172.21.15.2:0/547313329 shutdown_connections 2023-11-16T16:40:58.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.653+0000 7f3e78650700 1 --2- 172.21.15.2:0/547313329 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e700ffd00 0x7f3e70100110 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:58.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.653+0000 7f3e78650700 1 -- 172.21.15.2:0/547313329 >> 172.21.15.2:0/547313329 conn(0x7f3e700fb2b0 msgr2=0x7f3e700fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:58.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.653+0000 7f3e78650700 1 -- 172.21.15.2:0/547313329 shutdown_connections 2023-11-16T16:40:58.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.653+0000 7f3e78650700 1 -- 172.21.15.2:0/547313329 wait complete. 2023-11-16T16:40:58.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.654+0000 7f3e78650700 1 Processor -- start 2023-11-16T16:40:58.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.654+0000 7f3e78650700 1 -- start start 2023-11-16T16:40:58.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.655+0000 7f3e78650700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e700ffd00 0x7f3e70074890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:58.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.655+0000 7f3e78650700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e70074dd0 con 0x7f3e700ffd00 2023-11-16T16:40:58.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.655+0000 7f3e763ec700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e700ffd00 0x7f3e70074890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:58.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.655+0000 7f3e763ec700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e700ffd00 0x7f3e70074890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57490/0 (socket says 172.21.15.2:57490) 2023-11-16T16:40:58.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.655+0000 7f3e763ec700 1 -- 172.21.15.2:0/796166544 learned_addr learned my addr 172.21.15.2:0/796166544 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:40:58.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.655+0000 7f3e763ec700 1 -- 172.21.15.2:0/796166544 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6c0095f0 con 0x7f3e700ffd00 2023-11-16T16:40:58.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.655+0000 7f3e763ec700 1 --2- 172.21.15.2:0/796166544 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e700ffd00 0x7f3e70074890 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c004f40 tx=0x7f3e6c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:58.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.656+0000 7f3e677fe700 1 -- 172.21.15.2:0/796166544 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e6c01d070 con 0x7f3e700ffd00 2023-11-16T16:40:58.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.656+0000 7f3e677fe700 1 -- 172.21.15.2:0/796166544 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e6c00bc10 con 0x7f3e700ffd00 2023-11-16T16:40:58.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.656+0000 7f3e78650700 1 -- 172.21.15.2:0/796166544 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e70105bd0 con 0x7f3e700ffd00 2023-11-16T16:40:58.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.656+0000 7f3e677fe700 1 -- 172.21.15.2:0/796166544 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e6c003a70 con 0x7f3e700ffd00 2023-11-16T16:40:58.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.656+0000 7f3e78650700 1 -- 172.21.15.2:0/796166544 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e70105a40 con 0x7f3e700ffd00 2023-11-16T16:40:58.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.657+0000 7f3e78650700 1 -- 172.21.15.2:0/796166544 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e70105d60 con 0x7f3e700ffd00 2023-11-16T16:40:58.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.657+0000 7f3e677fe700 1 -- 172.21.15.2:0/796166544 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f3e6c003ce0 con 0x7f3e700ffd00 2023-11-16T16:40:58.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.657+0000 7f3e677fe700 1 --2- 172.21.15.2:0/796166544 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3e5c0388c0 0x7f3e5c03ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:40:58.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.657+0000 7f3e677fe700 1 -- 172.21.15.2:0/796166544 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3e6c04c550 con 0x7f3e700ffd00 2023-11-16T16:40:58.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.658+0000 7f3e75beb700 1 --2- 172.21.15.2:0/796166544 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3e5c0388c0 0x7f3e5c03ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:40:58.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.658+0000 7f3e75beb700 1 --2- 172.21.15.2:0/796166544 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3e5c0388c0 0x7f3e5c03ad70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3e60006fd0 tx=0x7f3e60006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:40:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.663+0000 7f3e677fe700 1 -- 172.21.15.2:0/796166544 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3e6c02abf0 con 0x7f3e700ffd00 2023-11-16T16:40:58.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.876+0000 7f3e78650700 1 -- 172.21.15.2:0/796166544 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e70073360 con 0x7f3e5c0388c0 2023-11-16T16:40:58.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.879+0000 7f3e677fe700 1 -- 172.21.15.2:0/796166544 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3e70073360 con 0x7f3e5c0388c0 2023-11-16T16:40:58.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.882+0000 7f3e78650700 1 -- 172.21.15.2:0/796166544 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3e5c0388c0 msgr2=0x7f3e5c03ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:58.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.882+0000 7f3e78650700 1 --2- 172.21.15.2:0/796166544 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3e5c0388c0 0x7f3e5c03ad70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3e60006fd0 tx=0x7f3e60006e40 comp rx=0 tx=0).stop 2023-11-16T16:40:58.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.882+0000 7f3e78650700 1 -- 172.21.15.2:0/796166544 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e700ffd00 msgr2=0x7f3e70074890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:40:58.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.882+0000 7f3e78650700 1 --2- 172.21.15.2:0/796166544 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e700ffd00 0x7f3e70074890 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c004f40 tx=0x7f3e6c0057c0 comp rx=0 tx=0).stop 2023-11-16T16:40:58.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.882+0000 7f3e78650700 1 -- 172.21.15.2:0/796166544 shutdown_connections 2023-11-16T16:40:58.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.882+0000 7f3e78650700 1 --2- 172.21.15.2:0/796166544 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3e5c0388c0 0x7f3e5c03ad70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:58.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.882+0000 7f3e78650700 1 --2- 172.21.15.2:0/796166544 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e700ffd00 0x7f3e70074890 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:40:58.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.882+0000 7f3e78650700 1 -- 172.21.15.2:0/796166544 >> 172.21.15.2:0/796166544 conn(0x7f3e700fb2b0 msgr2=0x7f3e700fbf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:40:58.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.882+0000 7f3e78650700 1 -- 172.21.15.2:0/796166544 shutdown_connections 2023-11-16T16:40:58.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:40:58.882+0000 7f3e78650700 1 -- 172.21.15.2:0/796166544 wait complete. 2023-11-16T16:40:59.494 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi141 2023-11-16T16:40:59.494 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-11-16T16:40:59.495 DEBUG:teuthology.orchestra.run.smithi141:> dd of=/etc/ceph/ceph.conf 2023-11-16T16:40:59.515 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-11-16T16:40:59.515 DEBUG:teuthology.orchestra.run.smithi141:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:40:59.575 INFO:tasks.cephadm:Adding host smithi141 to orchestrator... 2023-11-16T16:40:59.576 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi141 2023-11-16T16:41:00.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:59 smithi002 ceph-mon[128712]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:00.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:59 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:00.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:59 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:00.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:59 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:00.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:59 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:00.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:59 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:41:00.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:40:59 smithi002 ceph-mon[128712]: Updating smithi002:/etc/ceph/ceph.conf 2023-11-16T16:41:00.468 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:41:01.042 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:00 smithi002 ceph-mon[128712]: Updating smithi002:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:01.042 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:00 smithi002 ceph-mon[128712]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:01.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.982+0000 7f53aace0700 1 -- 172.21.15.2:0/237358312 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a40fff90 msgr2=0x7f53a4100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:01.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.982+0000 7f53aace0700 1 --2- 172.21.15.2:0/237358312 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a40fff90 0x7f53a4100360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5398009910 tx=0x7f5398009c20 comp rx=0 tx=0).stop 2023-11-16T16:41:01.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.983+0000 7f53aace0700 1 -- 172.21.15.2:0/237358312 shutdown_connections 2023-11-16T16:41:01.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.983+0000 7f53aace0700 1 --2- 172.21.15.2:0/237358312 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a40fff90 0x7f53a4100360 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:01.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.983+0000 7f53aace0700 1 -- 172.21.15.2:0/237358312 >> 172.21.15.2:0/237358312 conn(0x7f53a40fb850 msgr2=0x7f53a40fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:01.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.983+0000 7f53aace0700 1 -- 172.21.15.2:0/237358312 shutdown_connections 2023-11-16T16:41:01.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.983+0000 7f53aace0700 1 -- 172.21.15.2:0/237358312 wait complete. 2023-11-16T16:41:01.986 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.984+0000 7f53aace0700 1 Processor -- start 2023-11-16T16:41:01.986 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.984+0000 7f53aace0700 1 -- start start 2023-11-16T16:41:01.986 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.984+0000 7f53aace0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a40fff90 0x7f53a41973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:01.987 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.984+0000 7f53aace0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a41978e0 con 0x7f53a40fff90 2023-11-16T16:41:01.987 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.984+0000 7f53a8a7c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a40fff90 0x7f53a41973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:01.987 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.985+0000 7f53a8a7c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a40fff90 0x7f53a41973a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57506/0 (socket says 172.21.15.2:57506) 2023-11-16T16:41:01.987 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.985+0000 7f53a8a7c700 1 -- 172.21.15.2:0/651156950 learned_addr learned my addr 172.21.15.2:0/651156950 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:41:01.988 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.985+0000 7f53a8a7c700 1 -- 172.21.15.2:0/651156950 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53980095f0 con 0x7f53a40fff90 2023-11-16T16:41:01.988 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.985+0000 7f53a8a7c700 1 --2- 172.21.15.2:0/651156950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a40fff90 0x7f53a41973a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5398006010 tx=0x7f5398004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:01.988 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.985+0000 7f53a1ffb700 1 -- 172.21.15.2:0/651156950 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f539801d070 con 0x7f53a40fff90 2023-11-16T16:41:01.988 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.985+0000 7f53a1ffb700 1 -- 172.21.15.2:0/651156950 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f539800bc10 con 0x7f53a40fff90 2023-11-16T16:41:01.988 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.985+0000 7f53aace0700 1 -- 172.21.15.2:0/651156950 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a410a170 con 0x7f53a40fff90 2023-11-16T16:41:01.988 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.985+0000 7f53a1ffb700 1 -- 172.21.15.2:0/651156950 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5398003910 con 0x7f53a40fff90 2023-11-16T16:41:01.988 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.986+0000 7f53aace0700 1 -- 172.21.15.2:0/651156950 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a4109fe0 con 0x7f53a40fff90 2023-11-16T16:41:01.989 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.986+0000 7f53aace0700 1 -- 172.21.15.2:0/651156950 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a410a300 con 0x7f53a40fff90 2023-11-16T16:41:01.989 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.987+0000 7f53a1ffb700 1 -- 172.21.15.2:0/651156950 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f539800bd80 con 0x7f53a40fff90 2023-11-16T16:41:01.989 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.987+0000 7f53a1ffb700 1 --2- 172.21.15.2:0/651156950 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f53940388c0 0x7f539403ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:01.989 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.987+0000 7f53a1ffb700 1 -- 172.21.15.2:0/651156950 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f539804c5a0 con 0x7f53a40fff90 2023-11-16T16:41:01.989 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.987+0000 7f53a3fff700 1 --2- 172.21.15.2:0/651156950 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f53940388c0 0x7f539403ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:01.990 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.988+0000 7f53a3fff700 1 --2- 172.21.15.2:0/651156950 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f53940388c0 0x7f539403ad70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5390006fd0 tx=0x7f5390006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:01.993 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:01.992+0000 7f53a1ffb700 1 -- 172.21.15.2:0/651156950 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5398029360 con 0x7f53a40fff90 2023-11-16T16:41:02.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:01 smithi002 ceph-mon[128712]: Updating smithi002:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:41:02.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:01 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:02.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:01 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:02.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:01 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:02.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:02.201+0000 7f53aace0700 1 -- 172.21.15.2:0/651156950 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi141", "target": ["mon-mgr", ""]}) v1 -- 0x7f53a4197ee0 con 0x7f53940388c0 2023-11-16T16:41:03.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:03 smithi002 ceph-mon[128712]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi141", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:03.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:03 smithi002 ceph-mon[128712]: Deploying cephadm binary to smithi141 2023-11-16T16:41:04.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:04.344+0000 7f53a1ffb700 1 -- 172.21.15.2:0/651156950 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f53a4197ee0 con 0x7f53940388c0 2023-11-16T16:41:04.347 INFO:teuthology.orchestra.run.smithi002.stdout:Added host 'smithi141' with addr '172.21.15.141' 2023-11-16T16:41:04.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:04.347+0000 7f53aace0700 1 -- 172.21.15.2:0/651156950 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f53940388c0 msgr2=0x7f539403ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:04.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:04.347+0000 7f53aace0700 1 --2- 172.21.15.2:0/651156950 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f53940388c0 0x7f539403ad70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5390006fd0 tx=0x7f5390006e40 comp rx=0 tx=0).stop 2023-11-16T16:41:04.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:04.347+0000 7f53aace0700 1 -- 172.21.15.2:0/651156950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a40fff90 msgr2=0x7f53a41973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:04.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:04.347+0000 7f53aace0700 1 --2- 172.21.15.2:0/651156950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a40fff90 0x7f53a41973a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5398006010 tx=0x7f5398004ff0 comp rx=0 tx=0).stop 2023-11-16T16:41:04.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:04.347+0000 7f53aace0700 1 -- 172.21.15.2:0/651156950 shutdown_connections 2023-11-16T16:41:04.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:04.347+0000 7f53aace0700 1 --2- 172.21.15.2:0/651156950 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f53940388c0 0x7f539403ad70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:04.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:04.347+0000 7f53aace0700 1 --2- 172.21.15.2:0/651156950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a40fff90 0x7f53a41973a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:04.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:04.347+0000 7f53aace0700 1 -- 172.21.15.2:0/651156950 >> 172.21.15.2:0/651156950 conn(0x7f53a40fb850 msgr2=0x7f53a41038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:04.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:04.347+0000 7f53aace0700 1 -- 172.21.15.2:0/651156950 shutdown_connections 2023-11-16T16:41:04.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:04.347+0000 7f53aace0700 1 -- 172.21.15.2:0/651156950 wait complete. 2023-11-16T16:41:04.860 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-16T16:41:05.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:04 smithi002 ceph-mon[128712]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:05.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:04 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:05.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:04 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:05.720 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:41:06.087 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:05 smithi002 ceph-mon[128712]: Added host smithi141 2023-11-16T16:41:06.087 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:05 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:06.087 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:05 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:07.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:06 smithi002 ceph-mon[128712]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:07.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.336+0000 7f37d1061700 1 -- 172.21.15.2:0/2025263134 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37cc0fff90 msgr2=0x7f37cc100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:07.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.336+0000 7f37d1061700 1 --2- 172.21.15.2:0/2025263134 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37cc0fff90 0x7f37cc100360 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009910 tx=0x7f37bc009c20 comp rx=0 tx=0).stop 2023-11-16T16:41:07.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.336+0000 7f37d1061700 1 -- 172.21.15.2:0/2025263134 shutdown_connections 2023-11-16T16:41:07.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.336+0000 7f37d1061700 1 --2- 172.21.15.2:0/2025263134 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37cc0fff90 0x7f37cc100360 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:07.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.336+0000 7f37d1061700 1 -- 172.21.15.2:0/2025263134 >> 172.21.15.2:0/2025263134 conn(0x7f37cc0fb850 msgr2=0x7f37cc0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:07.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.336+0000 7f37d1061700 1 -- 172.21.15.2:0/2025263134 shutdown_connections 2023-11-16T16:41:07.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.336+0000 7f37d1061700 1 -- 172.21.15.2:0/2025263134 wait complete. 2023-11-16T16:41:07.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.337+0000 7f37d1061700 1 Processor -- start 2023-11-16T16:41:07.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.337+0000 7f37d1061700 1 -- start start 2023-11-16T16:41:07.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.338+0000 7f37d1061700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37cc0fff90 0x7f37cc072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:07.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.338+0000 7f37d1061700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc074d70 con 0x7f37cc0fff90 2023-11-16T16:41:07.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.338+0000 7f37cad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37cc0fff90 0x7f37cc072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:07.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.338+0000 7f37cad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37cc0fff90 0x7f37cc072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48636/0 (socket says 172.21.15.2:48636) 2023-11-16T16:41:07.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.338+0000 7f37cad9d700 1 -- 172.21.15.2:0/2828919386 learned_addr learned my addr 172.21.15.2:0/2828919386 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:41:07.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.338+0000 7f37cad9d700 1 -- 172.21.15.2:0/2828919386 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37bc0095f0 con 0x7f37cc0fff90 2023-11-16T16:41:07.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.339+0000 7f37cad9d700 1 --2- 172.21.15.2:0/2828919386 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37cc0fff90 0x7f37cc072df0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f37bc006010 tx=0x7f37bc004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:07.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.339+0000 7f37b3fff700 1 -- 172.21.15.2:0/2828919386 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f37bc01d070 con 0x7f37cc0fff90 2023-11-16T16:41:07.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.339+0000 7f37b3fff700 1 -- 172.21.15.2:0/2828919386 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37bc00bc10 con 0x7f37cc0fff90 2023-11-16T16:41:07.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.339+0000 7f37b3fff700 1 -- 172.21.15.2:0/2828919386 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f37bc003910 con 0x7f37cc0fff90 2023-11-16T16:41:07.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.339+0000 7f37d1061700 1 -- 172.21.15.2:0/2828919386 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37cc10a170 con 0x7f37cc0fff90 2023-11-16T16:41:07.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.339+0000 7f37d1061700 1 -- 172.21.15.2:0/2828919386 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37cc109fe0 con 0x7f37cc0fff90 2023-11-16T16:41:07.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.340+0000 7f37d1061700 1 -- 172.21.15.2:0/2828919386 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37cc10a300 con 0x7f37cc0fff90 2023-11-16T16:41:07.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.340+0000 7f37b3fff700 1 -- 172.21.15.2:0/2828919386 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f37bc00bd80 con 0x7f37cc0fff90 2023-11-16T16:41:07.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.341+0000 7f37b3fff700 1 --2- 172.21.15.2:0/2828919386 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f37b40388c0 0x7f37b403ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:07.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.341+0000 7f37b3fff700 1 -- 172.21.15.2:0/2828919386 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f37bc04c5a0 con 0x7f37cc0fff90 2023-11-16T16:41:07.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.341+0000 7f37ca59c700 1 --2- 172.21.15.2:0/2828919386 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f37b40388c0 0x7f37b403ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:07.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.342+0000 7f37ca59c700 1 --2- 172.21.15.2:0/2828919386 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f37b40388c0 0x7f37b403ad70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f37c0006fd0 tx=0x7f37c0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:07.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.346+0000 7f37b3fff700 1 -- 172.21.15.2:0/2828919386 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f37bc029360 con 0x7f37cc0fff90 2023-11-16T16:41:07.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.560+0000 7f37d1061700 1 -- 172.21.15.2:0/2828919386 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f37cc074320 con 0x7f37b40388c0 2023-11-16T16:41:07.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.562+0000 7f37b3fff700 1 -- 172.21.15.2:0/2828919386 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f37cc074320 con 0x7f37b40388c0 2023-11-16T16:41:07.564 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:41:07.564 INFO:teuthology.orchestra.run.smithi002.stdout:[{"addr": "172.21.15.2", "hostname": "smithi002", "labels": [], "status": ""}, {"addr": "172.21.15.141", "hostname": "smithi141", "labels": [], "status": ""}] 2023-11-16T16:41:07.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.564+0000 7f37d1061700 1 -- 172.21.15.2:0/2828919386 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f37b40388c0 msgr2=0x7f37b403ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:07.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.564+0000 7f37d1061700 1 --2- 172.21.15.2:0/2828919386 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f37b40388c0 0x7f37b403ad70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f37c0006fd0 tx=0x7f37c0006e40 comp rx=0 tx=0).stop 2023-11-16T16:41:07.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.564+0000 7f37d1061700 1 -- 172.21.15.2:0/2828919386 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37cc0fff90 msgr2=0x7f37cc072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:07.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.564+0000 7f37d1061700 1 --2- 172.21.15.2:0/2828919386 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37cc0fff90 0x7f37cc072df0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f37bc006010 tx=0x7f37bc004ff0 comp rx=0 tx=0).stop 2023-11-16T16:41:07.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.564+0000 7f37d1061700 1 -- 172.21.15.2:0/2828919386 shutdown_connections 2023-11-16T16:41:07.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.564+0000 7f37d1061700 1 --2- 172.21.15.2:0/2828919386 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f37b40388c0 0x7f37b403ad70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:07.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.564+0000 7f37d1061700 1 --2- 172.21.15.2:0/2828919386 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f37cc0fff90 0x7f37cc072df0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:07.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.564+0000 7f37d1061700 1 -- 172.21.15.2:0/2828919386 >> 172.21.15.2:0/2828919386 conn(0x7f37cc0fb850 msgr2=0x7f37cc104170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:07.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.564+0000 7f37d1061700 1 -- 172.21.15.2:0/2828919386 shutdown_connections 2023-11-16T16:41:07.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:07.564+0000 7f37d1061700 1 -- 172.21.15.2:0/2828919386 wait complete. 2023-11-16T16:41:08.103 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi155 2023-11-16T16:41:08.103 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-11-16T16:41:08.103 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/etc/ceph/ceph.conf 2023-11-16T16:41:08.124 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-11-16T16:41:08.124 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:08.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:07 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:08.184 INFO:tasks.cephadm:Adding host smithi155 to orchestrator... 2023-11-16T16:41:08.184 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi155 2023-11-16T16:41:09.042 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:41:09.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:08 smithi002 ceph-mon[128712]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:41:09.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:08 smithi002 ceph-mon[128712]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:10.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.580+0000 7f05f76e2700 1 -- 172.21.15.2:0/4036462084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05f00fff90 msgr2=0x7f05f0100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:10.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.580+0000 7f05f76e2700 1 --2- 172.21.15.2:0/4036462084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05f00fff90 0x7f05f0100360 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009910 tx=0x7f05ec009c20 comp rx=0 tx=0).stop 2023-11-16T16:41:10.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.580+0000 7f05f76e2700 1 -- 172.21.15.2:0/4036462084 shutdown_connections 2023-11-16T16:41:10.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.580+0000 7f05f76e2700 1 --2- 172.21.15.2:0/4036462084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05f00fff90 0x7f05f0100360 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:10.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.580+0000 7f05f76e2700 1 -- 172.21.15.2:0/4036462084 >> 172.21.15.2:0/4036462084 conn(0x7f05f00fb850 msgr2=0x7f05f00fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:10.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.580+0000 7f05f76e2700 1 -- 172.21.15.2:0/4036462084 shutdown_connections 2023-11-16T16:41:10.584 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.580+0000 7f05f76e2700 1 -- 172.21.15.2:0/4036462084 wait complete. 2023-11-16T16:41:10.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.581+0000 7f05f76e2700 1 Processor -- start 2023-11-16T16:41:10.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.582+0000 7f05f76e2700 1 -- start start 2023-11-16T16:41:10.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.582+0000 7f05f76e2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05f00fff90 0x7f05f01951c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:10.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.582+0000 7f05f76e2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f0195700 con 0x7f05f00fff90 2023-11-16T16:41:10.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.582+0000 7f05f547e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05f00fff90 0x7f05f01951c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:10.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.582+0000 7f05f547e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05f00fff90 0x7f05f01951c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48652/0 (socket says 172.21.15.2:48652) 2023-11-16T16:41:10.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.582+0000 7f05f547e700 1 -- 172.21.15.2:0/900160986 learned_addr learned my addr 172.21.15.2:0/900160986 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:41:10.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.583+0000 7f05f547e700 1 -- 172.21.15.2:0/900160986 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ec0095f0 con 0x7f05f00fff90 2023-11-16T16:41:10.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.583+0000 7f05f547e700 1 --2- 172.21.15.2:0/900160986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05f00fff90 0x7f05f01951c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f05ec006010 tx=0x7f05ec004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:10.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.583+0000 7f05e67fc700 1 -- 172.21.15.2:0/900160986 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f05ec01d070 con 0x7f05f00fff90 2023-11-16T16:41:10.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.583+0000 7f05e67fc700 1 -- 172.21.15.2:0/900160986 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05ec00bc10 con 0x7f05f00fff90 2023-11-16T16:41:10.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.583+0000 7f05f76e2700 1 -- 172.21.15.2:0/900160986 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f010a170 con 0x7f05f00fff90 2023-11-16T16:41:10.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.583+0000 7f05e67fc700 1 -- 172.21.15.2:0/900160986 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f05ec003910 con 0x7f05f00fff90 2023-11-16T16:41:10.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.583+0000 7f05f76e2700 1 -- 172.21.15.2:0/900160986 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f0109fe0 con 0x7f05f00fff90 2023-11-16T16:41:10.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.584+0000 7f05e67fc700 1 -- 172.21.15.2:0/900160986 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f05ec00bd80 con 0x7f05f00fff90 2023-11-16T16:41:10.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.584+0000 7f05f76e2700 1 -- 172.21.15.2:0/900160986 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f010a300 con 0x7f05f00fff90 2023-11-16T16:41:10.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.585+0000 7f05e67fc700 1 --2- 172.21.15.2:0/900160986 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f05dc0388c0 0x7f05dc03ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:10.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.585+0000 7f05e67fc700 1 -- 172.21.15.2:0/900160986 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f05ec04c5a0 con 0x7f05f00fff90 2023-11-16T16:41:10.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.585+0000 7f05f4c7d700 1 --2- 172.21.15.2:0/900160986 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f05dc0388c0 0x7f05dc03ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:10.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.586+0000 7f05f4c7d700 1 --2- 172.21.15.2:0/900160986 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f05dc0388c0 0x7f05dc03ad70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f05e0006fd0 tx=0x7f05e0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:10.592 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.590+0000 7f05e67fc700 1 -- 172.21.15.2:0/900160986 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f05ec029360 con 0x7f05f00fff90 2023-11-16T16:41:10.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:10.799+0000 7f05f76e2700 1 -- 172.21.15.2:0/900160986 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}) v1 -- 0x7f05f0195d00 con 0x7f05dc0388c0 2023-11-16T16:41:10.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:10.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:10.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:10.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:10.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi141", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:41:10.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:10.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:10.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:10 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:41:10.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:10 smithi002 ceph-mon[128712]: Updating smithi141:/etc/ceph/ceph.conf 2023-11-16T16:41:10.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:10 smithi002 ceph-mon[128712]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:10.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:10 smithi002 ceph-mon[128712]: Updating smithi141:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:10.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:10.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:10.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:11.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:11 smithi002 ceph-mon[128712]: Updating smithi141:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:11.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:11 smithi002 ceph-mon[128712]: Updating smithi141:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:41:11.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:11 smithi002 ceph-mon[128712]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:12.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:12 smithi002 ceph-mon[128712]: Deploying cephadm binary to smithi155 2023-11-16T16:41:12.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:12 smithi002 ceph-mon[128712]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:12.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:12.983+0000 7f05e67fc700 1 -- 172.21.15.2:0/900160986 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f05f0195d00 con 0x7f05dc0388c0 2023-11-16T16:41:12.986 INFO:teuthology.orchestra.run.smithi002.stdout:Added host 'smithi155' with addr '172.21.15.155' 2023-11-16T16:41:12.988 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:12.986+0000 7f05f76e2700 1 -- 172.21.15.2:0/900160986 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f05dc0388c0 msgr2=0x7f05dc03ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:12.988 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:12.986+0000 7f05f76e2700 1 --2- 172.21.15.2:0/900160986 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f05dc0388c0 0x7f05dc03ad70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f05e0006fd0 tx=0x7f05e0006e40 comp rx=0 tx=0).stop 2023-11-16T16:41:12.988 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:12.986+0000 7f05f76e2700 1 -- 172.21.15.2:0/900160986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05f00fff90 msgr2=0x7f05f01951c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:12.988 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:12.986+0000 7f05f76e2700 1 --2- 172.21.15.2:0/900160986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05f00fff90 0x7f05f01951c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f05ec006010 tx=0x7f05ec004ff0 comp rx=0 tx=0).stop 2023-11-16T16:41:12.988 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:12.986+0000 7f05f76e2700 1 -- 172.21.15.2:0/900160986 shutdown_connections 2023-11-16T16:41:12.988 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:12.986+0000 7f05f76e2700 1 --2- 172.21.15.2:0/900160986 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f05dc0388c0 0x7f05dc03ad70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:12.988 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:12.986+0000 7f05f76e2700 1 --2- 172.21.15.2:0/900160986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05f00fff90 0x7f05f01951c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:12.989 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:12.986+0000 7f05f76e2700 1 -- 172.21.15.2:0/900160986 >> 172.21.15.2:0/900160986 conn(0x7f05f00fb850 msgr2=0x7f05f0104170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:12.989 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:12.986+0000 7f05f76e2700 1 -- 172.21.15.2:0/900160986 shutdown_connections 2023-11-16T16:41:12.989 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:12.986+0000 7f05f76e2700 1 -- 172.21.15.2:0/900160986 wait complete. 2023-11-16T16:41:13.445 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-16T16:41:14.308 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:41:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:13 smithi002 ceph-mon[128712]: Added host smithi155 2023-11-16T16:41:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:15.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:14 smithi002 ceph-mon[128712]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:15.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:14 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:15.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.837+0000 7f4f08f76700 1 -- 172.21.15.2:0/3698859915 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f040fff90 msgr2=0x7f4f04100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:15.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.837+0000 7f4f08f76700 1 --2- 172.21.15.2:0/3698859915 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f040fff90 0x7f4f04100360 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4009910 tx=0x7f4ef4009c20 comp rx=0 tx=0).stop 2023-11-16T16:41:15.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.837+0000 7f4f08f76700 1 -- 172.21.15.2:0/3698859915 shutdown_connections 2023-11-16T16:41:15.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.837+0000 7f4f08f76700 1 --2- 172.21.15.2:0/3698859915 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f040fff90 0x7f4f04100360 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:15.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.837+0000 7f4f08f76700 1 -- 172.21.15.2:0/3698859915 >> 172.21.15.2:0/3698859915 conn(0x7f4f040fb850 msgr2=0x7f4f040fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:15.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.837+0000 7f4f08f76700 1 -- 172.21.15.2:0/3698859915 shutdown_connections 2023-11-16T16:41:15.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.837+0000 7f4f08f76700 1 -- 172.21.15.2:0/3698859915 wait complete. 2023-11-16T16:41:15.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.838+0000 7f4f08f76700 1 Processor -- start 2023-11-16T16:41:15.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.838+0000 7f4f08f76700 1 -- start start 2023-11-16T16:41:15.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.838+0000 7f4f08f76700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f040fff90 0x7f4f04072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:15.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.839+0000 7f4f08f76700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f04074d70 con 0x7f4f040fff90 2023-11-16T16:41:15.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.839+0000 7f4f0259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f040fff90 0x7f4f04072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:15.843 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.839+0000 7f4f0259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f040fff90 0x7f4f04072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47346/0 (socket says 172.21.15.2:47346) 2023-11-16T16:41:15.843 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.839+0000 7f4f0259c700 1 -- 172.21.15.2:0/768996964 learned_addr learned my addr 172.21.15.2:0/768996964 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:41:15.843 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.839+0000 7f4f0259c700 1 -- 172.21.15.2:0/768996964 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef40095f0 con 0x7f4f040fff90 2023-11-16T16:41:15.843 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.839+0000 7f4f0259c700 1 --2- 172.21.15.2:0/768996964 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f040fff90 0x7f4f04072df0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4006010 tx=0x7f4ef4004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:15.843 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.840+0000 7f4ef37fe700 1 -- 172.21.15.2:0/768996964 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ef401d070 con 0x7f4f040fff90 2023-11-16T16:41:15.843 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.840+0000 7f4ef37fe700 1 -- 172.21.15.2:0/768996964 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ef400bc10 con 0x7f4f040fff90 2023-11-16T16:41:15.843 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.840+0000 7f4ef37fe700 1 -- 172.21.15.2:0/768996964 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ef4003910 con 0x7f4f040fff90 2023-11-16T16:41:15.844 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.840+0000 7f4f08f76700 1 -- 172.21.15.2:0/768996964 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f0410a170 con 0x7f4f040fff90 2023-11-16T16:41:15.844 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.840+0000 7f4f08f76700 1 -- 172.21.15.2:0/768996964 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f04109fe0 con 0x7f4f040fff90 2023-11-16T16:41:15.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.841+0000 7f4f08f76700 1 -- 172.21.15.2:0/768996964 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0410a300 con 0x7f4f040fff90 2023-11-16T16:41:15.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.841+0000 7f4ef37fe700 1 -- 172.21.15.2:0/768996964 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f4ef400bd80 con 0x7f4f040fff90 2023-11-16T16:41:15.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.841+0000 7f4ef37fe700 1 --2- 172.21.15.2:0/768996964 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f4eec0388c0 0x7f4eec03ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:15.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.841+0000 7f4ef37fe700 1 -- 172.21.15.2:0/768996964 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4ef404c5a0 con 0x7f4f040fff90 2023-11-16T16:41:15.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.842+0000 7f4f01d9b700 1 --2- 172.21.15.2:0/768996964 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f4eec0388c0 0x7f4eec03ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:15.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.842+0000 7f4f01d9b700 1 --2- 172.21.15.2:0/768996964 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f4eec0388c0 0x7f4eec03ad70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8006fd0 tx=0x7f4ef8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:15.848 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:15.846+0000 7f4ef37fe700 1 -- 172.21.15.2:0/768996964 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4ef4029360 con 0x7f4f040fff90 2023-11-16T16:41:16.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:16.059+0000 7f4f08f76700 1 -- 172.21.15.2:0/768996964 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4f04074320 con 0x7f4eec0388c0 2023-11-16T16:41:16.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:16.060+0000 7f4ef37fe700 1 -- 172.21.15.2:0/768996964 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+239 (secure 0 0 0) 0x7f4f04074320 con 0x7f4eec0388c0 2023-11-16T16:41:16.062 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:41:16.063 INFO:teuthology.orchestra.run.smithi002.stdout:[{"addr": "172.21.15.2", "hostname": "smithi002", "labels": [], "status": ""}, {"addr": "172.21.15.141", "hostname": "smithi141", "labels": [], "status": ""}, {"addr": "172.21.15.155", "hostname": "smithi155", "labels": [], "status": ""}] 2023-11-16T16:41:16.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:16.063+0000 7f4f08f76700 1 -- 172.21.15.2:0/768996964 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f4eec0388c0 msgr2=0x7f4eec03ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:16.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:16.063+0000 7f4f08f76700 1 --2- 172.21.15.2:0/768996964 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f4eec0388c0 0x7f4eec03ad70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8006fd0 tx=0x7f4ef8006e40 comp rx=0 tx=0).stop 2023-11-16T16:41:16.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:16.063+0000 7f4f08f76700 1 -- 172.21.15.2:0/768996964 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f040fff90 msgr2=0x7f4f04072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:16.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:16.063+0000 7f4f08f76700 1 --2- 172.21.15.2:0/768996964 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f040fff90 0x7f4f04072df0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4006010 tx=0x7f4ef4004ff0 comp rx=0 tx=0).stop 2023-11-16T16:41:16.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:16.063+0000 7f4f08f76700 1 -- 172.21.15.2:0/768996964 shutdown_connections 2023-11-16T16:41:16.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:16.063+0000 7f4f08f76700 1 --2- 172.21.15.2:0/768996964 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f4eec0388c0 0x7f4eec03ad70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:16.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:16.063+0000 7f4f08f76700 1 --2- 172.21.15.2:0/768996964 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f040fff90 0x7f4f04072df0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:16.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:16.063+0000 7f4f08f76700 1 -- 172.21.15.2:0/768996964 >> 172.21.15.2:0/768996964 conn(0x7f4f040fb850 msgr2=0x7f4f04104170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:16.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:16.063+0000 7f4f08f76700 1 -- 172.21.15.2:0/768996964 shutdown_connections 2023-11-16T16:41:16.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:16.063+0000 7f4f08f76700 1 -- 172.21.15.2:0/768996964 wait complete. 2023-11-16T16:41:16.492 INFO:tasks.cephadm:Setting crush tunables to default 2023-11-16T16:41:16.492 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph osd crush tunables default 2023-11-16T16:41:16.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:16 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:16.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:16 smithi002 ceph-mon[128712]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:17.352 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:41:17.755 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:17 smithi002 ceph-mon[128712]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:41:18.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.893+0000 7f64a4433700 1 -- 172.21.15.2:0/2319055984 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f649c0fff90 msgr2=0x7f649c100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:18.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.893+0000 7f64a4433700 1 --2- 172.21.15.2:0/2319055984 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f649c0fff90 0x7f649c100360 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6498009910 tx=0x7f6498009c20 comp rx=0 tx=0).stop 2023-11-16T16:41:18.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.893+0000 7f64a4433700 1 -- 172.21.15.2:0/2319055984 shutdown_connections 2023-11-16T16:41:18.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.893+0000 7f64a4433700 1 --2- 172.21.15.2:0/2319055984 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f649c0fff90 0x7f649c100360 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:18.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.893+0000 7f64a4433700 1 -- 172.21.15.2:0/2319055984 >> 172.21.15.2:0/2319055984 conn(0x7f649c0fb850 msgr2=0x7f649c0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:18.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.893+0000 7f64a4433700 1 -- 172.21.15.2:0/2319055984 shutdown_connections 2023-11-16T16:41:18.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.893+0000 7f64a4433700 1 -- 172.21.15.2:0/2319055984 wait complete. 2023-11-16T16:41:18.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.894+0000 7f64a4433700 1 Processor -- start 2023-11-16T16:41:18.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.894+0000 7f64a4433700 1 -- start start 2023-11-16T16:41:18.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.895+0000 7f64a4433700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f649c0fff90 0x7f649c1973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:18.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.895+0000 7f64a4433700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649c1978e0 con 0x7f649c0fff90 2023-11-16T16:41:18.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.895+0000 7f64a21cf700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f649c0fff90 0x7f649c1973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:18.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.895+0000 7f64a21cf700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f649c0fff90 0x7f649c1973a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47378/0 (socket says 172.21.15.2:47378) 2023-11-16T16:41:18.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.895+0000 7f64a21cf700 1 -- 172.21.15.2:0/1384182039 learned_addr learned my addr 172.21.15.2:0/1384182039 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:41:18.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.895+0000 7f64a21cf700 1 -- 172.21.15.2:0/1384182039 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64980095f0 con 0x7f649c0fff90 2023-11-16T16:41:18.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.896+0000 7f64a21cf700 1 --2- 172.21.15.2:0/1384182039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f649c0fff90 0x7f649c1973a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6498006010 tx=0x7f6498004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:18.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.896+0000 7f64937fe700 1 -- 172.21.15.2:0/1384182039 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f649801d070 con 0x7f649c0fff90 2023-11-16T16:41:18.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.896+0000 7f64937fe700 1 -- 172.21.15.2:0/1384182039 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f649800bc10 con 0x7f649c0fff90 2023-11-16T16:41:18.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.896+0000 7f64a4433700 1 -- 172.21.15.2:0/1384182039 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f649c10a170 con 0x7f649c0fff90 2023-11-16T16:41:18.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.896+0000 7f64937fe700 1 -- 172.21.15.2:0/1384182039 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6498003910 con 0x7f649c0fff90 2023-11-16T16:41:18.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.896+0000 7f64a4433700 1 -- 172.21.15.2:0/1384182039 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f649c109fe0 con 0x7f649c0fff90 2023-11-16T16:41:18.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.897+0000 7f64937fe700 1 -- 172.21.15.2:0/1384182039 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f649800bd80 con 0x7f649c0fff90 2023-11-16T16:41:18.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.897+0000 7f64a4433700 1 -- 172.21.15.2:0/1384182039 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f649c10a300 con 0x7f649c0fff90 2023-11-16T16:41:18.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.898+0000 7f64937fe700 1 --2- 172.21.15.2:0/1384182039 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6488038870 0x7f648803ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:18.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.898+0000 7f64937fe700 1 -- 172.21.15.2:0/1384182039 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f649804c600 con 0x7f649c0fff90 2023-11-16T16:41:18.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.898+0000 7f64a19ce700 1 --2- 172.21.15.2:0/1384182039 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6488038870 0x7f648803ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:18.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.898+0000 7f64a19ce700 1 --2- 172.21.15.2:0/1384182039 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6488038870 0x7f648803ad20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f648c006fd0 tx=0x7f648c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:18.904 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:18.903+0000 7f64937fe700 1 -- 172.21.15.2:0/1384182039 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6498029360 con 0x7f649c0fff90 2023-11-16T16:41:18.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:18 smithi002 ceph-mon[128712]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:18.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:18 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:18.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:18 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:18.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:18 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:18.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:18 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:18.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:18 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi155", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:41:18.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:18 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:18.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:18 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:19.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:19.120+0000 7f64a4433700 1 -- 172.21.15.2:0/1384182039 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f649c062120 con 0x7f649c0fff90 2023-11-16T16:41:19.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:19 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:41:19.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:19 smithi002 ceph-mon[128712]: Updating smithi155:/etc/ceph/ceph.conf 2023-11-16T16:41:19.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:19 smithi002 ceph-mon[128712]: Updating smithi155:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:19.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:19 smithi002 ceph-mon[128712]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:19.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:19 smithi002 ceph-mon[128712]: Updating smithi155:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:41:19.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:19 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:19.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:19 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:19.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:19 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:19.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:19 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1384182039' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-16T16:41:20.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:20.055+0000 7f64937fe700 1 -- 172.21.15.2:0/1384182039 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6498026020 con 0x7f649c0fff90 2023-11-16T16:41:20.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:20.059+0000 7f64a4433700 1 -- 172.21.15.2:0/1384182039 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6488038870 msgr2=0x7f648803ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:20.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:20.059+0000 7f64a4433700 1 --2- 172.21.15.2:0/1384182039 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6488038870 0x7f648803ad20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f648c006fd0 tx=0x7f648c006e40 comp rx=0 tx=0).stop 2023-11-16T16:41:20.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:20.059+0000 7f64a4433700 1 -- 172.21.15.2:0/1384182039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f649c0fff90 msgr2=0x7f649c1973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:20.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:20.059+0000 7f64a4433700 1 --2- 172.21.15.2:0/1384182039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f649c0fff90 0x7f649c1973a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6498006010 tx=0x7f6498004ff0 comp rx=0 tx=0).stop 2023-11-16T16:41:20.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:20.059+0000 7f64a4433700 1 -- 172.21.15.2:0/1384182039 shutdown_connections 2023-11-16T16:41:20.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:20.059+0000 7f64a4433700 1 --2- 172.21.15.2:0/1384182039 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6488038870 0x7f648803ad20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:20.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:20.059+0000 7f64a4433700 1 --2- 172.21.15.2:0/1384182039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f649c0fff90 0x7f649c1973a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:20.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:20.059+0000 7f64a4433700 1 -- 172.21.15.2:0/1384182039 >> 172.21.15.2:0/1384182039 conn(0x7f649c0fb850 msgr2=0x7f649c1038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:20.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:20.059+0000 7f64a4433700 1 -- 172.21.15.2:0/1384182039 shutdown_connections 2023-11-16T16:41:20.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:20.059+0000 7f64a4433700 1 -- 172.21.15.2:0/1384182039 wait complete. 2023-11-16T16:41:20.062 INFO:teuthology.orchestra.run.smithi002.stderr:adjusted tunables profile to default 2023-11-16T16:41:20.538 INFO:tasks.cephadm:Adding mon.a on smithi002 2023-11-16T16:41:20.538 INFO:tasks.cephadm:Adding mon.b on smithi141 2023-11-16T16:41:20.538 INFO:tasks.cephadm:Adding mon.c on smithi155 2023-11-16T16:41:20.538 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch apply mon '3;smithi002:172.21.15.2=a;smithi141:172.21.15.141=b;smithi155:172.21.15.155=c' 2023-11-16T16:41:20.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:20 smithi002 ceph-mon[128712]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:20.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:20 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1384182039' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-16T16:41:20.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:20 smithi002 ceph-mon[128712]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:21.216 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:22.305 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.301+0000 7f75986cf700 1 -- 172.21.15.155:0/1264468278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75900fef00 msgr2=0x7f75900ff2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:22.305 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.301+0000 7f75986cf700 1 --2- 172.21.15.155:0/1264468278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75900fef00 0x7f75900ff2d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7580009910 tx=0x7f7580009c20 comp rx=0 tx=0).stop 2023-11-16T16:41:22.305 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.302+0000 7f75986cf700 1 -- 172.21.15.155:0/1264468278 shutdown_connections 2023-11-16T16:41:22.305 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.302+0000 7f75986cf700 1 --2- 172.21.15.155:0/1264468278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75900fef00 0x7f75900ff2d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:22.305 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.302+0000 7f75986cf700 1 -- 172.21.15.155:0/1264468278 >> 172.21.15.155:0/1264468278 conn(0x7f7590074930 msgr2=0x7f7590074d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:22.306 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.302+0000 7f75986cf700 1 -- 172.21.15.155:0/1264468278 shutdown_connections 2023-11-16T16:41:22.306 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.302+0000 7f75986cf700 1 -- 172.21.15.155:0/1264468278 wait complete. 2023-11-16T16:41:22.306 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.302+0000 7f75986cf700 1 Processor -- start 2023-11-16T16:41:22.306 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.302+0000 7f75986cf700 1 -- start start 2023-11-16T16:41:22.306 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.302+0000 7f75986cf700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75900fef00 0x7f7590196030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:22.306 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.302+0000 7f75986cf700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7590196570 con 0x7f75900fef00 2023-11-16T16:41:22.308 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.303+0000 7f759646b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75900fef00 0x7f7590196030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:22.308 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.303+0000 7f759646b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75900fef00 0x7f7590196030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.155:52334/0 (socket says 172.21.15.155:52334) 2023-11-16T16:41:22.308 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.303+0000 7f759646b700 1 -- 172.21.15.155:0/1821030758 learned_addr learned my addr 172.21.15.155:0/1821030758 (peer_addr_for_me v2:172.21.15.155:0/0) 2023-11-16T16:41:22.308 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.303+0000 7f759646b700 1 -- 172.21.15.155:0/1821030758 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75800095f0 con 0x7f75900fef00 2023-11-16T16:41:22.308 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.304+0000 7f759646b700 1 --2- 172.21.15.155:0/1821030758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75900fef00 0x7f7590196030 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7580006010 tx=0x7f7580004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:22.308 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.304+0000 7f75877fe700 1 -- 172.21.15.155:0/1821030758 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f758001d070 con 0x7f75900fef00 2023-11-16T16:41:22.308 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.304+0000 7f75986cf700 1 -- 172.21.15.155:0/1821030758 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7590109090 con 0x7f75900fef00 2023-11-16T16:41:22.308 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.304+0000 7f75877fe700 1 -- 172.21.15.155:0/1821030758 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f758000bc10 con 0x7f75900fef00 2023-11-16T16:41:22.308 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.304+0000 7f75877fe700 1 -- 172.21.15.155:0/1821030758 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7580003910 con 0x7f75900fef00 2023-11-16T16:41:22.308 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.304+0000 7f75986cf700 1 -- 172.21.15.155:0/1821030758 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7590108f00 con 0x7f75900fef00 2023-11-16T16:41:22.309 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.305+0000 7f75986cf700 1 -- 172.21.15.155:0/1821030758 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7590109220 con 0x7f75900fef00 2023-11-16T16:41:22.309 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.305+0000 7f75877fe700 1 -- 172.21.15.155:0/1821030758 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f758000bd80 con 0x7f75900fef00 2023-11-16T16:41:22.309 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.305+0000 7f75877fe700 1 --2- 172.21.15.155:0/1821030758 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f757c038870 0x7f757c03ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:22.309 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.305+0000 7f75877fe700 1 -- 172.21.15.155:0/1821030758 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f758004c5c0 con 0x7f75900fef00 2023-11-16T16:41:22.309 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.306+0000 7f7595c6a700 1 --2- 172.21.15.155:0/1821030758 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f757c038870 0x7f757c03ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:22.309 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.306+0000 7f7595c6a700 1 --2- 172.21.15.155:0/1821030758 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f757c038870 0x7f757c03ad20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f758c006fd0 tx=0x7f758c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:22.313 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.311+0000 7f75877fe700 1 -- 172.21.15.155:0/1821030758 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7580029360 con 0x7f75900fef00 2023-11-16T16:41:22.507 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.505+0000 7f75986cf700 1 -- 172.21.15.155:0/1821030758 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi141:172.21.15.141=b;smithi155:172.21.15.155=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f75900683a0 con 0x7f757c038870 2023-11-16T16:41:22.514 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.512+0000 7f75877fe700 1 -- 172.21.15.155:0/1821030758 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f75900683a0 con 0x7f757c038870 2023-11-16T16:41:22.514 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled mon update... 2023-11-16T16:41:22.516 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.514+0000 7f75986cf700 1 -- 172.21.15.155:0/1821030758 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f757c038870 msgr2=0x7f757c03ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:22.516 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.514+0000 7f75986cf700 1 --2- 172.21.15.155:0/1821030758 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f757c038870 0x7f757c03ad20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f758c006fd0 tx=0x7f758c006e40 comp rx=0 tx=0).stop 2023-11-16T16:41:22.516 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.514+0000 7f75986cf700 1 -- 172.21.15.155:0/1821030758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75900fef00 msgr2=0x7f7590196030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:22.517 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.514+0000 7f75986cf700 1 --2- 172.21.15.155:0/1821030758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75900fef00 0x7f7590196030 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7580006010 tx=0x7f7580004ff0 comp rx=0 tx=0).stop 2023-11-16T16:41:22.517 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.514+0000 7f75986cf700 1 -- 172.21.15.155:0/1821030758 shutdown_connections 2023-11-16T16:41:22.517 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.514+0000 7f75986cf700 1 --2- 172.21.15.155:0/1821030758 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f757c038870 0x7f757c03ad20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:22.517 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.514+0000 7f75986cf700 1 --2- 172.21.15.155:0/1821030758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75900fef00 0x7f7590196030 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:22.517 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.514+0000 7f75986cf700 1 -- 172.21.15.155:0/1821030758 >> 172.21.15.155:0/1821030758 conn(0x7f7590074930 msgr2=0x7f7590103090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:22.517 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.514+0000 7f75986cf700 1 -- 172.21.15.155:0/1821030758 shutdown_connections 2023-11-16T16:41:22.517 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:22.514+0000 7f75986cf700 1 -- 172.21.15.155:0/1821030758 wait complete. 2023-11-16T16:41:22.881 DEBUG:teuthology.orchestra.run.smithi141:mon.b> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mon.b.service 2023-11-16T16:41:22.883 DEBUG:teuthology.orchestra.run.smithi155:mon.c> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mon.c.service 2023-11-16T16:41:22.886 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-11-16T16:41:22.886 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T16:41:22.915 INFO:journalctl@ceph.mon.c.smithi155.stdout:-- Logs begin at Thu 2023-11-16 16:15:48 UTC. -- 2023-11-16T16:41:22.919 INFO:journalctl@ceph.mon.b.smithi141.stdout:-- Logs begin at Thu 2023-11-16 16:14:35 UTC. -- 2023-11-16T16:41:23.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:22 smithi002 ceph-mon[128712]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:23.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:22 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:22 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:22 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:22 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:22 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:22 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:22 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:23.732 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:24.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:23 smithi002 ceph-mon[128712]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi141:172.21.15.141=b;smithi155:172.21.15.155=c", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:24.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:23 smithi002 ceph-mon[128712]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi141:172.21.15.141=b;smithi155:172.21.15.155=c;count:3 2023-11-16T16:41:24.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:23 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:41:24.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:23 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:41:24.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:23 smithi002 ceph-mon[128712]: Deploying daemon mon.c on smithi155 2023-11-16T16:41:25.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:24 smithi002 ceph-mon[128712]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:25.471 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.467+0000 7fabc9c31700 1 -- 172.21.15.155:0/990440244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fabc40fef00 msgr2=0x7fabc40ff2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:25.471 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.467+0000 7fabc9c31700 1 --2- 172.21.15.155:0/990440244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fabc40fef00 0x7fabc40ff2d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009910 tx=0x7fabb4009c20 comp rx=0 tx=0).stop 2023-11-16T16:41:25.471 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.467+0000 7fabc9c31700 1 -- 172.21.15.155:0/990440244 shutdown_connections 2023-11-16T16:41:25.471 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.467+0000 7fabc9c31700 1 --2- 172.21.15.155:0/990440244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fabc40fef00 0x7fabc40ff2d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:25.471 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.467+0000 7fabc9c31700 1 -- 172.21.15.155:0/990440244 >> 172.21.15.155:0/990440244 conn(0x7fabc4074930 msgr2=0x7fabc4074d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:25.471 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.467+0000 7fabc9c31700 1 -- 172.21.15.155:0/990440244 shutdown_connections 2023-11-16T16:41:25.471 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.467+0000 7fabc9c31700 1 -- 172.21.15.155:0/990440244 wait complete. 2023-11-16T16:41:25.471 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.468+0000 7fabc9c31700 1 Processor -- start 2023-11-16T16:41:25.471 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.468+0000 7fabc9c31700 1 -- start start 2023-11-16T16:41:25.472 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.468+0000 7fabc9c31700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fabc40fef00 0x7fabc4195180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:25.473 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.468+0000 7fabc9c31700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc41956c0 con 0x7fabc40fef00 2023-11-16T16:41:25.473 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.469+0000 7fabc37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fabc40fef00 0x7fabc4195180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:25.473 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.469+0000 7fabc37fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fabc40fef00 0x7fabc4195180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.155:52340/0 (socket says 172.21.15.155:52340) 2023-11-16T16:41:25.473 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.469+0000 7fabc37fe700 1 -- 172.21.15.155:0/4089927105 learned_addr learned my addr 172.21.15.155:0/4089927105 (peer_addr_for_me v2:172.21.15.155:0/0) 2023-11-16T16:41:25.473 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.469+0000 7fabc37fe700 1 -- 172.21.15.155:0/4089927105 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb40095f0 con 0x7fabc40fef00 2023-11-16T16:41:25.473 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.469+0000 7fabc37fe700 1 --2- 172.21.15.155:0/4089927105 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fabc40fef00 0x7fabc4195180 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fabb4006010 tx=0x7fabb4004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:25.473 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.470+0000 7fabc0ff9700 1 -- 172.21.15.155:0/4089927105 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fabb401d070 con 0x7fabc40fef00 2023-11-16T16:41:25.473 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.470+0000 7fabc0ff9700 1 -- 172.21.15.155:0/4089927105 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabb400bc10 con 0x7fabc40fef00 2023-11-16T16:41:25.473 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.470+0000 7fabc9c31700 1 -- 172.21.15.155:0/4089927105 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc4109090 con 0x7fabc40fef00 2023-11-16T16:41:25.473 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.470+0000 7fabc0ff9700 1 -- 172.21.15.155:0/4089927105 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fabb4003910 con 0x7fabc40fef00 2023-11-16T16:41:25.473 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.470+0000 7fabc9c31700 1 -- 172.21.15.155:0/4089927105 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc4108f00 con 0x7fabc40fef00 2023-11-16T16:41:25.474 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.471+0000 7fabc0ff9700 1 -- 172.21.15.155:0/4089927105 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7fabb4003bd0 con 0x7fabc40fef00 2023-11-16T16:41:25.474 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.471+0000 7fabc9c31700 1 -- 172.21.15.155:0/4089927105 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc4109220 con 0x7fabc40fef00 2023-11-16T16:41:25.474 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.471+0000 7fabc0ff9700 1 --2- 172.21.15.155:0/4089927105 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fabac038870 0x7fabac03ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:25.475 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.471+0000 7fabc0ff9700 1 -- 172.21.15.155:0/4089927105 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fabb404c5e0 con 0x7fabc40fef00 2023-11-16T16:41:25.475 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.471+0000 7fabc2ffd700 1 --2- 172.21.15.155:0/4089927105 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fabac038870 0x7fabac03ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:25.475 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.472+0000 7fabc2ffd700 1 --2- 172.21.15.155:0/4089927105 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fabac038870 0x7fabac03ad20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fabb8006fd0 tx=0x7fabb8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:25.479 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.477+0000 7fabc0ff9700 1 -- 172.21.15.155:0/4089927105 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fabb4029360 con 0x7fabc40fef00 2023-11-16T16:41:25.727 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:25 smithi155 systemd[1]: Starting Ceph mon.c for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:41:25.762 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.760+0000 7fabc9c31700 1 -- 172.21.15.155:0/4089927105 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fabc4062120 con 0x7fabc40fef00 2023-11-16T16:41:25.763 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.760+0000 7fabc0ff9700 1 -- 172.21.15.155:0/4089927105 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+732 (secure 0 0 0) 0x7fabb4026030 con 0x7fabc40fef00 2023-11-16T16:41:25.763 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-11-16T16:41:25.763 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T16:39:39.719902Z","created":"2023-11-16T16:39:39.719902Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-16T16:41:25.765 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.762+0000 7fabc9c31700 1 -- 172.21.15.155:0/4089927105 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fabac038870 msgr2=0x7fabac03ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:25.765 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.763+0000 7fabc9c31700 1 --2- 172.21.15.155:0/4089927105 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fabac038870 0x7fabac03ad20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fabb8006fd0 tx=0x7fabb8006e40 comp rx=0 tx=0).stop 2023-11-16T16:41:25.765 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.763+0000 7fabc9c31700 1 -- 172.21.15.155:0/4089927105 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fabc40fef00 msgr2=0x7fabc4195180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:25.765 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.763+0000 7fabc9c31700 1 --2- 172.21.15.155:0/4089927105 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fabc40fef00 0x7fabc4195180 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fabb4006010 tx=0x7fabb4004ff0 comp rx=0 tx=0).stop 2023-11-16T16:41:25.765 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.763+0000 7fabc9c31700 1 -- 172.21.15.155:0/4089927105 shutdown_connections 2023-11-16T16:41:25.765 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.763+0000 7fabc9c31700 1 --2- 172.21.15.155:0/4089927105 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fabac038870 0x7fabac03ad20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:25.765 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.763+0000 7fabc9c31700 1 --2- 172.21.15.155:0/4089927105 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fabc40fef00 0x7fabc4195180 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:25.765 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.763+0000 7fabc9c31700 1 -- 172.21.15.155:0/4089927105 >> 172.21.15.155:0/4089927105 conn(0x7fabc4074930 msgr2=0x7fabc4067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:25.765 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.763+0000 7fabc9c31700 1 -- 172.21.15.155:0/4089927105 shutdown_connections 2023-11-16T16:41:25.766 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:25.763+0000 7fabc9c31700 1 -- 172.21.15.155:0/4089927105 wait complete. 2023-11-16T16:41:25.766 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2023-11-16T16:41:26.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:25 smithi155 podman[131546]: 2023-11-16T16:41:26.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:25 smithi002 ceph-mon[128712]: from='client.? 172.21.15.155:0/4089927105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T16:41:26.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: set uid:gid to 167:167 (ceph:ceph) 2023-11-16T16:41:26.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev), process ceph-mon, pid 7 2023-11-16T16:41:26.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: pidfile_write: ignore empty --pid-file 2023-11-16T16:41:26.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 bash[131546]: 5dbc5b6559385b1edf0268e758dca9306146da4dafdbc0de78513af7ced96e12 2023-11-16T16:41:26.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: load: jerasure load: lrc 2023-11-16T16:41:26.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: RocksDB version: 7.9.2 2023-11-16T16:41:26.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Git sha 0 2023-11-16T16:41:26.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Compile date 2023-11-15 23:53:54 2023-11-16T16:41:26.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: DB SUMMARY 2023-11-16T16:41:26.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: DB Session ID: 6DBA8L68VC06JMBSA0TC 2023-11-16T16:41:26.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: CURRENT file: CURRENT 2023-11-16T16:41:26.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: IDENTITY file: IDENTITY 2023-11-16T16:41:26.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-11-16T16:41:26.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-11-16T16:41:26.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-11-16T16:41:26.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.error_if_exists: 0 2023-11-16T16:41:26.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.create_if_missing: 0 2023-11-16T16:41:26.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.paranoid_checks: 1 2023-11-16T16:41:26.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.flush_verify_memtable_count: 1 2023-11-16T16:41:26.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-11-16T16:41:26.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-11-16T16:41:26.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.env: 0x55b4fc9d5720 2023-11-16T16:41:26.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.fs: PosixFileSystem 2023-11-16T16:41:26.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.info_log: 0x55b4ffd8d3c0 2023-11-16T16:41:26.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_file_opening_threads: 16 2023-11-16T16:41:26.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.statistics: (nil) 2023-11-16T16:41:26.622 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.use_fsync: 0 2023-11-16T16:41:26.622 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_log_file_size: 0 2023-11-16T16:41:26.622 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-16T16:41:26.622 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.log_file_time_to_roll: 0 2023-11-16T16:41:26.622 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.keep_log_file_num: 1000 2023-11-16T16:41:26.622 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.recycle_log_file_num: 0 2023-11-16T16:41:26.622 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.allow_fallocate: 1 2023-11-16T16:41:26.622 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.allow_mmap_reads: 0 2023-11-16T16:41:26.622 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.allow_mmap_writes: 0 2023-11-16T16:41:26.622 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.use_direct_reads: 0 2023-11-16T16:41:26.622 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-16T16:41:26.622 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.create_missing_column_families: 0 2023-11-16T16:41:26.622 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.db_log_dir: 2023-11-16T16:41:26.623 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.wal_dir: 2023-11-16T16:41:26.623 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.table_cache_numshardbits: 6 2023-11-16T16:41:26.623 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.WAL_ttl_seconds: 0 2023-11-16T16:41:26.623 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.WAL_size_limit_MB: 0 2023-11-16T16:41:26.623 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-16T16:41:26.623 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-16T16:41:26.623 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.is_fd_close_on_exec: 1 2023-11-16T16:41:26.623 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.advise_random_on_open: 1 2023-11-16T16:41:26.623 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.db_write_buffer_size: 0 2023-11-16T16:41:26.623 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.write_buffer_manager: 0x55b4feffe3c0 2023-11-16T16:41:26.623 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-16T16:41:26.623 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-16T16:41:26.623 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.use_adaptive_mutex: 0 2023-11-16T16:41:26.623 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.rate_limiter: (nil) 2023-11-16T16:41:26.624 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-16T16:41:26.624 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.wal_recovery_mode: 2 2023-11-16T16:41:26.624 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.enable_thread_tracking: 0 2023-11-16T16:41:26.624 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.enable_pipelined_write: 0 2023-11-16T16:41:26.624 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.unordered_write: 0 2023-11-16T16:41:26.624 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-16T16:41:26.624 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-16T16:41:26.624 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-16T16:41:26.624 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-16T16:41:26.624 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.row_cache: None 2023-11-16T16:41:26.624 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.wal_filter: None 2023-11-16T16:41:26.624 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-16T16:41:26.624 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.allow_ingest_behind: 0 2023-11-16T16:41:26.624 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.two_write_queues: 0 2023-11-16T16:41:26.624 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.manual_wal_flush: 0 2023-11-16T16:41:26.625 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.wal_compression: 0 2023-11-16T16:41:26.625 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.atomic_flush: 0 2023-11-16T16:41:26.625 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-16T16:41:26.625 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.persist_stats_to_disk: 0 2023-11-16T16:41:26.625 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.write_dbid_to_manifest: 0 2023-11-16T16:41:26.625 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.log_readahead_size: 0 2023-11-16T16:41:26.625 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-11-16T16:41:26.625 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.best_efforts_recovery: 0 2023-11-16T16:41:26.625 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-11-16T16:41:26.625 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-11-16T16:41:26.625 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.allow_data_in_errors: 0 2023-11-16T16:41:26.625 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.db_host_id: __hostname__ 2023-11-16T16:41:26.625 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.enforce_single_del_contracts: true 2023-11-16T16:41:26.626 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_background_jobs: 2 2023-11-16T16:41:26.626 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_background_compactions: -1 2023-11-16T16:41:26.626 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_subcompactions: 1 2023-11-16T16:41:26.626 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-16T16:41:26.626 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-16T16:41:26.626 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.delayed_write_rate : 16777216 2023-11-16T16:41:26.626 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_total_wal_size: 0 2023-11-16T16:41:26.626 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-16T16:41:26.626 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.stats_dump_period_sec: 600 2023-11-16T16:41:26.626 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.stats_persist_period_sec: 600 2023-11-16T16:41:26.626 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-16T16:41:26.626 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_open_files: -1 2023-11-16T16:41:26.626 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.bytes_per_sync: 0 2023-11-16T16:41:26.626 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.wal_bytes_per_sync: 0 2023-11-16T16:41:26.627 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.strict_bytes_per_sync: 0 2023-11-16T16:41:26.627 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compaction_readahead_size: 0 2023-11-16T16:41:26.627 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_background_flushes: -1 2023-11-16T16:41:26.627 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Compression algorithms supported: 2023-11-16T16:41:26.627 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-16T16:41:26.627 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: kZSTD supported: 0 2023-11-16T16:41:26.627 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: kXpressCompression supported: 0 2023-11-16T16:41:26.627 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: kLZ4HCCompression supported: 1 2023-11-16T16:41:26.627 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: kZlibCompression supported: 1 2023-11-16T16:41:26.627 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: kSnappyCompression supported: 1 2023-11-16T16:41:26.627 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: kLZ4Compression supported: 1 2023-11-16T16:41:26.627 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: kBZip2Compression supported: 0 2023-11-16T16:41:26.627 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Fast CRC32 supported: Supported on x86 2023-11-16T16:41:26.627 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: DMutex implementation: pthread_mutex_t 2023-11-16T16:41:26.628 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-11-16T16:41:26.628 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-11-16T16:41:26.628 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-16T16:41:26.628 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.merge_operator: 2023-11-16T16:41:26.628 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compaction_filter: None 2023-11-16T16:41:26.628 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compaction_filter_factory: None 2023-11-16T16:41:26.628 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.sst_partitioner_factory: None 2023-11-16T16:41:26.628 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.memtable_factory: SkipListFactory 2023-11-16T16:41:26.628 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.table_factory: BlockBasedTable 2023-11-16T16:41:26.628 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b4ffd8d4e0) 2023-11-16T16:41:26.628 INFO:journalctl@ceph.mon.c.smithi155.stdout: cache_index_and_filter_blocks: 1 2023-11-16T16:41:26.628 INFO:journalctl@ceph.mon.c.smithi155.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-16T16:41:26.628 INFO:journalctl@ceph.mon.c.smithi155.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-16T16:41:26.628 INFO:journalctl@ceph.mon.c.smithi155.stdout: pin_top_level_index_and_filter: 1 2023-11-16T16:41:26.629 INFO:journalctl@ceph.mon.c.smithi155.stdout: index_type: 0 2023-11-16T16:41:26.629 INFO:journalctl@ceph.mon.c.smithi155.stdout: data_block_index_type: 0 2023-11-16T16:41:26.629 INFO:journalctl@ceph.mon.c.smithi155.stdout: index_shortening: 1 2023-11-16T16:41:26.629 INFO:journalctl@ceph.mon.c.smithi155.stdout: data_block_hash_table_util_ratio: 0.750000 2023-11-16T16:41:26.629 INFO:journalctl@ceph.mon.c.smithi155.stdout: checksum: 4 2023-11-16T16:41:26.629 INFO:journalctl@ceph.mon.c.smithi155.stdout: no_block_cache: 0 2023-11-16T16:41:26.629 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_cache: 0x55b4ff09f1f0 2023-11-16T16:41:26.629 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_cache_name: BinnedLRUCache 2023-11-16T16:41:26.629 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_cache_options: 2023-11-16T16:41:26.629 INFO:journalctl@ceph.mon.c.smithi155.stdout: capacity : 536870912 2023-11-16T16:41:26.629 INFO:journalctl@ceph.mon.c.smithi155.stdout: num_shard_bits : 4 2023-11-16T16:41:26.629 INFO:journalctl@ceph.mon.c.smithi155.stdout: strict_capacity_limit : 0 2023-11-16T16:41:26.629 INFO:journalctl@ceph.mon.c.smithi155.stdout: high_pri_pool_ratio: 0.000 2023-11-16T16:41:26.630 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_cache_compressed: (nil) 2023-11-16T16:41:26.630 INFO:journalctl@ceph.mon.c.smithi155.stdout: persistent_cache: (nil) 2023-11-16T16:41:26.630 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_size: 4096 2023-11-16T16:41:26.630 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_size_deviation: 10 2023-11-16T16:41:26.630 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_restart_interval: 16 2023-11-16T16:41:26.630 INFO:journalctl@ceph.mon.c.smithi155.stdout: index_block_restart_interval: 1 2023-11-16T16:41:26.630 INFO:journalctl@ceph.mon.c.smithi155.stdout: metadata_block_size: 4096 2023-11-16T16:41:26.630 INFO:journalctl@ceph.mon.c.smithi155.stdout: partition_filters: 0 2023-11-16T16:41:26.630 INFO:journalctl@ceph.mon.c.smithi155.stdout: use_delta_encoding: 1 2023-11-16T16:41:26.630 INFO:journalctl@ceph.mon.c.smithi155.stdout: filter_policy: bloomfilter 2023-11-16T16:41:26.630 INFO:journalctl@ceph.mon.c.smithi155.stdout: whole_key_filtering: 1 2023-11-16T16:41:26.630 INFO:journalctl@ceph.mon.c.smithi155.stdout: verify_compression: 0 2023-11-16T16:41:26.630 INFO:journalctl@ceph.mon.c.smithi155.stdout: read_amp_bytes_per_bit: 0 2023-11-16T16:41:26.630 INFO:journalctl@ceph.mon.c.smithi155.stdout: format_version: 5 2023-11-16T16:41:26.631 INFO:journalctl@ceph.mon.c.smithi155.stdout: enable_index_compression: 1 2023-11-16T16:41:26.631 INFO:journalctl@ceph.mon.c.smithi155.stdout: block_align: 0 2023-11-16T16:41:26.631 INFO:journalctl@ceph.mon.c.smithi155.stdout: max_auto_readahead_size: 262144 2023-11-16T16:41:26.631 INFO:journalctl@ceph.mon.c.smithi155.stdout: prepopulate_block_cache: 0 2023-11-16T16:41:26.631 INFO:journalctl@ceph.mon.c.smithi155.stdout: initial_auto_readahead_size: 8192 2023-11-16T16:41:26.631 INFO:journalctl@ceph.mon.c.smithi155.stdout: num_file_reads_for_auto_readahead: 2 2023-11-16T16:41:26.631 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.write_buffer_size: 33554432 2023-11-16T16:41:26.631 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_write_buffer_number: 2 2023-11-16T16:41:26.631 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compression: NoCompression 2023-11-16T16:41:26.631 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.bottommost_compression: Disabled 2023-11-16T16:41:26.631 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.prefix_extractor: nullptr 2023-11-16T16:41:26.631 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-16T16:41:26.631 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.num_levels: 7 2023-11-16T16:41:26.631 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-16T16:41:26.631 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-16T16:41:26.632 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-16T16:41:26.632 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-16T16:41:26.632 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-16T16:41:26.632 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-16T16:41:26.632 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-16T16:41:26.632 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-16T16:41:26.632 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-11-16T16:41:26.632 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-16T16:41:26.632 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-11-16T16:41:26.632 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-11-16T16:41:26.632 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compression_opts.window_bits: -14 2023-11-16T16:41:26.632 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compression_opts.level: 32767 2023-11-16T16:41:26.632 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compression_opts.strategy: 0 2023-11-16T16:41:26.632 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-16T16:41:26.632 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-16T16:41:26.633 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-11-16T16:41:26.633 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-11-16T16:41:26.633 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compression_opts.enabled: false 2023-11-16T16:41:26.633 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-11-16T16:41:26.633 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-16T16:41:26.633 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-16T16:41:26.633 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-16T16:41:26.633 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.target_file_size_base: 67108864 2023-11-16T16:41:26.633 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.target_file_size_multiplier: 1 2023-11-16T16:41:26.633 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-16T16:41:26.633 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-16T16:41:26.633 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-16T16:41:26.633 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-16T16:41:26.633 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-16T16:41:26.634 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-16T16:41:26.634 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-16T16:41:26.634 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-16T16:41:26.634 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-16T16:41:26.634 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-16T16:41:26.634 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-16T16:41:26.634 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-16T16:41:26.634 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-11-16T16:41:26.634 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.arena_block_size: 1048576 2023-11-16T16:41:26.634 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-16T16:41:26.634 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-16T16:41:26.634 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.disable_auto_compactions: 0 2023-11-16T16:41:26.634 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-16T16:41:26.635 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-16T16:41:26.635 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-16T16:41:26.635 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-16T16:41:26.635 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-16T16:41:26.635 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-16T16:41:26.635 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-16T16:41:26.635 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-16T16:41:26.635 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-16T16:41:26.635 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-16T16:41:26.635 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.table_properties_collectors: 2023-11-16T16:41:26.635 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.inplace_update_support: 0 2023-11-16T16:41:26.635 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.inplace_update_num_locks: 10000 2023-11-16T16:41:26.635 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-16T16:41:26.635 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-16T16:41:26.636 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.memtable_huge_page_size: 0 2023-11-16T16:41:26.636 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.bloom_locality: 0 2023-11-16T16:41:26.636 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.max_successive_merges: 0 2023-11-16T16:41:26.636 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.optimize_filters_for_hits: 0 2023-11-16T16:41:26.636 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.paranoid_file_checks: 0 2023-11-16T16:41:26.636 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.force_consistency_checks: 1 2023-11-16T16:41:26.636 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.report_bg_io_stats: 0 2023-11-16T16:41:26.636 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.ttl: 2592000 2023-11-16T16:41:26.636 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.periodic_compaction_seconds: 0 2023-11-16T16:41:26.636 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-11-16T16:41:26.636 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-11-16T16:41:26.636 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.enable_blob_files: false 2023-11-16T16:41:26.636 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.min_blob_size: 0 2023-11-16T16:41:26.636 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.blob_file_size: 268435456 2023-11-16T16:41:26.636 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.blob_compression_type: NoCompression 2023-11-16T16:41:26.637 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.enable_blob_garbage_collection: false 2023-11-16T16:41:26.637 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-11-16T16:41:26.637 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-11-16T16:41:26.637 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-11-16T16:41:26.637 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.blob_file_starting_level: 0 2023-11-16T16:41:26.637 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-11-16T16:41:26.637 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-16T16:41:26.637 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-11-16T16:41:26.637 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d364beab-8956-416a-aec2-9b2767554a44 2023-11-16T16:41:26.637 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700152886124918, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-11-16T16:41:26.637 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-11-16T16:41:26.637 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700152886125321, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1700152886, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d364beab-8956-416a-aec2-9b2767554a44", "db_session_id": "6DBA8L68VC06JMBSA0TC", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-11-16T16:41:26.637 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700152886125388, "job": 1, "event": "recovery_finished"} 2023-11-16T16:41:26.638 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-11-16T16:41:26.638 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-16T16:41:26.638 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b4ff13c000 2023-11-16T16:41:26.638 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: DB pointer 0x55b4ff128000 2023-11-16T16:41:26.638 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-11-16T16:41:26.638 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-11-16T16:41:26.638 INFO:journalctl@ceph.mon.c.smithi155.stdout: ** DB Stats ** 2023-11-16T16:41:26.638 INFO:journalctl@ceph.mon.c.smithi155.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T16:41:26.638 INFO:journalctl@ceph.mon.c.smithi155.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-16T16:41:26.638 INFO:journalctl@ceph.mon.c.smithi155.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T16:41:26.638 INFO:journalctl@ceph.mon.c.smithi155.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T16:41:26.638 INFO:journalctl@ceph.mon.c.smithi155.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-16T16:41:26.638 INFO:journalctl@ceph.mon.c.smithi155.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T16:41:26.639 INFO:journalctl@ceph.mon.c.smithi155.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T16:41:26.639 INFO:journalctl@ceph.mon.c.smithi155.stdout: 2023-11-16T16:41:26.639 INFO:journalctl@ceph.mon.c.smithi155.stdout: ** Compaction Stats [default] ** 2023-11-16T16:41:26.639 INFO:journalctl@ceph.mon.c.smithi155.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T16:41:26.639 INFO:journalctl@ceph.mon.c.smithi155.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-11-16T16:41:26.639 INFO:journalctl@ceph.mon.c.smithi155.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T16:41:26.639 INFO:journalctl@ceph.mon.c.smithi155.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T16:41:26.639 INFO:journalctl@ceph.mon.c.smithi155.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T16:41:26.639 INFO:journalctl@ceph.mon.c.smithi155.stdout: 2023-11-16T16:41:26.639 INFO:journalctl@ceph.mon.c.smithi155.stdout: ** Compaction Stats [default] ** 2023-11-16T16:41:26.639 INFO:journalctl@ceph.mon.c.smithi155.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T16:41:26.639 INFO:journalctl@ceph.mon.c.smithi155.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T16:41:26.639 INFO:journalctl@ceph.mon.c.smithi155.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T16:41:26.639 INFO:journalctl@ceph.mon.c.smithi155.stdout: 2023-11-16T16:41:26.640 INFO:journalctl@ceph.mon.c.smithi155.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-11-16T16:41:26.640 INFO:journalctl@ceph.mon.c.smithi155.stdout: 2023-11-16T16:41:26.640 INFO:journalctl@ceph.mon.c.smithi155.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T16:41:26.640 INFO:journalctl@ceph.mon.c.smithi155.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T16:41:26.640 INFO:journalctl@ceph.mon.c.smithi155.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T16:41:26.640 INFO:journalctl@ceph.mon.c.smithi155.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T16:41:26.640 INFO:journalctl@ceph.mon.c.smithi155.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T16:41:26.640 INFO:journalctl@ceph.mon.c.smithi155.stdout: AddFile(Keys): cumulative 0, interval 0 2023-11-16T16:41:26.640 INFO:journalctl@ceph.mon.c.smithi155.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T16:41:26.640 INFO:journalctl@ceph.mon.c.smithi155.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T16:41:26.640 INFO:journalctl@ceph.mon.c.smithi155.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T16:41:26.640 INFO:journalctl@ceph.mon.c.smithi155.stdout: Block cache BinnedLRUCache@0x55b4ff09f1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-11-16T16:41:26.640 INFO:journalctl@ceph.mon.c.smithi155.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-11-16T16:41:26.640 INFO:journalctl@ceph.mon.c.smithi155.stdout: 2023-11-16T16:41:26.641 INFO:journalctl@ceph.mon.c.smithi155.stdout: ** File Read Latency Histogram By Level [default] ** 2023-11-16T16:41:26.641 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-11-16T16:41:26.641 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: using public_addr v2:172.21.15.155:0/0 -> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2023-11-16T16:41:26.641 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: starting mon.c rank -1 at public addrs [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] at bind addrs [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:41:26.641 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: mon.c@-1(???) e0 preinit fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:41:26.641 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: mon.c@-1(synchronizing).mds e1 new map 2023-11-16T16:41:26.641 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: mon.c@-1(synchronizing).mds e1 print_map 2023-11-16T16:41:26.641 INFO:journalctl@ceph.mon.c.smithi155.stdout: e1 2023-11-16T16:41:26.641 INFO:journalctl@ceph.mon.c.smithi155.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-11-16T16:41:26.641 INFO:journalctl@ceph.mon.c.smithi155.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T16:41:26.641 INFO:journalctl@ceph.mon.c.smithi155.stdout: legacy client fscid: -1 2023-11-16T16:41:26.641 INFO:journalctl@ceph.mon.c.smithi155.stdout: 2023-11-16T16:41:26.641 INFO:journalctl@ceph.mon.c.smithi155.stdout: No filesystems configured 2023-11-16T16:41:26.642 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-11-16T16:41:26.642 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-11-16T16:41:26.642 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-11-16T16:41:26.642 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-11-16T16:41:26.642 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-11-16T16:41:26.642 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-11-16T16:41:26.642 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-16T16:41:26.642 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T16:41:26.642 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T16:41:26.642 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T16:41:26.642 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.642 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:41:26.642 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:26.643 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.643 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.643 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.643 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.643 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi141", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:41:26.643 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:26.643 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:26.643 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:41:26.643 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: Updating smithi141:/etc/ceph/ceph.conf 2023-11-16T16:41:26.643 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:26.643 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: Updating smithi141:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:26.643 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.643 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.644 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.644 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: Updating smithi141:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:26.644 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: Updating smithi141:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:41:26.644 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:26.644 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: Deploying cephadm binary to smithi155 2023-11-16T16:41:26.644 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:26.644 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.644 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: Added host smithi155 2023-11-16T16:41:26.644 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:26.644 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.644 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:26.644 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.644 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.645 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:26.645 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:41:26.645 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:26.645 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.645 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.645 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.645 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.645 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi155", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:41:26.645 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:26.645 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:26.645 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:41:26.645 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: Updating smithi155:/etc/ceph/ceph.conf 2023-11-16T16:41:26.645 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: Updating smithi155:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:26.645 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:26.646 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: Updating smithi155:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:41:26.646 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.646 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.646 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.646 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1384182039' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-16T16:41:26.646 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:26.646 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1384182039' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-16T16:41:26.646 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:26.646 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:26.647 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.647 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:26.647 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:26.647 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:26.647 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:26.647 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:26.647 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:26.647 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi141:172.21.15.141=b;smithi155:172.21.15.155=c", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:26.647 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi141:172.21.15.141=b;smithi155:172.21.15.155=c;count:3 2023-11-16T16:41:26.647 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:41:26.647 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:41:26.647 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: Deploying daemon mon.c on smithi155 2023-11-16T16:41:26.647 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:26.648 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: from='client.? 172.21.15.155:0/4089927105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T16:41:26.648 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-11-16T16:41:26.648 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-11-16T16:41:26.648 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: expand_channel_meta expand map: {default=false} 2023-11-16T16:41:26.648 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: expand_channel_meta from 'false' to 'false' 2023-11-16T16:41:26.648 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: expand_channel_meta expanded map: {default=false} 2023-11-16T16:41:26.648 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: expand_channel_meta expand map: {default=info} 2023-11-16T16:41:26.648 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: expand_channel_meta from 'info' to 'info' 2023-11-16T16:41:26.648 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: expand_channel_meta expanded map: {default=info} 2023-11-16T16:41:26.648 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: expand_channel_meta expand map: {default=daemon} 2023-11-16T16:41:26.648 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: expand_channel_meta from 'daemon' to 'daemon' 2023-11-16T16:41:26.648 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: expand_channel_meta expanded map: {default=daemon} 2023-11-16T16:41:26.648 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: expand_channel_meta expand map: {default=debug} 2023-11-16T16:41:26.648 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: expand_channel_meta from 'debug' to 'debug' 2023-11-16T16:41:26.649 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 ceph-mon[131607]: expand_channel_meta expanded map: {default=debug} 2023-11-16T16:41:26.649 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:26 smithi155 systemd[1]: Started Ceph mon.c for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:41:27.209 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-11-16T16:41:27.209 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T16:41:27.860 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:41:31.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:31 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:41:31.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:31 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:31.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:31 smithi155 ceph-mon[131607]: mon.a calling monitor election 2023-11-16T16:41:31.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:31 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:31.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:31 smithi155 ceph-mon[131607]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:31.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:31 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:31.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:31 smithi155 ceph-mon[131607]: mon.c calling monitor election 2023-11-16T16:41:31.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:31 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:31.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:31 smithi155 ceph-mon[131607]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:31.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:31 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:31.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:31 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:31.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:31 smithi155 ceph-mon[131607]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-11-16T16:41:31.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:31 smithi155 ceph-mon[131607]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],c=[v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0]} removed_ranks: {} 2023-11-16T16:41:31.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:31 smithi155 ceph-mon[131607]: fsmap 2023-11-16T16:41:31.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:31 smithi155 ceph-mon[131607]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:31.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:31 smithi155 ceph-mon[131607]: mgrmap e13: x(active, since 47s) 2023-11-16T16:41:31.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:31 smithi155 ceph-mon[131607]: overall HEALTH_OK 2023-11-16T16:41:31.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:31 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:31.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:31 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:31.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:31 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:41:31.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:31 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:31.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:31 smithi002 ceph-mon[128712]: mon.a calling monitor election 2023-11-16T16:41:31.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:31 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:31.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:31 smithi002 ceph-mon[128712]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:31.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:31 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:31.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:31 smithi002 ceph-mon[128712]: mon.c calling monitor election 2023-11-16T16:41:31.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:31 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:31.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:31 smithi002 ceph-mon[128712]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:31.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:31 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:31.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:31 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:31.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:31 smithi002 ceph-mon[128712]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-11-16T16:41:31.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:31 smithi002 ceph-mon[128712]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],c=[v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0]} removed_ranks: {} 2023-11-16T16:41:31.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:31 smithi002 ceph-mon[128712]: fsmap 2023-11-16T16:41:31.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:31 smithi002 ceph-mon[128712]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:31.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:31 smithi002 ceph-mon[128712]: mgrmap e13: x(active, since 47s) 2023-11-16T16:41:31.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:31 smithi002 ceph-mon[128712]: overall HEALTH_OK 2023-11-16T16:41:31.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:31 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:31.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:31 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:32.062 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.059+0000 7f2d83d47700 1 -- 172.21.15.155:0/836894648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d640035c0 msgr2=0x7f2d64005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:32.062 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.059+0000 7f2d83d47700 1 --2- 172.21.15.155:0/836894648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d640035c0 0x7f2d64005a40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c0fec90 tx=0x7f2d6c009e80 comp rx=0 tx=0).stop 2023-11-16T16:41:32.062 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.059+0000 7f2d83d47700 1 -- 172.21.15.155:0/836894648 shutdown_connections 2023-11-16T16:41:32.062 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.059+0000 7f2d83d47700 1 --2- 172.21.15.155:0/836894648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d640035c0 0x7f2d64005a40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:32.062 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.059+0000 7f2d83d47700 1 -- 172.21.15.155:0/836894648 >> 172.21.15.155:0/836894648 conn(0x7f2d7c0fa8a0 msgr2=0x7f2d7c0fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:32.062 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.059+0000 7f2d83d47700 1 -- 172.21.15.155:0/836894648 shutdown_connections 2023-11-16T16:41:32.062 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.059+0000 7f2d83d47700 1 -- 172.21.15.155:0/836894648 wait complete. 2023-11-16T16:41:32.065 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.060+0000 7f2d83d47700 1 Processor -- start 2023-11-16T16:41:32.065 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.060+0000 7f2d83d47700 1 -- start start 2023-11-16T16:41:32.065 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.061+0000 7f2d83d47700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d640035c0 0x7f2d7c195280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:32.065 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.061+0000 7f2d83d47700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d7c1957c0 0x7f2d7c199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:32.065 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.061+0000 7f2d83d47700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c195d40 con 0x7f2d7c1957c0 2023-11-16T16:41:32.066 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.061+0000 7f2d83d47700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c195eb0 con 0x7f2d640035c0 2023-11-16T16:41:32.066 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.061+0000 7f2d81ae3700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d640035c0 0x7f2d7c195280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:32.066 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.061+0000 7f2d81ae3700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d640035c0 0x7f2d7c195280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:38952/0 (socket says 172.21.15.155:38952) 2023-11-16T16:41:32.066 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.061+0000 7f2d81ae3700 1 -- 172.21.15.155:0/1265481410 learned_addr learned my addr 172.21.15.155:0/1265481410 (peer_addr_for_me v2:172.21.15.155:0/0) 2023-11-16T16:41:32.066 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.061+0000 7f2d81ae3700 1 -- 172.21.15.155:0/1265481410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d640035c0 msgr2=0x7f2d7c195280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-11-16T16:41:32.066 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.061+0000 7f2d81ae3700 1 -- 172.21.15.155:0/1265481410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d640035c0 msgr2=0x7f2d7c195280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-16T16:41:32.066 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.061+0000 7f2d81ae3700 1 --2- 172.21.15.155:0/1265481410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d640035c0 0x7f2d7c195280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-16T16:41:32.066 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.061+0000 7f2d81ae3700 1 --2- 172.21.15.155:0/1265481410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d640035c0 0x7f2d7c195280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:41:32.067 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.061+0000 7f2d812e2700 1 --2- 172.21.15.155:0/1265481410 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d7c1957c0 0x7f2d7c199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:32.067 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.061+0000 7f2d812e2700 1 -- 172.21.15.155:0/1265481410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d640035c0 msgr2=0x7f2d7c195280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:41:32.067 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.061+0000 7f2d812e2700 1 --2- 172.21.15.155:0/1265481410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d640035c0 0x7f2d7c195280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:32.067 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.061+0000 7f2d812e2700 1 -- 172.21.15.155:0/1265481410 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6c0098d0 con 0x7f2d7c1957c0 2023-11-16T16:41:32.067 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.062+0000 7f2d812e2700 1 --2- 172.21.15.155:0/1265481410 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d7c1957c0 0x7f2d7c199c30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2d7000b4d0 tx=0x7f2d7000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:32.067 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.062+0000 7f2d7affd700 1 -- 172.21.15.155:0/1265481410 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f2d70011630 con 0x7f2d7c1957c0 2023-11-16T16:41:32.067 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.062+0000 7f2d7affd700 1 -- 172.21.15.155:0/1265481410 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d70011c50 con 0x7f2d7c1957c0 2023-11-16T16:41:32.067 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.062+0000 7f2d83d47700 1 -- 172.21.15.155:0/1265481410 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7c106bb0 con 0x7f2d7c1957c0 2023-11-16T16:41:32.068 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.062+0000 7f2d7affd700 1 -- 172.21.15.155:0/1265481410 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f2d70010d90 con 0x7f2d7c1957c0 2023-11-16T16:41:32.068 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.062+0000 7f2d83d47700 1 -- 172.21.15.155:0/1265481410 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7c106a20 con 0x7f2d7c1957c0 2023-11-16T16:41:32.069 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.063+0000 7f2d7affd700 1 -- 172.21.15.155:0/1265481410 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f2d7001a400 con 0x7f2d7c1957c0 2023-11-16T16:41:32.069 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.063+0000 7f2d83d47700 1 -- 172.21.15.155:0/1265481410 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7c106d40 con 0x7f2d7c1957c0 2023-11-16T16:41:32.069 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.063+0000 7f2d7affd700 1 --2- 172.21.15.155:0/1265481410 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2d680386b0 0x7f2d6803ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:32.069 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.063+0000 7f2d7affd700 1 -- 172.21.15.155:0/1265481410 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2d7004ae40 con 0x7f2d7c1957c0 2023-11-16T16:41:32.069 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.064+0000 7f2d81ae3700 1 --2- 172.21.15.155:0/1265481410 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2d680386b0 0x7f2d6803ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:32.069 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.064+0000 7f2d81ae3700 1 --2- 172.21.15.155:0/1265481410 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2d680386b0 0x7f2d6803ab60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c007590 tx=0x7f2d6c0074e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:32.071 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.069+0000 7f2d7affd700 1 -- 172.21.15.155:0/1265481410 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2d700176d0 con 0x7f2d7c1957c0 2023-11-16T16:41:32.340 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.337+0000 7f2d83d47700 1 -- 172.21.15.155:0/1265481410 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d7c19fed0 con 0x7f2d7c1957c0 2023-11-16T16:41:32.340 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.338+0000 7f2d7affd700 1 -- 172.21.15.155:0/1265481410 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1002 (secure 0 0 0) 0x7f2d700172c0 con 0x7f2d7c1957c0 2023-11-16T16:41:32.341 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-11-16T16:41:32.341 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":2,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T16:41:26.150126Z","created":"2023-11-16T16:39:39.719902Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-11-16T16:41:32.343 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.340+0000 7f2d83d47700 1 -- 172.21.15.155:0/1265481410 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2d680386b0 msgr2=0x7f2d6803ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:32.343 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.340+0000 7f2d83d47700 1 --2- 172.21.15.155:0/1265481410 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2d680386b0 0x7f2d6803ab60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c007590 tx=0x7f2d6c0074e0 comp rx=0 tx=0).stop 2023-11-16T16:41:32.343 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.340+0000 7f2d83d47700 1 -- 172.21.15.155:0/1265481410 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d7c1957c0 msgr2=0x7f2d7c199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:32.343 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.340+0000 7f2d83d47700 1 --2- 172.21.15.155:0/1265481410 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d7c1957c0 0x7f2d7c199c30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2d7000b4d0 tx=0x7f2d7000b890 comp rx=0 tx=0).stop 2023-11-16T16:41:32.343 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.340+0000 7f2d83d47700 1 -- 172.21.15.155:0/1265481410 shutdown_connections 2023-11-16T16:41:32.343 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.340+0000 7f2d83d47700 1 --2- 172.21.15.155:0/1265481410 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2d680386b0 0x7f2d6803ab60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:32.344 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.340+0000 7f2d83d47700 1 --2- 172.21.15.155:0/1265481410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d640035c0 0x7f2d7c195280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:32.344 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.340+0000 7f2d83d47700 1 --2- 172.21.15.155:0/1265481410 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d7c1957c0 0x7f2d7c199c30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:32.344 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.340+0000 7f2d83d47700 1 -- 172.21.15.155:0/1265481410 >> 172.21.15.155:0/1265481410 conn(0x7f2d7c0fa8a0 msgr2=0x7f2d7c069370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:32.344 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.340+0000 7f2d83d47700 1 -- 172.21.15.155:0/1265481410 shutdown_connections 2023-11-16T16:41:32.344 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:32.340+0000 7f2d83d47700 1 -- 172.21.15.155:0/1265481410 wait complete. 2023-11-16T16:41:32.344 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 2 2023-11-16T16:41:32.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:32 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:32.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:32 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:32.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:32 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:32.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:32 smithi155 ceph-mon[131607]: Deploying daemon mon.b on smithi141 2023-11-16T16:41:32.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:32 smithi155 ceph-mon[131607]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:32.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:32 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:32.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:32 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:32.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:32 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:32.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:32 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:32.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:32 smithi002 ceph-mon[128712]: Deploying daemon mon.b on smithi141 2023-11-16T16:41:32.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:32 smithi002 ceph-mon[128712]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:32.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:32 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:33.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:33 smithi155 ceph-mon[131607]: from='client.? 172.21.15.155:0/1265481410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T16:41:33.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:33 smithi002 ceph-mon[128712]: from='client.? 172.21.15.155:0/1265481410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T16:41:33.723 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-11-16T16:41:33.724 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-16T16:41:34.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:33 smithi141 systemd[1]: Starting Ceph mon.b for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:41:34.359 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 podman[131144]: 2023-11-16T16:41:34.359 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: set uid:gid to 167:167 (ceph:ceph) 2023-11-16T16:41:34.359 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev), process ceph-mon, pid 7 2023-11-16T16:41:34.359 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: pidfile_write: ignore empty --pid-file 2023-11-16T16:41:34.359 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: load: jerasure load: lrc 2023-11-16T16:41:34.360 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: RocksDB version: 7.9.2 2023-11-16T16:41:34.360 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Git sha 0 2023-11-16T16:41:34.360 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Compile date 2023-11-15 23:53:54 2023-11-16T16:41:34.360 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: DB SUMMARY 2023-11-16T16:41:34.360 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: DB Session ID: EYIBH9ZVN9477VYRMBNM 2023-11-16T16:41:34.360 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: CURRENT file: CURRENT 2023-11-16T16:41:34.360 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: IDENTITY file: IDENTITY 2023-11-16T16:41:34.360 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-11-16T16:41:34.360 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-11-16T16:41:34.361 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-11-16T16:41:34.361 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.error_if_exists: 0 2023-11-16T16:41:34.361 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.create_if_missing: 0 2023-11-16T16:41:34.361 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.paranoid_checks: 1 2023-11-16T16:41:34.361 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.flush_verify_memtable_count: 1 2023-11-16T16:41:34.361 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-11-16T16:41:34.361 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-11-16T16:41:34.361 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.env: 0x558aa1282720 2023-11-16T16:41:34.362 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.fs: PosixFileSystem 2023-11-16T16:41:34.362 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.info_log: 0x558aa2ffb3c0 2023-11-16T16:41:34.362 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_file_opening_threads: 16 2023-11-16T16:41:34.362 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.statistics: (nil) 2023-11-16T16:41:34.362 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.use_fsync: 0 2023-11-16T16:41:34.362 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_log_file_size: 0 2023-11-16T16:41:34.362 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-16T16:41:34.362 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.log_file_time_to_roll: 0 2023-11-16T16:41:34.363 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.keep_log_file_num: 1000 2023-11-16T16:41:34.363 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.recycle_log_file_num: 0 2023-11-16T16:41:34.363 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.allow_fallocate: 1 2023-11-16T16:41:34.363 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.allow_mmap_reads: 0 2023-11-16T16:41:34.363 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.allow_mmap_writes: 0 2023-11-16T16:41:34.363 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.use_direct_reads: 0 2023-11-16T16:41:34.363 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-16T16:41:34.363 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.create_missing_column_families: 0 2023-11-16T16:41:34.363 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.db_log_dir: 2023-11-16T16:41:34.363 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.wal_dir: 2023-11-16T16:41:34.363 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.table_cache_numshardbits: 6 2023-11-16T16:41:34.364 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.WAL_ttl_seconds: 0 2023-11-16T16:41:34.364 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.WAL_size_limit_MB: 0 2023-11-16T16:41:34.364 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-16T16:41:34.364 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-16T16:41:34.364 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.is_fd_close_on_exec: 1 2023-11-16T16:41:34.364 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.advise_random_on_open: 1 2023-11-16T16:41:34.364 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.db_write_buffer_size: 0 2023-11-16T16:41:34.364 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.write_buffer_manager: 0x558aa226c3c0 2023-11-16T16:41:34.364 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-16T16:41:34.364 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-16T16:41:34.365 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.use_adaptive_mutex: 0 2023-11-16T16:41:34.365 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.rate_limiter: (nil) 2023-11-16T16:41:34.365 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-16T16:41:34.365 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.wal_recovery_mode: 2 2023-11-16T16:41:34.365 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.enable_thread_tracking: 0 2023-11-16T16:41:34.365 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.enable_pipelined_write: 0 2023-11-16T16:41:34.365 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.unordered_write: 0 2023-11-16T16:41:34.365 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-16T16:41:34.365 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-16T16:41:34.365 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-16T16:41:34.365 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-16T16:41:34.366 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.row_cache: None 2023-11-16T16:41:34.366 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.wal_filter: None 2023-11-16T16:41:34.366 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-16T16:41:34.366 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.allow_ingest_behind: 0 2023-11-16T16:41:34.366 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.two_write_queues: 0 2023-11-16T16:41:34.366 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.manual_wal_flush: 0 2023-11-16T16:41:34.366 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.wal_compression: 0 2023-11-16T16:41:34.366 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.atomic_flush: 0 2023-11-16T16:41:34.366 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-16T16:41:34.366 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.persist_stats_to_disk: 0 2023-11-16T16:41:34.367 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.write_dbid_to_manifest: 0 2023-11-16T16:41:34.367 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.log_readahead_size: 0 2023-11-16T16:41:34.367 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-11-16T16:41:34.367 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.best_efforts_recovery: 0 2023-11-16T16:41:34.367 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-11-16T16:41:34.367 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-11-16T16:41:34.367 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.allow_data_in_errors: 0 2023-11-16T16:41:34.367 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.db_host_id: __hostname__ 2023-11-16T16:41:34.367 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.enforce_single_del_contracts: true 2023-11-16T16:41:34.367 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_background_jobs: 2 2023-11-16T16:41:34.367 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_background_compactions: -1 2023-11-16T16:41:34.368 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_subcompactions: 1 2023-11-16T16:41:34.368 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-16T16:41:34.368 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-16T16:41:34.368 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.delayed_write_rate : 16777216 2023-11-16T16:41:34.368 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_total_wal_size: 0 2023-11-16T16:41:34.368 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-16T16:41:34.368 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.stats_dump_period_sec: 600 2023-11-16T16:41:34.368 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.stats_persist_period_sec: 600 2023-11-16T16:41:34.368 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-16T16:41:34.368 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_open_files: -1 2023-11-16T16:41:34.369 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.bytes_per_sync: 0 2023-11-16T16:41:34.369 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.wal_bytes_per_sync: 0 2023-11-16T16:41:34.369 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.strict_bytes_per_sync: 0 2023-11-16T16:41:34.369 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compaction_readahead_size: 0 2023-11-16T16:41:34.369 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_background_flushes: -1 2023-11-16T16:41:34.369 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Compression algorithms supported: 2023-11-16T16:41:34.369 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-16T16:41:34.369 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: kZSTD supported: 0 2023-11-16T16:41:34.369 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: kXpressCompression supported: 0 2023-11-16T16:41:34.369 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: kLZ4HCCompression supported: 1 2023-11-16T16:41:34.370 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: kZlibCompression supported: 1 2023-11-16T16:41:34.370 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: kSnappyCompression supported: 1 2023-11-16T16:41:34.370 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: kLZ4Compression supported: 1 2023-11-16T16:41:34.370 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: kBZip2Compression supported: 0 2023-11-16T16:41:34.370 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Fast CRC32 supported: Supported on x86 2023-11-16T16:41:34.370 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: DMutex implementation: pthread_mutex_t 2023-11-16T16:41:34.370 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-11-16T16:41:34.370 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-11-16T16:41:34.370 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-16T16:41:34.370 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.merge_operator: 2023-11-16T16:41:34.371 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compaction_filter: None 2023-11-16T16:41:34.371 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compaction_filter_factory: None 2023-11-16T16:41:34.371 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.sst_partitioner_factory: None 2023-11-16T16:41:34.371 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.memtable_factory: SkipListFactory 2023-11-16T16:41:34.371 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.table_factory: BlockBasedTable 2023-11-16T16:41:34.371 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558aa2ffb4e0) 2023-11-16T16:41:34.371 INFO:journalctl@ceph.mon.b.smithi141.stdout: cache_index_and_filter_blocks: 1 2023-11-16T16:41:34.371 INFO:journalctl@ceph.mon.b.smithi141.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-16T16:41:34.371 INFO:journalctl@ceph.mon.b.smithi141.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-16T16:41:34.371 INFO:journalctl@ceph.mon.b.smithi141.stdout: pin_top_level_index_and_filter: 1 2023-11-16T16:41:34.371 INFO:journalctl@ceph.mon.b.smithi141.stdout: index_type: 0 2023-11-16T16:41:34.372 INFO:journalctl@ceph.mon.b.smithi141.stdout: data_block_index_type: 0 2023-11-16T16:41:34.372 INFO:journalctl@ceph.mon.b.smithi141.stdout: index_shortening: 1 2023-11-16T16:41:34.372 INFO:journalctl@ceph.mon.b.smithi141.stdout: data_block_hash_table_util_ratio: 0.750000 2023-11-16T16:41:34.372 INFO:journalctl@ceph.mon.b.smithi141.stdout: checksum: 4 2023-11-16T16:41:34.372 INFO:journalctl@ceph.mon.b.smithi141.stdout: no_block_cache: 0 2023-11-16T16:41:34.372 INFO:journalctl@ceph.mon.b.smithi141.stdout: block_cache: 0x558aa230d1f0 2023-11-16T16:41:34.372 INFO:journalctl@ceph.mon.b.smithi141.stdout: block_cache_name: BinnedLRUCache 2023-11-16T16:41:34.372 INFO:journalctl@ceph.mon.b.smithi141.stdout: block_cache_options: 2023-11-16T16:41:34.373 INFO:journalctl@ceph.mon.b.smithi141.stdout: capacity : 536870912 2023-11-16T16:41:34.373 INFO:journalctl@ceph.mon.b.smithi141.stdout: num_shard_bits : 4 2023-11-16T16:41:34.373 INFO:journalctl@ceph.mon.b.smithi141.stdout: strict_capacity_limit : 0 2023-11-16T16:41:34.373 INFO:journalctl@ceph.mon.b.smithi141.stdout: high_pri_pool_ratio: 0.000 2023-11-16T16:41:34.373 INFO:journalctl@ceph.mon.b.smithi141.stdout: block_cache_compressed: (nil) 2023-11-16T16:41:34.373 INFO:journalctl@ceph.mon.b.smithi141.stdout: persistent_cache: (nil) 2023-11-16T16:41:34.373 INFO:journalctl@ceph.mon.b.smithi141.stdout: block_size: 4096 2023-11-16T16:41:34.373 INFO:journalctl@ceph.mon.b.smithi141.stdout: block_size_deviation: 10 2023-11-16T16:41:34.373 INFO:journalctl@ceph.mon.b.smithi141.stdout: block_restart_interval: 16 2023-11-16T16:41:34.373 INFO:journalctl@ceph.mon.b.smithi141.stdout: index_block_restart_interval: 1 2023-11-16T16:41:34.373 INFO:journalctl@ceph.mon.b.smithi141.stdout: metadata_block_size: 4096 2023-11-16T16:41:34.374 INFO:journalctl@ceph.mon.b.smithi141.stdout: partition_filters: 0 2023-11-16T16:41:34.374 INFO:journalctl@ceph.mon.b.smithi141.stdout: use_delta_encoding: 1 2023-11-16T16:41:34.374 INFO:journalctl@ceph.mon.b.smithi141.stdout: filter_policy: bloomfilter 2023-11-16T16:41:34.374 INFO:journalctl@ceph.mon.b.smithi141.stdout: whole_key_filtering: 1 2023-11-16T16:41:34.374 INFO:journalctl@ceph.mon.b.smithi141.stdout: verify_compression: 0 2023-11-16T16:41:34.374 INFO:journalctl@ceph.mon.b.smithi141.stdout: read_amp_bytes_per_bit: 0 2023-11-16T16:41:34.374 INFO:journalctl@ceph.mon.b.smithi141.stdout: format_version: 5 2023-11-16T16:41:34.374 INFO:journalctl@ceph.mon.b.smithi141.stdout: enable_index_compression: 1 2023-11-16T16:41:34.374 INFO:journalctl@ceph.mon.b.smithi141.stdout: block_align: 0 2023-11-16T16:41:34.375 INFO:journalctl@ceph.mon.b.smithi141.stdout: max_auto_readahead_size: 262144 2023-11-16T16:41:34.375 INFO:journalctl@ceph.mon.b.smithi141.stdout: prepopulate_block_cache: 0 2023-11-16T16:41:34.375 INFO:journalctl@ceph.mon.b.smithi141.stdout: initial_auto_readahead_size: 8192 2023-11-16T16:41:34.375 INFO:journalctl@ceph.mon.b.smithi141.stdout: num_file_reads_for_auto_readahead: 2 2023-11-16T16:41:34.375 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.write_buffer_size: 33554432 2023-11-16T16:41:34.375 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_write_buffer_number: 2 2023-11-16T16:41:34.375 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compression: NoCompression 2023-11-16T16:41:34.375 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.bottommost_compression: Disabled 2023-11-16T16:41:34.375 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.prefix_extractor: nullptr 2023-11-16T16:41:34.375 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-16T16:41:34.375 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.num_levels: 7 2023-11-16T16:41:34.376 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-16T16:41:34.376 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-16T16:41:34.376 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-16T16:41:34.376 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-16T16:41:34.376 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-16T16:41:34.376 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-16T16:41:34.376 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-16T16:41:34.377 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:41:34.378 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-16T16:41:34.378 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-11-16T16:41:34.378 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-16T16:41:34.378 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-11-16T16:41:34.378 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-11-16T16:41:34.379 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compression_opts.window_bits: -14 2023-11-16T16:41:34.379 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compression_opts.level: 32767 2023-11-16T16:41:34.379 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compression_opts.strategy: 0 2023-11-16T16:41:34.379 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-16T16:41:34.379 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-16T16:41:34.379 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-11-16T16:41:34.379 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-11-16T16:41:34.379 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 bash[131144]: 78c734a0494849d3fd09cfe22adc0e7563a4f23c032253e67b666f3b7b0dd428 2023-11-16T16:41:34.379 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compression_opts.enabled: false 2023-11-16T16:41:34.379 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-11-16T16:41:34.380 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-16T16:41:34.380 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-16T16:41:34.380 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-16T16:41:34.380 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.target_file_size_base: 67108864 2023-11-16T16:41:34.380 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.target_file_size_multiplier: 1 2023-11-16T16:41:34.380 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-16T16:41:34.380 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-16T16:41:34.380 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-16T16:41:34.380 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-16T16:41:34.380 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-16T16:41:34.381 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-16T16:41:34.381 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-16T16:41:34.381 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-16T16:41:34.381 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-16T16:41:34.381 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-16T16:41:34.381 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-16T16:41:34.381 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-16T16:41:34.381 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-11-16T16:41:34.381 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.arena_block_size: 1048576 2023-11-16T16:41:34.381 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-16T16:41:34.381 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-16T16:41:34.382 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.disable_auto_compactions: 0 2023-11-16T16:41:34.382 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-16T16:41:34.382 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-16T16:41:34.382 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-16T16:41:34.382 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-16T16:41:34.382 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-16T16:41:34.382 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-16T16:41:34.382 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-16T16:41:34.382 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-16T16:41:34.383 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-16T16:41:34.383 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-16T16:41:34.383 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.table_properties_collectors: 2023-11-16T16:41:34.383 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.inplace_update_support: 0 2023-11-16T16:41:34.383 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.inplace_update_num_locks: 10000 2023-11-16T16:41:34.383 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-16T16:41:34.383 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-16T16:41:34.383 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.memtable_huge_page_size: 0 2023-11-16T16:41:34.383 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.bloom_locality: 0 2023-11-16T16:41:34.383 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.max_successive_merges: 0 2023-11-16T16:41:34.384 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.optimize_filters_for_hits: 0 2023-11-16T16:41:34.384 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.paranoid_file_checks: 0 2023-11-16T16:41:34.384 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.force_consistency_checks: 1 2023-11-16T16:41:34.384 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.report_bg_io_stats: 0 2023-11-16T16:41:34.384 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.ttl: 2592000 2023-11-16T16:41:34.384 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.periodic_compaction_seconds: 0 2023-11-16T16:41:34.384 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-11-16T16:41:34.384 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-11-16T16:41:34.384 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.enable_blob_files: false 2023-11-16T16:41:34.384 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.min_blob_size: 0 2023-11-16T16:41:34.384 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.blob_file_size: 268435456 2023-11-16T16:41:34.385 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.blob_compression_type: NoCompression 2023-11-16T16:41:34.385 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.enable_blob_garbage_collection: false 2023-11-16T16:41:34.385 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-11-16T16:41:34.385 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-11-16T16:41:34.385 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-11-16T16:41:34.385 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.blob_file_starting_level: 0 2023-11-16T16:41:34.385 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-11-16T16:41:34.385 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-16T16:41:34.385 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-11-16T16:41:34.385 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 54a766df-b80f-4994-af69-cc292e63b66d 2023-11-16T16:41:34.385 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700152894257639, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-11-16T16:41:34.385 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-11-16T16:41:34.386 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700152894258052, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1700152894, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "54a766df-b80f-4994-af69-cc292e63b66d", "db_session_id": "EYIBH9ZVN9477VYRMBNM", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-11-16T16:41:34.386 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: EVENT_LOG_v1 {"time_micros": 1700152894258117, "job": 1, "event": "recovery_finished"} 2023-11-16T16:41:34.386 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-11-16T16:41:34.386 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-16T16:41:34.386 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558aa23aa000 2023-11-16T16:41:34.386 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: DB pointer 0x558aa2396000 2023-11-16T16:41:34.386 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-11-16T16:41:34.386 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-11-16T16:41:34.386 INFO:journalctl@ceph.mon.b.smithi141.stdout: ** DB Stats ** 2023-11-16T16:41:34.386 INFO:journalctl@ceph.mon.b.smithi141.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T16:41:34.387 INFO:journalctl@ceph.mon.b.smithi141.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-16T16:41:34.387 INFO:journalctl@ceph.mon.b.smithi141.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T16:41:34.387 INFO:journalctl@ceph.mon.b.smithi141.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T16:41:34.387 INFO:journalctl@ceph.mon.b.smithi141.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-16T16:41:34.387 INFO:journalctl@ceph.mon.b.smithi141.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-16T16:41:34.387 INFO:journalctl@ceph.mon.b.smithi141.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-16T16:41:34.387 INFO:journalctl@ceph.mon.b.smithi141.stdout: 2023-11-16T16:41:34.387 INFO:journalctl@ceph.mon.b.smithi141.stdout: ** Compaction Stats [default] ** 2023-11-16T16:41:34.387 INFO:journalctl@ceph.mon.b.smithi141.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T16:41:34.387 INFO:journalctl@ceph.mon.b.smithi141.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-11-16T16:41:34.387 INFO:journalctl@ceph.mon.b.smithi141.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T16:41:34.388 INFO:journalctl@ceph.mon.b.smithi141.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T16:41:34.388 INFO:journalctl@ceph.mon.b.smithi141.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T16:41:34.388 INFO:journalctl@ceph.mon.b.smithi141.stdout: 2023-11-16T16:41:34.388 INFO:journalctl@ceph.mon.b.smithi141.stdout: ** Compaction Stats [default] ** 2023-11-16T16:41:34.388 INFO:journalctl@ceph.mon.b.smithi141.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-16T16:41:34.388 INFO:journalctl@ceph.mon.b.smithi141.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-16T16:41:34.388 INFO:journalctl@ceph.mon.b.smithi141.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-11-16T16:41:34.388 INFO:journalctl@ceph.mon.b.smithi141.stdout: 2023-11-16T16:41:34.388 INFO:journalctl@ceph.mon.b.smithi141.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-11-16T16:41:34.388 INFO:journalctl@ceph.mon.b.smithi141.stdout: 2023-11-16T16:41:34.389 INFO:journalctl@ceph.mon.b.smithi141.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-16T16:41:34.389 INFO:journalctl@ceph.mon.b.smithi141.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-11-16T16:41:34.389 INFO:journalctl@ceph.mon.b.smithi141.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-16T16:41:34.389 INFO:journalctl@ceph.mon.b.smithi141.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-11-16T16:41:34.389 INFO:journalctl@ceph.mon.b.smithi141.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-11-16T16:41:34.389 INFO:journalctl@ceph.mon.b.smithi141.stdout: AddFile(Keys): cumulative 0, interval 0 2023-11-16T16:41:34.389 INFO:journalctl@ceph.mon.b.smithi141.stdout: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T16:41:34.389 INFO:journalctl@ceph.mon.b.smithi141.stdout: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-16T16:41:34.389 INFO:journalctl@ceph.mon.b.smithi141.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-16T16:41:34.389 INFO:journalctl@ceph.mon.b.smithi141.stdout: Block cache BinnedLRUCache@0x558aa230d1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-11-16T16:41:34.390 INFO:journalctl@ceph.mon.b.smithi141.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-11-16T16:41:34.390 INFO:journalctl@ceph.mon.b.smithi141.stdout: 2023-11-16T16:41:34.390 INFO:journalctl@ceph.mon.b.smithi141.stdout: ** File Read Latency Histogram By Level [default] ** 2023-11-16T16:41:34.390 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-11-16T16:41:34.390 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: using public_addr v2:172.21.15.141:0/0 -> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] 2023-11-16T16:41:34.390 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: starting mon.b rank -1 at public addrs [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] at bind addrs [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:41:34.390 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mon.b@-1(???) e0 preinit fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:41:34.390 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mon.b@-1(synchronizing).mds e1 new map 2023-11-16T16:41:34.390 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mon.b@-1(synchronizing).mds e1 print_map 2023-11-16T16:41:34.390 INFO:journalctl@ceph.mon.b.smithi141.stdout: e1 2023-11-16T16:41:34.391 INFO:journalctl@ceph.mon.b.smithi141.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-11-16T16:41:34.391 INFO:journalctl@ceph.mon.b.smithi141.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T16:41:34.391 INFO:journalctl@ceph.mon.b.smithi141.stdout: legacy client fscid: -1 2023-11-16T16:41:34.391 INFO:journalctl@ceph.mon.b.smithi141.stdout: 2023-11-16T16:41:34.391 INFO:journalctl@ceph.mon.b.smithi141.stdout: No filesystems configured 2023-11-16T16:41:34.391 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-11-16T16:41:34.391 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-11-16T16:41:34.391 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-11-16T16:41:34.391 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-11-16T16:41:34.391 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-11-16T16:41:34.391 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-11-16T16:41:34.392 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-16T16:41:34.392 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T16:41:34.392 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T16:41:34.392 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-16T16:41:34.392 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.392 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:41:34.392 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:34.392 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.392 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.392 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.392 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.393 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi141", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:41:34.393 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:34.393 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:34.393 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:41:34.393 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: Updating smithi141:/etc/ceph/ceph.conf 2023-11-16T16:41:34.393 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:34.393 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: Updating smithi141:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:34.393 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.393 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.393 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.393 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: Updating smithi141:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:34.394 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: Updating smithi141:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:41:34.394 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:34.394 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: Deploying cephadm binary to smithi155 2023-11-16T16:41:34.394 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:34.394 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.394 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: Added host smithi155 2023-11-16T16:41:34.394 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:34.394 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.394 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:34.394 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.395 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.395 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:34.395 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:41:34.395 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:34.395 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.395 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.395 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.395 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.395 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi155", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:41:34.395 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:34.395 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:34.396 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:41:34.396 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: Updating smithi155:/etc/ceph/ceph.conf 2023-11-16T16:41:34.396 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: Updating smithi155:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:34.396 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:34.396 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: Updating smithi155:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:41:34.396 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.396 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.396 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.396 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1384182039' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-16T16:41:34.396 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:34.396 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1384182039' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-16T16:41:34.397 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:34.397 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:34.397 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.397 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:34.397 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:34.397 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:34.397 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.397 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:34.397 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:34.397 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi141:172.21.15.141=b;smithi155:172.21.15.155=c", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:34.397 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi141:172.21.15.141=b;smithi155:172.21.15.155=c;count:3 2023-11-16T16:41:34.398 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:41:34.398 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:41:34.398 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: Deploying daemon mon.c on smithi155 2023-11-16T16:41:34.398 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:34.398 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='client.? 172.21.15.155:0/4089927105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T16:41:34.398 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:41:34.398 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:34.398 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mon.a calling monitor election 2023-11-16T16:41:34.398 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:34.398 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:34.399 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:34.399 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mon.c calling monitor election 2023-11-16T16:41:34.399 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:34.399 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:34.399 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:34.399 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:34.399 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-11-16T16:41:34.399 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],c=[v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0]} removed_ranks: {} 2023-11-16T16:41:34.399 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: fsmap 2023-11-16T16:41:34.399 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:34.400 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mgrmap e13: x(active, since 47s) 2023-11-16T16:41:34.400 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: overall HEALTH_OK 2023-11-16T16:41:34.400 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.400 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.400 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:34.400 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:34.400 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:34.400 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: Deploying daemon mon.b on smithi141 2023-11-16T16:41:34.400 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:34.400 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:34.400 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: from='client.? 172.21.15.155:0/1265481410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T16:41:34.401 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-11-16T16:41:34.401 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-11-16T16:41:34.401 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: expand_channel_meta expand map: {default=false} 2023-11-16T16:41:34.401 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: expand_channel_meta from 'false' to 'false' 2023-11-16T16:41:34.401 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: expand_channel_meta expanded map: {default=false} 2023-11-16T16:41:34.401 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: expand_channel_meta expand map: {default=info} 2023-11-16T16:41:34.401 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: expand_channel_meta from 'info' to 'info' 2023-11-16T16:41:34.401 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: expand_channel_meta expanded map: {default=info} 2023-11-16T16:41:34.401 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: expand_channel_meta expand map: {default=daemon} 2023-11-16T16:41:34.401 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: expand_channel_meta from 'daemon' to 'daemon' 2023-11-16T16:41:34.401 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: expand_channel_meta expanded map: {default=daemon} 2023-11-16T16:41:34.402 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: expand_channel_meta expand map: {default=debug} 2023-11-16T16:41:34.402 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: expand_channel_meta from 'debug' to 'debug' 2023-11-16T16:41:34.402 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 ceph-mon[131184]: expand_channel_meta expanded map: {default=debug} 2023-11-16T16:41:34.402 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:34 smithi141 systemd[1]: Started Ceph mon.b for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:41:39.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: mon.c calling monitor election 2023-11-16T16:41:39.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:41:39.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: mon.a calling monitor election 2023-11-16T16:41:39.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:39.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:39.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:39.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:39.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:39.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:39.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:39.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:39.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:39.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-11-16T16:41:39.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0],c=[v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0]} removed_ranks: {} 2023-11-16T16:41:39.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: fsmap 2023-11-16T16:41:39.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:39.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: mgrmap e13: x(active, since 55s) 2023-11-16T16:41:39.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-11-16T16:41:39.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-11-16T16:41:39.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-11-16T16:41:39.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: mon.b (rank 2) addr [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] is down (out of quorum) 2023-11-16T16:41:39.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:39.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:39.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:39.621 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:39 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:39.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: mon.c calling monitor election 2023-11-16T16:41:39.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:41:39.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: mon.a calling monitor election 2023-11-16T16:41:39.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:39.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:39.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:39.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:39.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:39.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:39.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:39.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:39.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:39.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-11-16T16:41:39.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0],c=[v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0]} removed_ranks: {} 2023-11-16T16:41:39.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: fsmap 2023-11-16T16:41:39.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:39.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: mgrmap e13: x(active, since 55s) 2023-11-16T16:41:39.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-11-16T16:41:39.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-11-16T16:41:39.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-11-16T16:41:39.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: mon.b (rank 2) addr [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] is down (out of quorum) 2023-11-16T16:41:39.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:39.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:39.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:39.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:39 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:39.986 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.984+0000 7fbcf0467700 1 -- 172.21.15.155:0/2727578812 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcd40035c0 msgr2=0x7fbcd4005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:39.986 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.984+0000 7fbcf0467700 1 --2- 172.21.15.155:0/2727578812 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcd40035c0 0x7fbcd4005a40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbce80fec90 tx=0x7fbce400fe60 comp rx=0 tx=0).stop 2023-11-16T16:41:39.987 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.984+0000 7fbcf0467700 1 -- 172.21.15.155:0/2727578812 shutdown_connections 2023-11-16T16:41:39.987 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.984+0000 7fbcf0467700 1 --2- 172.21.15.155:0/2727578812 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcd40035c0 0x7fbcd4005a40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:39.987 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.984+0000 7fbcf0467700 1 -- 172.21.15.155:0/2727578812 >> 172.21.15.155:0/2727578812 conn(0x7fbce80fa8a0 msgr2=0x7fbce80fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:39.987 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.984+0000 7fbcf0467700 1 -- 172.21.15.155:0/2727578812 shutdown_connections 2023-11-16T16:41:39.988 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.984+0000 7fbcf0467700 1 -- 172.21.15.155:0/2727578812 wait complete. 2023-11-16T16:41:39.992 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.985+0000 7fbcf0467700 1 Processor -- start 2023-11-16T16:41:39.992 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbcf0467700 1 -- start start 2023-11-16T16:41:39.992 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbcf0467700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcd40035c0 0x7fbce8194310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:39.993 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbcf0467700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce8194850 0x7fbce8198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:39.993 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbcf0467700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbce8199200 0x7fbce81996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:39.993 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbcf0467700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce8194f20 con 0x7fbce8199200 2023-11-16T16:41:39.993 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbcf0467700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce81a0f70 con 0x7fbce8194850 2023-11-16T16:41:39.993 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbcf0467700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce81a10b0 con 0x7fbcd40035c0 2023-11-16T16:41:39.993 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbcee203700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcd40035c0 0x7fbce8194310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:39.993 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbcee203700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcd40035c0 0x7fbce8194310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:38978/0 (socket says 172.21.15.155:38978) 2023-11-16T16:41:39.993 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbcee203700 1 -- 172.21.15.155:0/3788952960 learned_addr learned my addr 172.21.15.155:0/3788952960 (peer_addr_for_me v2:172.21.15.155:0/0) 2023-11-16T16:41:39.994 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbcee203700 1 -- 172.21.15.155:0/3788952960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcd40035c0 msgr2=0x7fbce8194310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-11-16T16:41:39.994 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbcee203700 1 -- 172.21.15.155:0/3788952960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcd40035c0 msgr2=0x7fbce8194310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-16T16:41:39.994 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbcee203700 1 --2- 172.21.15.155:0/3788952960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcd40035c0 0x7fbce8194310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-16T16:41:39.994 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbceda02700 1 --2- 172.21.15.155:0/3788952960 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce8194850 0x7fbce8198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:39.994 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbcee203700 1 --2- 172.21.15.155:0/3788952960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcd40035c0 0x7fbce8194310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:41:39.994 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbceea04700 1 --2- 172.21.15.155:0/3788952960 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbce8199200 0x7fbce81996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:39.995 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbceda02700 1 -- 172.21.15.155:0/3788952960 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce8194850 msgr2=0x7fbce8198cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-11-16T16:41:39.995 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbceda02700 1 -- 172.21.15.155:0/3788952960 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce8194850 msgr2=0x7fbce8198cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-16T16:41:39.995 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbceda02700 1 --2- 172.21.15.155:0/3788952960 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce8194850 0x7fbce8198cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-16T16:41:39.995 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.986+0000 7fbceda02700 1 --2- 172.21.15.155:0/3788952960 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce8194850 0x7fbce8198cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:41:39.995 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.987+0000 7fbceea04700 1 -- 172.21.15.155:0/3788952960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcd40035c0 msgr2=0x7fbce8194310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:41:39.995 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.987+0000 7fbceea04700 1 --2- 172.21.15.155:0/3788952960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcd40035c0 0x7fbce8194310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:39.995 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.987+0000 7fbceea04700 1 -- 172.21.15.155:0/3788952960 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce8194850 msgr2=0x7fbce8198cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:41:39.996 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.987+0000 7fbceea04700 1 --2- 172.21.15.155:0/3788952960 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce8194850 0x7fbce8198cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:39.996 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.987+0000 7fbceea04700 1 -- 172.21.15.155:0/3788952960 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce400f8b0 con 0x7fbce8199200 2023-11-16T16:41:39.996 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.987+0000 7fbceea04700 1 --2- 172.21.15.155:0/3788952960 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbce8199200 0x7fbce81996a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbce000bd80 tx=0x7fbce000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:39.996 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.987+0000 7fbcdf7fe700 1 -- 172.21.15.155:0/3788952960 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce000cd10 con 0x7fbce8199200 2023-11-16T16:41:39.996 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.987+0000 7fbcf0467700 1 -- 172.21.15.155:0/3788952960 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbce8106b60 con 0x7fbce8199200 2023-11-16T16:41:39.996 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.987+0000 7fbcdf7fe700 1 -- 172.21.15.155:0/3788952960 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbce000ce70 con 0x7fbce8199200 2023-11-16T16:41:39.997 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.987+0000 7fbcdf7fe700 1 -- 172.21.15.155:0/3788952960 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce0012880 con 0x7fbce8199200 2023-11-16T16:41:39.997 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.987+0000 7fbcf0467700 1 -- 172.21.15.155:0/3788952960 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce81069d0 con 0x7fbce8199200 2023-11-16T16:41:39.997 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.988+0000 7fbcdf7fe700 1 -- 172.21.15.155:0/3788952960 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7fbce00129e0 con 0x7fbce8199200 2023-11-16T16:41:39.997 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.988+0000 7fbcdf7fe700 1 --2- 172.21.15.155:0/3788952960 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbccc038d40 0x7fbccc03b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:39.997 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.988+0000 7fbcdf7fe700 1 -- 172.21.15.155:0/3788952960 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbce004bd70 con 0x7fbce8199200 2023-11-16T16:41:39.997 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.989+0000 7fbcf0467700 1 -- 172.21.15.155:0/3788952960 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce8106cf0 con 0x7fbce8199200 2023-11-16T16:41:39.997 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.989+0000 7fbcee203700 1 --2- 172.21.15.155:0/3788952960 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbccc038d40 0x7fbccc03b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:39.998 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.990+0000 7fbcee203700 1 --2- 172.21.15.155:0/3788952960 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbccc038d40 0x7fbccc03b1f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbce4015040 tx=0x7fbce4001e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:39.998 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:39.994+0000 7fbcdf7fe700 1 -- 172.21.15.155:0/3788952960 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbce001d3b0 con 0x7fbce8199200 2023-11-16T16:41:40.274 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:40.272+0000 7fbcf0467700 1 -- 172.21.15.155:0/3788952960 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbce8195670 con 0x7fbce8199200 2023-11-16T16:41:40.275 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:40.273+0000 7fbcdf7fe700 1 -- 172.21.15.155:0/3788952960 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1270 (secure 0 0 0) 0x7fbce00112a0 con 0x7fbce8199200 2023-11-16T16:41:40.275 INFO:teuthology.orchestra.run.smithi155.stdout: 2023-11-16T16:41:40.275 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":3,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","modified":"2023-11-16T16:41:34.283879Z","created":"2023-11-16T16:39:39.719902Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:3300","nonce":0},{"type":"v1","addr":"172.21.15.141:6789","nonce":0}]},"addr":"172.21.15.141:6789/0","public_addr":"172.21.15.141:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-11-16T16:41:40.277 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:40.275+0000 7fbcf0467700 1 -- 172.21.15.155:0/3788952960 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbccc038d40 msgr2=0x7fbccc03b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:40.278 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:40.275+0000 7fbcf0467700 1 --2- 172.21.15.155:0/3788952960 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbccc038d40 0x7fbccc03b1f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbce4015040 tx=0x7fbce4001e50 comp rx=0 tx=0).stop 2023-11-16T16:41:40.278 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:40.275+0000 7fbcf0467700 1 -- 172.21.15.155:0/3788952960 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbce8199200 msgr2=0x7fbce81996a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:40.278 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:40.275+0000 7fbcf0467700 1 --2- 172.21.15.155:0/3788952960 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbce8199200 0x7fbce81996a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbce000bd80 tx=0x7fbce000be60 comp rx=0 tx=0).stop 2023-11-16T16:41:40.278 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:40.275+0000 7fbcf0467700 1 -- 172.21.15.155:0/3788952960 shutdown_connections 2023-11-16T16:41:40.278 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:40.275+0000 7fbcf0467700 1 --2- 172.21.15.155:0/3788952960 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbccc038d40 0x7fbccc03b1f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:40.279 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:40.275+0000 7fbcf0467700 1 --2- 172.21.15.155:0/3788952960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcd40035c0 0x7fbce8194310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:40.279 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:40.275+0000 7fbcf0467700 1 --2- 172.21.15.155:0/3788952960 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbce8199200 0x7fbce81996a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:40.279 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:40.275+0000 7fbcf0467700 1 --2- 172.21.15.155:0/3788952960 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbce8194850 0x7fbce8198cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:40.279 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:40.275+0000 7fbcf0467700 1 -- 172.21.15.155:0/3788952960 >> 172.21.15.155:0/3788952960 conn(0x7fbce80fa8a0 msgr2=0x7fbce818c160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:40.279 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:40.275+0000 7fbcf0467700 1 -- 172.21.15.155:0/3788952960 shutdown_connections 2023-11-16T16:41:40.279 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:40.275+0000 7fbcf0467700 1 -- 172.21.15.155:0/3788952960 wait complete. 2023-11-16T16:41:40.279 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 3 2023-11-16T16:41:40.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:40 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:40.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:40 smithi155 ceph-mon[131607]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:40.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:40 smithi155 ceph-mon[131607]: from='client.? 172.21.15.155:0/3788952960' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T16:41:40.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:40 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:40.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:40 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:40.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:40 smithi002 ceph-mon[128712]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:40.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:40 smithi002 ceph-mon[128712]: from='client.? 172.21.15.155:0/3788952960' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T16:41:40.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:40 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:40.729 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-11-16T16:41:40.729 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph config generate-minimal-conf 2023-11-16T16:41:41.633 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:41:41.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: mon.c calling monitor election 2023-11-16T16:41:41.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:41:41.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: mon.a calling monitor election 2023-11-16T16:41:41.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:41.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:41:41.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:41.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:41.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:41.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:41.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:41.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:41.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:41.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-11-16T16:41:41.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0],c=[v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0]} removed_ranks: {} 2023-11-16T16:41:41.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: fsmap 2023-11-16T16:41:41.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:41.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: mgrmap e13: x(active, since 55s) 2023-11-16T16:41:41.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-11-16T16:41:41.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-11-16T16:41:41.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-11-16T16:41:41.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: mon.b (rank 2) addr [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] is down (out of quorum) 2023-11-16T16:41:41.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:41.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:41.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:41.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:41.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:41.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:41.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: from='client.? 172.21.15.155:0/3788952960' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-16T16:41:41.768 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: mon.b calling monitor election 2023-11-16T16:41:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:41:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: Updating smithi002:/etc/ceph/ceph.conf 2023-11-16T16:41:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: Updating smithi141:/etc/ceph/ceph.conf 2023-11-16T16:41:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: Updating smithi155:/etc/ceph/ceph.conf 2023-11-16T16:41:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: mon.b calling monitor election 2023-11-16T16:41:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: mon.c calling monitor election 2023-11-16T16:41:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: mon.a calling monitor election 2023-11-16T16:41:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-11-16T16:41:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: Updating smithi141:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: Updating smithi155:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0],c=[v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0]} removed_ranks: {} 2023-11-16T16:41:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: fsmap 2023-11-16T16:41:42.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:42.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: mgrmap e13: x(active, since 57s) 2023-11-16T16:41:42.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-11-16T16:41:42.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: Cluster is now healthy 2023-11-16T16:41:42.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: overall HEALTH_OK 2023-11-16T16:41:42.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:42.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:42.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:42.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:42.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: Updating smithi002:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:42.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:42 smithi002 ceph-mon[128712]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:42.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: mon.b calling monitor election 2023-11-16T16:41:42.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:41:42.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: Updating smithi002:/etc/ceph/ceph.conf 2023-11-16T16:41:42.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: Updating smithi141:/etc/ceph/ceph.conf 2023-11-16T16:41:42.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: Updating smithi155:/etc/ceph/ceph.conf 2023-11-16T16:41:42.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: mon.b calling monitor election 2023-11-16T16:41:42.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: mon.c calling monitor election 2023-11-16T16:41:42.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: mon.a calling monitor election 2023-11-16T16:41:42.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-11-16T16:41:42.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: Updating smithi141:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:42.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: Updating smithi155:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:42.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0],c=[v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0]} removed_ranks: {} 2023-11-16T16:41:42.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: fsmap 2023-11-16T16:41:42.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:42.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: mgrmap e13: x(active, since 57s) 2023-11-16T16:41:42.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-11-16T16:41:42.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: Cluster is now healthy 2023-11-16T16:41:42.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: overall HEALTH_OK 2023-11-16T16:41:42.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:42.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:42.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:42.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:42.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: Updating smithi002:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:42.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:42 smithi141 ceph-mon[131184]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: mon.b calling monitor election 2023-11-16T16:41:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:41:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: Updating smithi002:/etc/ceph/ceph.conf 2023-11-16T16:41:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: Updating smithi141:/etc/ceph/ceph.conf 2023-11-16T16:41:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: Updating smithi155:/etc/ceph/ceph.conf 2023-11-16T16:41:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: mon.b calling monitor election 2023-11-16T16:41:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: mon.c calling monitor election 2023-11-16T16:41:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: mon.a calling monitor election 2023-11-16T16:41:42.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-11-16T16:41:42.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: Updating smithi141:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:42.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: Updating smithi155:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:42.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0],c=[v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0]} removed_ranks: {} 2023-11-16T16:41:42.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: fsmap 2023-11-16T16:41:42.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: osdmap e4: 0 total, 0 up, 0 in 2023-11-16T16:41:42.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: mgrmap e13: x(active, since 57s) 2023-11-16T16:41:42.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-11-16T16:41:42.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: Cluster is now healthy 2023-11-16T16:41:42.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: overall HEALTH_OK 2023-11-16T16:41:42.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:42.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:42.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:42.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:42.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: Updating smithi002:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:41:42.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:42 smithi155 ceph-mon[131607]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:43.180 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.178+0000 7fd883fed700 1 -- 172.21.15.2:0/2375153655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd87c072e90 msgr2=0x7fd87c073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:43.180 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.178+0000 7fd883fed700 1 --2- 172.21.15.2:0/2375153655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd87c072e90 0x7fd87c073260 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd878009910 tx=0x7fd878009c20 comp rx=0 tx=0).stop 2023-11-16T16:41:43.180 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.178+0000 7fd883fed700 1 -- 172.21.15.2:0/2375153655 shutdown_connections 2023-11-16T16:41:43.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.178+0000 7fd883fed700 1 --2- 172.21.15.2:0/2375153655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd87c072e90 0x7fd87c073260 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:43.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.178+0000 7fd883fed700 1 -- 172.21.15.2:0/2375153655 >> 172.21.15.2:0/2375153655 conn(0x7fd87c0fb850 msgr2=0x7fd87c0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:43.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.178+0000 7fd883fed700 1 -- 172.21.15.2:0/2375153655 shutdown_connections 2023-11-16T16:41:43.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.179+0000 7fd883fed700 1 -- 172.21.15.2:0/2375153655 wait complete. 2023-11-16T16:41:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.180+0000 7fd883fed700 1 Processor -- start 2023-11-16T16:41:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.180+0000 7fd883fed700 1 -- start start 2023-11-16T16:41:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.181+0000 7fd883fed700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd87c072e90 0x7fd87c19b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.181+0000 7fd883fed700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd87c19bec0 0x7fd87c1a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.181+0000 7fd883fed700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd87c1a08a0 0x7fd87c1a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:43.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.181+0000 7fd883fed700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c19c560 con 0x7fd87c1a08a0 2023-11-16T16:41:43.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.181+0000 7fd883fed700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c1a8610 con 0x7fd87c19bec0 2023-11-16T16:41:43.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.181+0000 7fd883fed700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c1a8750 con 0x7fd87c072e90 2023-11-16T16:41:43.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.181+0000 7fd88258a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd87c1a08a0 0x7fd87c1a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:43.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.181+0000 7fd88258a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd87c1a08a0 0x7fd87c1a0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33388/0 (socket says 172.21.15.2:33388) 2023-11-16T16:41:43.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.181+0000 7fd88258a700 1 -- 172.21.15.2:0/2513653839 learned_addr learned my addr 172.21.15.2:0/2513653839 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:41:43.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.181+0000 7fd881d89700 1 --2- 172.21.15.2:0/2513653839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd87c072e90 0x7fd87c19b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:43.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.181+0000 7fd881588700 1 --2- 172.21.15.2:0/2513653839 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd87c19bec0 0x7fd87c1a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:43.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.182+0000 7fd88258a700 1 -- 172.21.15.2:0/2513653839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd87c072e90 msgr2=0x7fd87c19b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:43.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.182+0000 7fd88258a700 1 --2- 172.21.15.2:0/2513653839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd87c072e90 0x7fd87c19b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:43.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.182+0000 7fd88258a700 1 -- 172.21.15.2:0/2513653839 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd87c19bec0 msgr2=0x7fd87c1a0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:43.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.182+0000 7fd88258a700 1 --2- 172.21.15.2:0/2513653839 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd87c19bec0 0x7fd87c1a0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:43.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.182+0000 7fd88258a700 1 -- 172.21.15.2:0/2513653839 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8780095f0 con 0x7fd87c1a08a0 2023-11-16T16:41:43.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.182+0000 7fd88258a700 1 --2- 172.21.15.2:0/2513653839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd87c1a08a0 0x7fd87c1a0d10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd87400ba50 tx=0x7fd87400bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:43.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.182+0000 7fd872ffd700 1 -- 172.21.15.2:0/2513653839 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87400c930 con 0x7fd87c1a08a0 2023-11-16T16:41:43.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.182+0000 7fd872ffd700 1 -- 172.21.15.2:0/2513653839 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd874014440 con 0x7fd87c1a08a0 2023-11-16T16:41:43.187 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.182+0000 7fd883fed700 1 -- 172.21.15.2:0/2513653839 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd87c110780 con 0x7fd87c1a08a0 2023-11-16T16:41:43.187 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.182+0000 7fd872ffd700 1 -- 172.21.15.2:0/2513653839 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8740075b0 con 0x7fd87c1a08a0 2023-11-16T16:41:43.187 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.183+0000 7fd883fed700 1 -- 172.21.15.2:0/2513653839 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd87c1105f0 con 0x7fd87c1a08a0 2023-11-16T16:41:43.188 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.184+0000 7fd883fed700 1 -- 172.21.15.2:0/2513653839 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd87c110910 con 0x7fd87c1a08a0 2023-11-16T16:41:43.188 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.184+0000 7fd872ffd700 1 -- 172.21.15.2:0/2513653839 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7fd87400ca90 con 0x7fd87c1a08a0 2023-11-16T16:41:43.188 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.184+0000 7fd872ffd700 1 --2- 172.21.15.2:0/2513653839 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd868038d40 0x7fd86803b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:43.188 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.184+0000 7fd872ffd700 1 -- 172.21.15.2:0/2513653839 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd87404a910 con 0x7fd87c1a08a0 2023-11-16T16:41:43.188 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.185+0000 7fd881d89700 1 --2- 172.21.15.2:0/2513653839 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd868038d40 0x7fd86803b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:43.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.185+0000 7fd881d89700 1 --2- 172.21.15.2:0/2513653839 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd868038d40 0x7fd86803b1f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd878009fd0 tx=0x7fd8780058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:43.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.190+0000 7fd872ffd700 1 -- 172.21.15.2:0/2513653839 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd8740148b0 con 0x7fd87c1a08a0 2023-11-16T16:41:43.409 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.408+0000 7fd883fed700 1 -- 172.21.15.2:0/2513653839 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd87c1a9230 con 0x7fd87c1a08a0 2023-11-16T16:41:43.410 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.409+0000 7fd872ffd700 1 -- 172.21.15.2:0/2513653839 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+273 (secure 0 0 0) 0x7fd874012340 con 0x7fd87c1a08a0 2023-11-16T16:41:43.410 INFO:teuthology.orchestra.run.smithi002.stdout:# minimal ceph.conf for 875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:41:43.410 INFO:teuthology.orchestra.run.smithi002.stdout:[global] 2023-11-16T16:41:43.410 INFO:teuthology.orchestra.run.smithi002.stdout: fsid = 875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:41:43.411 INFO:teuthology.orchestra.run.smithi002.stdout: mon_host = [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2023-11-16T16:41:43.413 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.411+0000 7fd883fed700 1 -- 172.21.15.2:0/2513653839 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd868038d40 msgr2=0x7fd86803b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:43.413 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.411+0000 7fd883fed700 1 --2- 172.21.15.2:0/2513653839 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd868038d40 0x7fd86803b1f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd878009fd0 tx=0x7fd8780058e0 comp rx=0 tx=0).stop 2023-11-16T16:41:43.413 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.411+0000 7fd883fed700 1 -- 172.21.15.2:0/2513653839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd87c1a08a0 msgr2=0x7fd87c1a0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:43.413 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.411+0000 7fd883fed700 1 --2- 172.21.15.2:0/2513653839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd87c1a08a0 0x7fd87c1a0d10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd87400ba50 tx=0x7fd87400bd60 comp rx=0 tx=0).stop 2023-11-16T16:41:43.413 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.411+0000 7fd883fed700 1 -- 172.21.15.2:0/2513653839 shutdown_connections 2023-11-16T16:41:43.413 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.411+0000 7fd883fed700 1 --2- 172.21.15.2:0/2513653839 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd868038d40 0x7fd86803b1f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:43.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.411+0000 7fd883fed700 1 --2- 172.21.15.2:0/2513653839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd87c072e90 0x7fd87c19b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:43.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.411+0000 7fd883fed700 1 --2- 172.21.15.2:0/2513653839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd87c1a08a0 0x7fd87c1a0d10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:43.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.411+0000 7fd883fed700 1 --2- 172.21.15.2:0/2513653839 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd87c19bec0 0x7fd87c1a0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:43.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.411+0000 7fd883fed700 1 -- 172.21.15.2:0/2513653839 >> 172.21.15.2:0/2513653839 conn(0x7fd87c0fb850 msgr2=0x7fd87c0fbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:43.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.411+0000 7fd883fed700 1 -- 172.21.15.2:0/2513653839 shutdown_connections 2023-11-16T16:41:43.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:43.411+0000 7fd883fed700 1 -- 172.21.15.2:0/2513653839 wait complete. 2023-11-16T16:41:43.465 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:43.465 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:43.465 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:43.465 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:43 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:41:43.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:43.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:43 smithi002 ceph-mon[128712]: Reconfiguring mon.a (unknown last config time)... 2023-11-16T16:41:43.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:43.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T16:41:43.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:43.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:43 smithi002 ceph-mon[128712]: Reconfiguring daemon mon.a on smithi002 2023-11-16T16:41:43.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:43.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:43.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:43.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:43 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:41:43.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:43.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:43 smithi141 ceph-mon[131184]: Reconfiguring mon.a (unknown last config time)... 2023-11-16T16:41:43.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:43.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T16:41:43.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:43.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:43 smithi141 ceph-mon[131184]: Reconfiguring daemon mon.a on smithi002 2023-11-16T16:41:43.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:43.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:43.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:43.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:43 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:41:43.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:41:43.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:43 smithi155 ceph-mon[131607]: Reconfiguring mon.a (unknown last config time)... 2023-11-16T16:41:43.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:43.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T16:41:43.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:43.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:43 smithi155 ceph-mon[131607]: Reconfiguring daemon mon.a on smithi002 2023-11-16T16:41:44.036 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-11-16T16:41:44.037 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T16:41:44.037 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.conf 2023-11-16T16:41:44.113 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T16:41:44.113 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:44.226 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-11-16T16:41:44.226 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/etc/ceph/ceph.conf 2023-11-16T16:41:44.263 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-11-16T16:41:44.263 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:44.333 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-11-16T16:41:44.334 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.conf 2023-11-16T16:41:44.371 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-11-16T16:41:44.371 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:41:44.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:44 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2513653839' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:44.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:44 smithi002 ceph-mon[128712]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:44.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:44.442 INFO:tasks.cephadm:Adding mgr.x on smithi002 2023-11-16T16:41:44.442 INFO:tasks.cephadm:Adding mgr.y on smithi141 2023-11-16T16:41:44.442 INFO:tasks.cephadm:Adding mgr.z on smithi155 2023-11-16T16:41:44.442 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch apply mgr '3;smithi002=x;smithi141=y;smithi155=z' 2023-11-16T16:41:44.509 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:44 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2513653839' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:44.509 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:44 smithi155 ceph-mon[131607]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:44.509 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:44.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:44 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2513653839' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:44.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:44 smithi141 ceph-mon[131184]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:44.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:45.123 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:41:45.811 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:45 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:45.811 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:45 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:45.811 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:45 smithi141 ceph-mon[131184]: Reconfiguring mon.b (monmap changed)... 2023-11-16T16:41:45.811 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:45 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:45.811 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:45 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T16:41:45.811 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:45 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:45.812 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:45 smithi141 ceph-mon[131184]: Reconfiguring daemon mon.b on smithi141 2023-11-16T16:41:45.863 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:45 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:45.864 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:45 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:45.864 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:45 smithi155 ceph-mon[131607]: Reconfiguring mon.b (monmap changed)... 2023-11-16T16:41:45.864 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:45 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:45.864 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:45 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T16:41:45.864 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:45 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:45.864 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:45 smithi155 ceph-mon[131607]: Reconfiguring daemon mon.b on smithi141 2023-11-16T16:41:45.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:45 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:45.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:45 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:45.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:45 smithi002 ceph-mon[128712]: Reconfiguring mon.b (monmap changed)... 2023-11-16T16:41:45.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:45 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:45.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:45 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T16:41:45.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:45 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:45.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:45 smithi002 ceph-mon[128712]: Reconfiguring daemon mon.b on smithi141 2023-11-16T16:41:46.336 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.331+0000 7f4f80a86700 1 -- 172.21.15.155:0/2668357229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f7c0ff260 msgr2=0x7f4f7c0ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:46.336 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.331+0000 7f4f80a86700 1 --2- 172.21.15.155:0/2668357229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f7c0ff260 0x7f4f7c0ff670 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4f70009910 tx=0x7f4f70009c20 comp rx=0 tx=0).stop 2023-11-16T16:41:46.336 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.331+0000 7f4f80a86700 1 -- 172.21.15.155:0/2668357229 shutdown_connections 2023-11-16T16:41:46.336 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.331+0000 7f4f80a86700 1 --2- 172.21.15.155:0/2668357229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f7c0ff260 0x7f4f7c0ff670 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:46.336 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.331+0000 7f4f80a86700 1 -- 172.21.15.155:0/2668357229 >> 172.21.15.155:0/2668357229 conn(0x7f4f7c0f9a70 msgr2=0x7f4f7c0fbea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:46.337 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.331+0000 7f4f80a86700 1 -- 172.21.15.155:0/2668357229 shutdown_connections 2023-11-16T16:41:46.337 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.331+0000 7f4f80a86700 1 -- 172.21.15.155:0/2668357229 wait complete. 2023-11-16T16:41:46.337 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.331+0000 7f4f80a86700 1 Processor -- start 2023-11-16T16:41:46.337 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.331+0000 7f4f80a86700 1 -- start start 2023-11-16T16:41:46.337 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f80a86700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f7c0ff260 0x7f4f7c0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:46.337 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f80a86700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4f7c197e10 0x7f4f7c19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:46.337 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f80a86700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f7c19a780 0x7f4f7c19cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:46.337 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f80a86700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c19d3a0 con 0x7f4f7c19a780 2023-11-16T16:41:46.338 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f80a86700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c19d510 con 0x7f4f7c197e10 2023-11-16T16:41:46.338 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f80a86700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c19d6d0 con 0x7f4f7c0ff260 2023-11-16T16:41:46.338 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f7a59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f7c0ff260 0x7f4f7c0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:46.338 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f7a59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f7c0ff260 0x7f4f7c0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:38988/0 (socket says 172.21.15.155:38988) 2023-11-16T16:41:46.338 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f7a59c700 1 -- 172.21.15.155:0/464627692 learned_addr learned my addr 172.21.15.155:0/464627692 (peer_addr_for_me v2:172.21.15.155:0/0) 2023-11-16T16:41:46.338 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f79d9b700 1 --2- 172.21.15.155:0/464627692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4f7c197e10 0x7f4f7c19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:46.339 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f7ad9d700 1 --2- 172.21.15.155:0/464627692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f7c19a780 0x7f4f7c19cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:46.339 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f7a59c700 1 -- 172.21.15.155:0/464627692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4f7c197e10 msgr2=0x7f4f7c19a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:46.339 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f7a59c700 1 --2- 172.21.15.155:0/464627692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4f7c197e10 0x7f4f7c19a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:46.339 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f7a59c700 1 -- 172.21.15.155:0/464627692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f7c19a780 msgr2=0x7f4f7c19cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:46.339 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f7a59c700 1 --2- 172.21.15.155:0/464627692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f7c19a780 0x7f4f7c19cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:46.339 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f7a59c700 1 -- 172.21.15.155:0/464627692 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f700095f0 con 0x7f4f7c0ff260 2023-11-16T16:41:46.340 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f7a59c700 1 --2- 172.21.15.155:0/464627692 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f7c0ff260 0x7f4f7c0fe920 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4f70000c00 tx=0x7f4f70004cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:46.340 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f6b7fe700 1 -- 172.21.15.155:0/464627692 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7001d070 con 0x7f4f7c0ff260 2023-11-16T16:41:46.340 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f80a86700 1 -- 172.21.15.155:0/464627692 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f7c104bc0 con 0x7f4f7c0ff260 2023-11-16T16:41:46.340 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.332+0000 7f4f6b7fe700 1 -- 172.21.15.155:0/464627692 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f7000be20 con 0x7f4f7c0ff260 2023-11-16T16:41:46.340 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.333+0000 7f4f6b7fe700 1 -- 172.21.15.155:0/464627692 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f70022de0 con 0x7f4f7c0ff260 2023-11-16T16:41:46.340 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.333+0000 7f4f80a86700 1 -- 172.21.15.155:0/464627692 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7c104a30 con 0x7f4f7c0ff260 2023-11-16T16:41:46.340 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.333+0000 7f4f6b7fe700 1 -- 172.21.15.155:0/464627692 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f4f7000f460 con 0x7f4f7c0ff260 2023-11-16T16:41:46.340 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.333+0000 7f4f80a86700 1 -- 172.21.15.155:0/464627692 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f7c18d8f0 con 0x7f4f7c0ff260 2023-11-16T16:41:46.341 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.333+0000 7f4f6b7fe700 1 --2- 172.21.15.155:0/464627692 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f4f64038cf0 0x7f4f6403b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:46.341 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.334+0000 7f4f6b7fe700 1 -- 172.21.15.155:0/464627692 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4f7004c770 con 0x7f4f7c0ff260 2023-11-16T16:41:46.341 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.334+0000 7f4f79d9b700 1 --2- 172.21.15.155:0/464627692 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f4f64038cf0 0x7f4f6403b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:46.341 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.334+0000 7f4f79d9b700 1 --2- 172.21.15.155:0/464627692 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f4f64038cf0 0x7f4f6403b1a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c0feea0 tx=0x7f4f6c009210 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:46.342 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.339+0000 7f4f6b7fe700 1 -- 172.21.15.155:0/464627692 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4f7002c440 con 0x7f4f7c0ff260 2023-11-16T16:41:46.548 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.547+0000 7f4f80a86700 1 -- 172.21.15.155:0/464627692 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi141=y;smithi155=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f7c061bf0 con 0x7f4f64038cf0 2023-11-16T16:41:46.555 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.554+0000 7f4f6b7fe700 1 -- 172.21.15.155:0/464627692 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4f7c061bf0 con 0x7f4f64038cf0 2023-11-16T16:41:46.556 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled mgr update... 2023-11-16T16:41:46.558 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.556+0000 7f4f80a86700 1 -- 172.21.15.155:0/464627692 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f4f64038cf0 msgr2=0x7f4f6403b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:46.558 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.556+0000 7f4f80a86700 1 --2- 172.21.15.155:0/464627692 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f4f64038cf0 0x7f4f6403b1a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c0feea0 tx=0x7f4f6c009210 comp rx=0 tx=0).stop 2023-11-16T16:41:46.559 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.556+0000 7f4f80a86700 1 -- 172.21.15.155:0/464627692 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f7c0ff260 msgr2=0x7f4f7c0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:46.559 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.556+0000 7f4f80a86700 1 --2- 172.21.15.155:0/464627692 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f7c0ff260 0x7f4f7c0fe920 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4f70000c00 tx=0x7f4f70004cb0 comp rx=0 tx=0).stop 2023-11-16T16:41:46.559 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.556+0000 7f4f80a86700 1 -- 172.21.15.155:0/464627692 shutdown_connections 2023-11-16T16:41:46.559 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.556+0000 7f4f80a86700 1 --2- 172.21.15.155:0/464627692 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f4f64038cf0 0x7f4f6403b1a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:46.559 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.556+0000 7f4f80a86700 1 --2- 172.21.15.155:0/464627692 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f7c0ff260 0x7f4f7c0fe920 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:46.559 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.556+0000 7f4f80a86700 1 --2- 172.21.15.155:0/464627692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f7c19a780 0x7f4f7c19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:46.560 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.556+0000 7f4f80a86700 1 --2- 172.21.15.155:0/464627692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4f7c197e10 0x7f4f7c19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:46.560 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.556+0000 7f4f80a86700 1 -- 172.21.15.155:0/464627692 >> 172.21.15.155:0/464627692 conn(0x7f4f7c0f9a70 msgr2=0x7f4f7c0fa720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:46.560 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.556+0000 7f4f80a86700 1 -- 172.21.15.155:0/464627692 shutdown_connections 2023-11-16T16:41:46.560 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:41:46.556+0000 7f4f80a86700 1 -- 172.21.15.155:0/464627692 wait complete. 2023-11-16T16:41:47.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:46 smithi155 ceph-mon[131607]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:47.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:46 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:47.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:46 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:47.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:46 smithi155 ceph-mon[131607]: Reconfiguring mon.c (monmap changed)... 2023-11-16T16:41:47.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:46 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:47.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:46 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T16:41:47.120 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:46 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:47.120 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:46 smithi155 ceph-mon[131607]: Reconfiguring daemon mon.c on smithi155 2023-11-16T16:41:47.120 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:46 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:47.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:46 smithi002 ceph-mon[128712]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:47.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:46 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:47.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:46 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:47.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:46 smithi002 ceph-mon[128712]: Reconfiguring mon.c (monmap changed)... 2023-11-16T16:41:47.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:46 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:47.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:46 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T16:41:47.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:46 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:47.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:46 smithi002 ceph-mon[128712]: Reconfiguring daemon mon.c on smithi155 2023-11-16T16:41:47.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:46 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:47.196 DEBUG:teuthology.orchestra.run.smithi141:mgr.y> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mgr.y.service 2023-11-16T16:41:47.200 DEBUG:teuthology.orchestra.run.smithi155:mgr.z> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mgr.z.service 2023-11-16T16:41:47.228 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:46 smithi141 ceph-mon[131184]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:47.228 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:46 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:47.228 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:46 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:47.228 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:46 smithi141 ceph-mon[131184]: Reconfiguring mon.c (monmap changed)... 2023-11-16T16:41:47.229 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:46 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-16T16:41:47.229 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:46 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-16T16:41:47.229 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:46 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:47.229 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:46 smithi141 ceph-mon[131184]: Reconfiguring daemon mon.c on smithi155 2023-11-16T16:41:47.230 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:46 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:47.234 INFO:journalctl@ceph.mgr.y.smithi141.stdout:-- Logs begin at Thu 2023-11-16 16:14:35 UTC. -- 2023-11-16T16:41:47.240 INFO:tasks.cephadm:Deploying OSDs... 2023-11-16T16:41:47.240 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T16:41:47.240 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/scratch_devs of=/dev/stdout 2023-11-16T16:41:47.270 INFO:journalctl@ceph.mgr.z.smithi155.stdout:-- Logs begin at Thu 2023-11-16 16:15:48 UTC. -- 2023-11-16T16:41:47.291 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-16T16:41:47.291 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_1 2023-11-16T16:41:47.381 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-16T16:41:47.381 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:47.381 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 452 Links: 1 2023-11-16T16:41:47.381 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:47.382 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:47.382 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-11-16 16:40:53.462148916 +0000 2023-11-16T16:41:47.382 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-11-16 16:34:05.582077186 +0000 2023-11-16T16:41:47.382 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-11-16 16:34:05.582077186 +0000 2023-11-16T16:41:47.382 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2023-11-16 16:34:05.557077731 +0000 2023-11-16T16:41:47.382 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-16T16:41:47.452 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-11-16T16:41:47.452 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-11-16T16:41:47.452 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000197355 s, 2.6 MB/s 2023-11-16T16:41:47.455 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-16T16:41:47.502 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_2 2023-11-16T16:41:47.587 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-16T16:41:47.587 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:47.587 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 458 Links: 1 2023-11-16T16:41:47.588 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:47.588 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:47.588 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-11-16 16:40:53.462148916 +0000 2023-11-16T16:41:47.588 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-11-16 16:34:06.404059266 +0000 2023-11-16T16:41:47.588 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-11-16 16:34:06.404059266 +0000 2023-11-16T16:41:47.588 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2023-11-16 16:34:06.381059767 +0000 2023-11-16T16:41:47.588 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-16T16:41:47.695 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-11-16T16:41:47.695 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-11-16T16:41:47.695 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.0001923 s, 2.7 MB/s 2023-11-16T16:41:47.698 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-16T16:41:47.745 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_3 2023-11-16T16:41:47.832 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-16T16:41:47.832 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:47.833 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 464 Links: 1 2023-11-16T16:41:47.833 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:47.833 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:47.833 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-11-16 16:40:53.462148916 +0000 2023-11-16T16:41:47.833 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-11-16 16:34:07.232041215 +0000 2023-11-16T16:41:47.833 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-11-16 16:34:07.232041215 +0000 2023-11-16T16:41:47.833 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2023-11-16 16:34:07.209041716 +0000 2023-11-16T16:41:47.833 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-16T16:41:47.942 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-11-16T16:41:47.942 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-11-16T16:41:47.942 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000191947 s, 2.7 MB/s 2023-11-16T16:41:47.945 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-16T16:41:47.998 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_4 2023-11-16T16:41:48.047 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-16T16:41:48.048 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:48.048 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 470 Links: 1 2023-11-16T16:41:48.048 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:48.048 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:48.048 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-11-16 16:40:53.462148916 +0000 2023-11-16T16:41:48.048 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-11-16 16:34:08.028023862 +0000 2023-11-16T16:41:48.048 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-11-16 16:34:08.028023862 +0000 2023-11-16T16:41:48.049 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2023-11-16 16:34:08.003024407 +0000 2023-11-16T16:41:48.049 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-16T16:41:48.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:47 smithi002 ceph-mon[128712]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi141=y;smithi155=z", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:48.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:47 smithi002 ceph-mon[128712]: Saving service mgr spec with placement smithi002=x;smithi141=y;smithi155=z;count:3 2023-11-16T16:41:48.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:47 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:48.109 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:47 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:48.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:47 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:48.116 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2023-11-16T16:41:48.116 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2023-11-16T16:41:48.116 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000174138 s, 2.9 MB/s 2023-11-16T16:41:48.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:47 smithi155 ceph-mon[131607]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi141=y;smithi155=z", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:48.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:47 smithi155 ceph-mon[131607]: Saving service mgr spec with placement smithi002=x;smithi141=y;smithi155=z;count:3 2023-11-16T16:41:48.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:47 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:48.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:47 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:48.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:47 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:48.119 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-16T16:41:48.170 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-11-16T16:41:48.170 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/scratch_devs of=/dev/stdout 2023-11-16T16:41:48.191 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-16T16:41:48.191 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_1 2023-11-16T16:41:48.251 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-16T16:41:48.251 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:48.251 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 387562 Links: 1 2023-11-16T16:41:48.251 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:48.252 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:48.252 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2023-11-16 16:41:33.726837826 +0000 2023-11-16T16:41:48.252 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2023-11-16 16:30:44.705737817 +0000 2023-11-16T16:41:48.252 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2023-11-16 16:30:44.705737817 +0000 2023-11-16T16:41:48.252 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2023-11-16T16:41:48.252 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-16T16:41:48.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:47 smithi141 ceph-mon[131184]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi141=y;smithi155=z", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:48.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:47 smithi141 ceph-mon[131184]: Saving service mgr spec with placement smithi002=x;smithi141=y;smithi155=z;count:3 2023-11-16T16:41:48.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:47 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:48.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:47 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:48.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:47 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:48.294 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2023-11-16T16:41:48.294 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2023-11-16T16:41:48.294 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000277177 s, 1.8 MB/s 2023-11-16T16:41:48.295 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-16T16:41:48.355 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_2 2023-11-16T16:41:48.413 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-16T16:41:48.413 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:48.413 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 388726 Links: 1 2023-11-16T16:41:48.413 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:48.413 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:48.413 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2023-11-16 16:41:33.726837826 +0000 2023-11-16T16:41:48.413 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2023-11-16 16:30:45.190728001 +0000 2023-11-16T16:41:48.414 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2023-11-16 16:30:45.190728001 +0000 2023-11-16T16:41:48.414 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2023-11-16T16:41:48.414 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-16T16:41:48.480 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2023-11-16T16:41:48.480 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2023-11-16T16:41:48.481 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000233088 s, 2.2 MB/s 2023-11-16T16:41:48.482 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-16T16:41:48.541 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_3 2023-11-16T16:41:48.599 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-16T16:41:48.599 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:48.599 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 395301 Links: 1 2023-11-16T16:41:48.599 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:48.599 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:48.599 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2023-11-16 16:41:33.727837806 +0000 2023-11-16T16:41:48.599 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2023-11-16 16:30:45.691717861 +0000 2023-11-16T16:41:48.600 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2023-11-16 16:30:45.691717861 +0000 2023-11-16T16:41:48.600 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2023-11-16T16:41:48.600 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-16T16:41:48.667 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2023-11-16T16:41:48.667 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2023-11-16T16:41:48.667 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.00021705 s, 2.4 MB/s 2023-11-16T16:41:48.668 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-16T16:41:48.727 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_4 2023-11-16T16:41:48.784 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-16T16:41:48.785 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:48.785 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 393363 Links: 1 2023-11-16T16:41:48.785 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:48.785 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:48.785 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2023-11-16 16:41:33.727837806 +0000 2023-11-16T16:41:48.785 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2023-11-16 16:30:46.183707904 +0000 2023-11-16T16:41:48.785 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2023-11-16 16:30:46.183707904 +0000 2023-11-16T16:41:48.785 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2023-11-16T16:41:48.785 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-16T16:41:48.851 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2023-11-16T16:41:48.851 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2023-11-16T16:41:48.851 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000190266 s, 2.7 MB/s 2023-11-16T16:41:48.852 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-16T16:41:48.911 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2023-11-16T16:41:48.911 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2023-11-16T16:41:48.927 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-16T16:41:48.928 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_1 2023-11-16T16:41:48.985 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-16T16:41:48.985 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:48.985 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 6h/6d Inode: 389814 Links: 1 2023-11-16T16:41:48.985 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:48.985 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:48.985 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2023-11-16 16:41:25.375814618 +0000 2023-11-16T16:41:48.985 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2023-11-16 16:30:51.754030945 +0000 2023-11-16T16:41:48.985 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2023-11-16 16:30:51.754030945 +0000 2023-11-16T16:41:48.985 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2023-11-16T16:41:48.986 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-16T16:41:49.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:48 smithi141 ceph-mon[131184]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:49.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:48 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:49.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:48 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:49.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:48 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:49.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:48 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T16:41:49.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:48 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T16:41:49.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:48 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T16:41:49.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:48 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:49.052 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2023-11-16T16:41:49.052 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2023-11-16T16:41:49.052 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000228842 s, 2.2 MB/s 2023-11-16T16:41:49.053 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-16T16:41:49.101 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:48 smithi155 ceph-mon[131607]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:49.101 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:48 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:49.101 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:48 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:49.101 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:48 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:49.101 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:48 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T16:41:49.101 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:48 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T16:41:49.102 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:48 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T16:41:49.102 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:48 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:49.110 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_2 2023-11-16T16:41:49.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:48 smithi002 ceph-mon[128712]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:49.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:48 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:49.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:48 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:49.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:48 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:49.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:48 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T16:41:49.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:48 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T16:41:49.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:48 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T16:41:49.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:48 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:49.168 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-16T16:41:49.168 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:49.168 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 6h/6d Inode: 384903 Links: 1 2023-11-16T16:41:49.168 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:49.168 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:49.168 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2023-11-16 16:41:25.375814618 +0000 2023-11-16T16:41:49.168 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2023-11-16 16:30:52.265021054 +0000 2023-11-16T16:41:49.169 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2023-11-16 16:30:52.265021054 +0000 2023-11-16T16:41:49.169 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2023-11-16T16:41:49.169 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-16T16:41:49.235 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2023-11-16T16:41:49.235 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2023-11-16T16:41:49.235 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000227529 s, 2.3 MB/s 2023-11-16T16:41:49.236 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-16T16:41:49.294 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_3 2023-11-16T16:41:49.351 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-16T16:41:49.351 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:49.352 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 6h/6d Inode: 388692 Links: 1 2023-11-16T16:41:49.352 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:49.352 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:49.352 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2023-11-16 16:41:25.375814618 +0000 2023-11-16T16:41:49.352 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2023-11-16 16:30:52.766011357 +0000 2023-11-16T16:41:49.352 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2023-11-16 16:30:52.766011357 +0000 2023-11-16T16:41:49.352 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2023-11-16T16:41:49.352 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-16T16:41:49.418 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2023-11-16T16:41:49.419 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2023-11-16T16:41:49.419 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000235424 s, 2.2 MB/s 2023-11-16T16:41:49.420 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-16T16:41:49.479 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_4 2023-11-16T16:41:49.535 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-16T16:41:49.535 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-16T16:41:49.536 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 6h/6d Inode: 392903 Links: 1 2023-11-16T16:41:49.536 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:41:49.536 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2023-11-16T16:41:49.536 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2023-11-16 16:41:25.375814618 +0000 2023-11-16T16:41:49.536 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2023-11-16 16:30:53.269001620 +0000 2023-11-16T16:41:49.536 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2023-11-16 16:30:53.269001620 +0000 2023-11-16T16:41:49.536 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2023-11-16T16:41:49.536 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-16T16:41:49.601 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2023-11-16T16:41:49.601 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2023-11-16T16:41:49.601 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000173583 s, 2.9 MB/s 2023-11-16T16:41:49.602 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-16T16:41:49.661 INFO:tasks.cephadm:Deploying osd.0 on smithi002 with /dev/vg_nvme/lv_4... 2023-11-16T16:41:49.661 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-11-16T16:41:50.096 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:49 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:41:50.096 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:49 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:41:50.096 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:49 smithi155 ceph-mon[131607]: Deploying daemon mgr.z on smithi155 2023-11-16T16:41:50.096 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:49 smithi155 systemd[1]: Starting Ceph mgr.z for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:41:50.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:49 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:41:50.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:49 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:41:50.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:49 smithi002 ceph-mon[128712]: Deploying daemon mgr.z on smithi155 2023-11-16T16:41:50.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:49 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:41:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:49 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:41:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:49 smithi141 ceph-mon[131184]: Deploying daemon mgr.z on smithi155 2023-11-16T16:41:50.367 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:50 smithi155 podman[133708]: 2023-11-16T16:41:50.367 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:50 smithi155 ceph-mgr[133748]: -- 172.21.15.155:0/2634685561 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559f84c95520 con 0x559f84c6a800 2023-11-16T16:41:50.368 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:50 smithi155 bash[133708]: 2f95791bfcd546e1f9d102866146103a74108266b2560ebf1f5331530bef1b14 2023-11-16T16:41:50.528 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:41:50.715 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:50 smithi155 systemd[1]: Started Ceph mgr.z for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:41:50.715 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:50 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:50.451+0000 7f08e9c83200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T16:41:50.716 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:50 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:50.534+0000 7f08e9c83200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T16:41:50.716 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:50 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:50.622+0000 7f08e9c83200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T16:41:51.118 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:50 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:50.714+0000 7f08e9c83200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T16:41:51.118 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:50 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:50.883+0000 7f08e9c83200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T16:41:51.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:50 smithi155 ceph-mon[131607]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:51.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:51.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:51.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:51.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T16:41:51.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T16:41:51.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T16:41:51.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:51.134 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:50 smithi002 ceph-mon[128712]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:51.134 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:51.134 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:51.134 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:51.134 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T16:41:51.134 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T16:41:51.134 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T16:41:51.134 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:51.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:50 smithi141 ceph-mon[131184]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:51.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:51.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:51.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:51.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T16:41:51.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-16T16:41:51.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T16:41:51.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:51.502 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:51 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:51.189+0000 7f08e9c83200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T16:41:51.502 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:51 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:51.277+0000 7f08e9c83200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T16:41:51.502 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:51 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:51.408+0000 7f08e9c83200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T16:41:51.852 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:51 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:51.501+0000 7f08e9c83200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T16:41:52.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:51 smithi155 ceph-mon[131607]: Deploying daemon mgr.y on smithi141 2023-11-16T16:41:52.118 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:51 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:51.920+0000 7f08e9c83200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T16:41:52.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:51 smithi002 ceph-mon[128712]: Deploying daemon mgr.y on smithi141 2023-11-16T16:41:52.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:51 smithi141 ceph-mon[131184]: Deploying daemon mgr.y on smithi141 2023-11-16T16:41:52.265 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:52 smithi141 systemd[1]: Starting Ceph mgr.y for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:41:52.544 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-11-16T16:41:52.544 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-11-16T16:41:52.544 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2023-11-16T16:41:52.544 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2023-11-16T16:41:52.544 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0404579 s, 259 MB/s 2023-11-16T16:41:52.544 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2023-11-16T16:41:52.549 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:52 smithi141 podman[132634]: 2023-11-16T16:41:52.846 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:52 smithi141 bash[132634]: a93e0f19fa7be1ffb98ba2bb3d9a8c2d49691b33f6550080dc4f931221efb3b7 2023-11-16T16:41:52.846 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:52 smithi141 ceph-mgr[132674]: -- 172.21.15.141:0/2256310374 <== mon.2 v2:172.21.15.141:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564cc81df520 con 0x564cc81b4c00 2023-11-16T16:41:52.846 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:52 smithi141 systemd[1]: Started Ceph mgr.y for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:41:52.846 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:52 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:52.675+0000 7ff941e17200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T16:41:52.846 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:52 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:52.754+0000 7ff941e17200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T16:41:52.846 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:52 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:52.844+0000 7ff941e17200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T16:41:52.854 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:52 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:52.491+0000 7f08e9c83200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T16:41:52.854 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:52 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:52.661+0000 7f08e9c83200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T16:41:53.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:52 smithi002 ceph-mon[128712]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:53.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:53.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:53.113 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:53.114 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:53.116 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi002:vg_nvme/lv_4 2023-11-16T16:41:53.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:52 smithi155 ceph-mon[131607]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:53.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:53.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:53.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:53.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:53.119 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:52 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:52.941+0000 7ff941e17200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T16:41:53.120 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:52 smithi141 ceph-mon[131184]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:53.120 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:53.120 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:53.120 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:53.120 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:53.442 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:53 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:53.118+0000 7ff941e17200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T16:41:53.732 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:53 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:53.458+0000 7f08e9c83200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:41:53.732 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:53 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:53.555+0000 7f08e9c83200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T16:41:53.764 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:53 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:53.440+0000 7ff941e17200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T16:41:53.765 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:53 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:53.533+0000 7ff941e17200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T16:41:53.765 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:53 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:53.675+0000 7ff941e17200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T16:41:53.974 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:41:54.022 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:53 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:53.730+0000 7f08e9c83200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:41:54.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:53 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:41:54.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:53 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:54.022 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:53 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:54.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:53 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:41:54.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:53 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:54.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:53 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:54.224 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:53 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:53.772+0000 7ff941e17200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T16:41:54.225 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:53 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:41:54.225 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:53 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:54.225 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:53 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:54.368 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:54 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:54.020+0000 7f08e9c83200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T16:41:54.368 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:54 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:54.268+0000 7f08e9c83200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T16:41:54.514 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:54 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:54.222+0000 7ff941e17200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T16:41:54.684 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:54 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:54.421+0000 7f08e9c83200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T16:41:55.117 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:54 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:54.683+0000 7f08e9c83200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T16:41:55.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:54 smithi155 ceph-mon[131607]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:55.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:55.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:55.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:55.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:55.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:55.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:55.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:55.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T16:41:55.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T16:41:55.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:55.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:54 smithi002 ceph-mon[128712]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:55.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:55.119 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:55.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:55.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:55.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:55.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:55.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:55.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T16:41:55.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T16:41:55.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:55.264 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:54 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:54.814+0000 7ff941e17200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T16:41:55.265 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:54 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:54.984+0000 7ff941e17200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T16:41:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:54 smithi141 ceph-mon[131184]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:55.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-16T16:41:55.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-16T16:41:55.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:55.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.414+0000 7f3926bfa700 1 -- 172.21.15.2:0/50308808 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39201081c0 msgr2=0x7f392010a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:55.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.414+0000 7f3926bfa700 1 --2- 172.21.15.2:0/50308808 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39201081c0 0x7f392010a600 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f391800b0a0 tx=0x7f391800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:41:55.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.415+0000 7f3926bfa700 1 -- 172.21.15.2:0/50308808 shutdown_connections 2023-11-16T16:41:55.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.415+0000 7f3926bfa700 1 --2- 172.21.15.2:0/50308808 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39201081c0 0x7f392010a600 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:55.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.415+0000 7f3926bfa700 1 --2- 172.21.15.2:0/50308808 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3920074b20 0x7f3920072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:55.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.415+0000 7f3926bfa700 1 --2- 172.21.15.2:0/50308808 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3920073550 0x7f39200739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:55.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.415+0000 7f3926bfa700 1 -- 172.21.15.2:0/50308808 >> 172.21.15.2:0/50308808 conn(0x7f39200fb900 msgr2=0x7f39200fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:41:55.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.415+0000 7f3926bfa700 1 -- 172.21.15.2:0/50308808 shutdown_connections 2023-11-16T16:41:55.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.415+0000 7f3926bfa700 1 -- 172.21.15.2:0/50308808 wait complete. 2023-11-16T16:41:55.418 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.416+0000 7f3926bfa700 1 Processor -- start 2023-11-16T16:41:55.418 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.416+0000 7f3926bfa700 1 -- start start 2023-11-16T16:41:55.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.417+0000 7f3926bfa700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3920073550 0x7f3920198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:55.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.417+0000 7f3926bfa700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3920074b20 0x7f3920198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:55.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.417+0000 7f3926bfa700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39201081c0 0x7f392019f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:55.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.417+0000 7f3926bfa700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392019fec0 con 0x7f39201081c0 2023-11-16T16:41:55.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.417+0000 7f3926bfa700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39201a0000 con 0x7f3920074b20 2023-11-16T16:41:55.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.417+0000 7f3926bfa700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39201a0190 con 0x7f3920073550 2023-11-16T16:41:55.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.417+0000 7f3924996700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3920073550 0x7f3920198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:55.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.417+0000 7f391ffff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3920074b20 0x7f3920198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:55.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.417+0000 7f391ffff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3920074b20 0x7f3920198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:51880/0 (socket says 172.21.15.2:51880) 2023-11-16T16:41:55.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.417+0000 7f3924996700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3920073550 0x7f3920198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:51338/0 (socket says 172.21.15.2:51338) 2023-11-16T16:41:55.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.417+0000 7f391ffff700 1 -- 172.21.15.2:0/596227716 learned_addr learned my addr 172.21.15.2:0/596227716 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:41:55.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.417+0000 7f3925197700 1 --2- 172.21.15.2:0/596227716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39201081c0 0x7f392019f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:55.421 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.417+0000 7f391ffff700 1 -- 172.21.15.2:0/596227716 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3920073550 msgr2=0x7f3920198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:55.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.417+0000 7f391ffff700 1 --2- 172.21.15.2:0/596227716 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3920073550 0x7f3920198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:55.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.418+0000 7f391ffff700 1 -- 172.21.15.2:0/596227716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39201081c0 msgr2=0x7f392019f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:41:55.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.418+0000 7f391ffff700 1 --2- 172.21.15.2:0/596227716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39201081c0 0x7f392019f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:41:55.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.418+0000 7f391ffff700 1 -- 172.21.15.2:0/596227716 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3918009d00 con 0x7f3920074b20 2023-11-16T16:41:55.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.418+0000 7f391ffff700 1 --2- 172.21.15.2:0/596227716 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3920074b20 0x7f3920198870 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f390c00e9a0 tx=0x7f390c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:55.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.418+0000 7f391dffb700 1 -- 172.21.15.2:0/596227716 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390c009790 con 0x7f3920074b20 2023-11-16T16:41:55.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.418+0000 7f3926bfa700 1 -- 172.21.15.2:0/596227716 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392010de90 con 0x7f3920074b20 2023-11-16T16:41:55.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.418+0000 7f391dffb700 1 -- 172.21.15.2:0/596227716 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f390c010430 con 0x7f3920074b20 2023-11-16T16:41:55.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.418+0000 7f391dffb700 1 -- 172.21.15.2:0/596227716 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390c018680 con 0x7f3920074b20 2023-11-16T16:41:55.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.418+0000 7f3926bfa700 1 -- 172.21.15.2:0/596227716 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f392010dd00 con 0x7f3920074b20 2023-11-16T16:41:55.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.419+0000 7f391dffb700 1 -- 172.21.15.2:0/596227716 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 13) v1 ==== 45538+0+0 (secure 0 0 0) 0x7f390c00ca10 con 0x7f3920074b20 2023-11-16T16:41:55.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.419+0000 7f3926bfa700 1 -- 172.21.15.2:0/596227716 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3920106870 con 0x7f3920074b20 2023-11-16T16:41:55.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.420+0000 7f391dffb700 1 --2- 172.21.15.2:0/596227716 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f39100389e0 0x7f391003ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:41:55.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.420+0000 7f391dffb700 1 -- 172.21.15.2:0/596227716 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f390c014070 con 0x7f3920074b20 2023-11-16T16:41:55.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.420+0000 7f3924996700 1 --2- 172.21.15.2:0/596227716 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f39100389e0 0x7f391003ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:41:55.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.420+0000 7f3924996700 1 --2- 172.21.15.2:0/596227716 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f39100389e0 0x7f391003ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3914009c60 tx=0x7f3914009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:41:55.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.425+0000 7f391dffb700 1 -- 172.21.15.2:0/596227716 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f390c010880 con 0x7f3920074b20 2023-11-16T16:41:55.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:55.646+0000 7f3926bfa700 1 -- 172.21.15.2:0/596227716 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f3920061c60 con 0x7f39100389e0 2023-11-16T16:41:55.709 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:55 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:55.386+0000 7f08e9c83200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T16:41:55.709 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:55 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:55.516+0000 7f08e9c83200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T16:41:55.709 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:55 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:55.708+0000 7f08e9c83200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T16:41:56.006 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:55 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:55.743+0000 7ff941e17200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:41:56.006 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:55 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:55.833+0000 7ff941e17200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T16:41:56.006 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:55 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:41:56.006 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:55 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:41:56.006 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:55 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:41:56.007 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:55 smithi141 ceph-mon[131184]: Reconfiguring mgr.x (unknown last config time)... 2023-11-16T16:41:56.007 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:55 smithi141 ceph-mon[131184]: Reconfiguring daemon mgr.x on smithi002 2023-11-16T16:41:56.007 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:55 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:41:56.007 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:55 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:41:56.007 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:55 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:56.041 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:55 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:55.822+0000 7f08e9c83200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T16:41:56.041 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:55 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:41:56.041 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:55 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:41:56.041 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:55 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:41:56.041 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:55 smithi155 ceph-mon[131607]: Reconfiguring mgr.x (unknown last config time)... 2023-11-16T16:41:56.042 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:55 smithi155 ceph-mon[131607]: Reconfiguring daemon mgr.x on smithi002 2023-11-16T16:41:56.042 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:55 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:41:56.042 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:55 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:41:56.042 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:55 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:56.091 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:55 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:41:56.091 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:55 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:41:56.091 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:55 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:41:56.092 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:55 smithi002 ceph-mon[128712]: Reconfiguring mgr.x (unknown last config time)... 2023-11-16T16:41:56.092 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:55 smithi002 ceph-mon[128712]: Reconfiguring daemon mgr.x on smithi002 2023-11-16T16:41:56.092 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:55 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:41:56.092 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:55 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:41:56.092 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:55 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:56.264 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:56 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:56.004+0000 7ff941e17200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:41:56.367 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:56 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:56.040+0000 7f08e9c83200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T16:41:56.655 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:56 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:56.280+0000 7ff941e17200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T16:41:56.655 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:56 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:56.507+0000 7ff941e17200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T16:41:56.921 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:56 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:56.653+0000 7ff941e17200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T16:41:56.921 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:56 smithi141 ceph-mon[131184]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:56.921 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:56 smithi141 ceph-mon[131184]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:56.922 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:56 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:56.922 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:56 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:56.922 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:56 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:56.922 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:56 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:56.922 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:56 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:56.922 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:56 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:57.117 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:41:56 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:41:56.893+0000 7f08e9c83200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T16:41:57.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:56 smithi155 ceph-mon[131607]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:57.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:56 smithi155 ceph-mon[131607]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:57.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:56 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:57.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:56 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:57.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:56 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:57.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:56 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:57.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:56 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:57.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:56 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:57.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:56 smithi002 ceph-mon[128712]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:41:57.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:56 smithi002 ceph-mon[128712]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:57.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:56 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:57.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:56 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:57.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:56 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:41:57.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:56 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:41:57.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:56 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:41:57.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:56 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:41:57.264 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:56 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:56.919+0000 7ff941e17200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T16:41:57.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:57.869+0000 7f391dffb700 1 -- 172.21.15.2:0/596227716 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mgrmap(e 14) v1 ==== 90598+0+0 (secure 0 0 0) 0x7f390c0187e0 con 0x7f3920074b20 2023-11-16T16:41:57.944 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:57 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:57.620+0000 7ff941e17200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T16:41:57.944 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:57 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:57.751+0000 7ff941e17200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T16:41:57.944 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:57 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:41:57.944 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:57 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:41:57.944 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:57 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:41:57.944 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:57 smithi141 ceph-mon[131184]: Standby manager daemon z started 2023-11-16T16:41:57.944 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:57 smithi141 ceph-mon[131184]: from='mgr.? 172.21.15.155:0/2388692694' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T16:41:57.944 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:57 smithi141 ceph-mon[131184]: from='mgr.? 172.21.15.155:0/2388692694' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:41:57.944 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:57 smithi141 ceph-mon[131184]: from='mgr.? 172.21.15.155:0/2388692694' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T16:41:57.944 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:57 smithi141 ceph-mon[131184]: from='mgr.? 172.21.15.155:0/2388692694' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:41:57.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:57 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:41:57.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:57 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:41:57.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:57 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:41:57.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:57 smithi002 ceph-mon[128712]: Standby manager daemon z started 2023-11-16T16:41:57.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:57 smithi002 ceph-mon[128712]: from='mgr.? 172.21.15.155:0/2388692694' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T16:41:57.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:57 smithi002 ceph-mon[128712]: from='mgr.? 172.21.15.155:0/2388692694' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:41:57.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:57 smithi002 ceph-mon[128712]: from='mgr.? 172.21.15.155:0/2388692694' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T16:41:57.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:57 smithi002 ceph-mon[128712]: from='mgr.? 172.21.15.155:0/2388692694' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:41:58.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:57 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:41:58.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:57 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:41:58.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:57 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:41:58.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:57 smithi155 ceph-mon[131607]: Standby manager daemon z started 2023-11-16T16:41:58.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:57 smithi155 ceph-mon[131607]: from='mgr.? 172.21.15.155:0/2388692694' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T16:41:58.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:57 smithi155 ceph-mon[131607]: from='mgr.? 172.21.15.155:0/2388692694' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:41:58.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:57 smithi155 ceph-mon[131607]: from='mgr.? 172.21.15.155:0/2388692694' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T16:41:58.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:57 smithi155 ceph-mon[131607]: from='mgr.? 172.21.15.155:0/2388692694' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:41:58.264 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:57 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:57.942+0000 7ff941e17200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T16:41:58.265 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:58 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:58.055+0000 7ff941e17200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T16:41:58.764 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:58 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:58.269+0000 7ff941e17200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T16:41:59.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:58 smithi002 ceph-mon[128712]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:59.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:58 smithi002 ceph-mon[128712]: mgrmap e14: x(active, since 74s), standbys: z 2023-11-16T16:41:59.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:58 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T16:41:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:58 smithi141 ceph-mon[131184]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:58 smithi141 ceph-mon[131184]: mgrmap e14: x(active, since 74s), standbys: z 2023-11-16T16:41:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:58 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T16:41:59.265 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:41:59 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:41:59.117+0000 7ff941e17200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T16:41:59.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:58 smithi155 ceph-mon[131607]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:41:59.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:58 smithi155 ceph-mon[131607]: mgrmap e14: x(active, since 74s), standbys: z 2023-11-16T16:41:59.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:58 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T16:41:59.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:41:59.880+0000 7f391dffb700 1 -- 172.21.15.2:0/596227716 <== mon.2 v2:172.21.15.141:3300/0 8 ==== mgrmap(e 15) v1 ==== 135658+0+0 (secure 0 0 0) 0x7f390c016330 con 0x7f3920074b20 2023-11-16T16:42:00.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:59 smithi002 ceph-mon[128712]: Standby manager daemon y started 2023-11-16T16:42:00.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:59 smithi002 ceph-mon[128712]: from='mgr.? 172.21.15.141:0/61376742' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T16:42:00.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:59 smithi002 ceph-mon[128712]: from='mgr.? 172.21.15.141:0/61376742' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:42:00.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:59 smithi002 ceph-mon[128712]: from='mgr.? 172.21.15.141:0/61376742' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T16:42:00.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:59 smithi002 ceph-mon[128712]: from='mgr.? 172.21.15.141:0/61376742' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:42:00.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:59 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/88760527' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bfde47f9-e428-42b2-a86a-d1a39a7f8084"}]: dispatch 2023-11-16T16:42:00.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:59 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/88760527' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bfde47f9-e428-42b2-a86a-d1a39a7f8084"}]': finished 2023-11-16T16:42:00.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:59 smithi002 ceph-mon[128712]: osdmap e5: 1 total, 0 up, 1 in 2023-11-16T16:42:00.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:41:59 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:00.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:59 smithi141 ceph-mon[131184]: Standby manager daemon y started 2023-11-16T16:42:00.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:59 smithi141 ceph-mon[131184]: from='mgr.? 172.21.15.141:0/61376742' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T16:42:00.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:59 smithi141 ceph-mon[131184]: from='mgr.? 172.21.15.141:0/61376742' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:42:00.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:59 smithi141 ceph-mon[131184]: from='mgr.? 172.21.15.141:0/61376742' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T16:42:00.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:59 smithi141 ceph-mon[131184]: from='mgr.? 172.21.15.141:0/61376742' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:42:00.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:59 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/88760527' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bfde47f9-e428-42b2-a86a-d1a39a7f8084"}]: dispatch 2023-11-16T16:42:00.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:59 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/88760527' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bfde47f9-e428-42b2-a86a-d1a39a7f8084"}]': finished 2023-11-16T16:42:00.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:59 smithi141 ceph-mon[131184]: osdmap e5: 1 total, 0 up, 1 in 2023-11-16T16:42:00.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:41:59 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:00.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:59 smithi155 ceph-mon[131607]: Standby manager daemon y started 2023-11-16T16:42:00.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:59 smithi155 ceph-mon[131607]: from='mgr.? 172.21.15.141:0/61376742' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T16:42:00.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:59 smithi155 ceph-mon[131607]: from='mgr.? 172.21.15.141:0/61376742' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:42:00.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:59 smithi155 ceph-mon[131607]: from='mgr.? 172.21.15.141:0/61376742' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T16:42:00.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:59 smithi155 ceph-mon[131607]: from='mgr.? 172.21.15.141:0/61376742' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:42:00.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:59 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/88760527' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bfde47f9-e428-42b2-a86a-d1a39a7f8084"}]: dispatch 2023-11-16T16:42:00.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:59 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/88760527' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bfde47f9-e428-42b2-a86a-d1a39a7f8084"}]': finished 2023-11-16T16:42:00.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:59 smithi155 ceph-mon[131607]: osdmap e5: 1 total, 0 up, 1 in 2023-11-16T16:42:00.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:41:59 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:01.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:00 smithi002 ceph-mon[128712]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:01.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:00 smithi002 ceph-mon[128712]: mgrmap e15: x(active, since 76s), standbys: z, y 2023-11-16T16:42:01.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:00 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T16:42:01.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:00 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2237927754' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:42:01.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:00 smithi141 ceph-mon[131184]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:01.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:00 smithi141 ceph-mon[131184]: mgrmap e15: x(active, since 76s), standbys: z, y 2023-11-16T16:42:01.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:00 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T16:42:01.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:00 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2237927754' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:42:01.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:00 smithi155 ceph-mon[131607]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:01.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:00 smithi155 ceph-mon[131607]: mgrmap e15: x(active, since 76s), standbys: z, y 2023-11-16T16:42:01.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:00 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T16:42:01.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:00 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2237927754' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:42:03.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:02 smithi002 ceph-mon[128712]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:03.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:02 smithi141 ceph-mon[131184]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:03.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:02 smithi155 ceph-mon[131607]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:04.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:04 smithi141 ceph-mon[131184]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:04.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:04 smithi155 ceph-mon[131607]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:04.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:04 smithi002 ceph-mon[128712]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:07.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:06 smithi155 ceph-mon[131607]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:07.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:06 smithi002 ceph-mon[128712]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:07.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:06 smithi141 ceph-mon[131184]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:08.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:07 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-11-16T16:42:08.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:07 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:08.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:07 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-11-16T16:42:08.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:07 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:08.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:07 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-11-16T16:42:08.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:07 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:09.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:08 smithi155 ceph-mon[131607]: Deploying daemon osd.0 on smithi002 2023-11-16T16:42:09.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:08 smithi155 ceph-mon[131607]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:09.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:08 smithi002 ceph-mon[128712]: Deploying daemon osd.0 on smithi002 2023-11-16T16:42:09.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:08 smithi002 ceph-mon[128712]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:09.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:08 smithi141 ceph-mon[131184]: Deploying daemon osd.0 on smithi002 2023-11-16T16:42:09.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:08 smithi141 ceph-mon[131184]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:11.056 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:10 smithi002 ceph-mon[128712]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:11.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:10 smithi155 ceph-mon[131607]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:10 smithi141 ceph-mon[131184]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:13.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:12 smithi155 ceph-mon[131607]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:13.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:12 smithi002 ceph-mon[128712]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:13.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:12 smithi141 ceph-mon[131184]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:15.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:14 smithi002 ceph-mon[128712]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:15.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:14 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:15.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:14 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:15.112 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:14 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:15.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:14 smithi155 ceph-mon[131607]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:15.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:14 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:15.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:14 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:15.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:14 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:14 smithi141 ceph-mon[131184]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:14 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:14 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:14 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:17.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:16 smithi155 ceph-mon[131607]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:17.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:16 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:17.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:16 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:17.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:16 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:17.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:16 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:17.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:16 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:17.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:16 smithi002 ceph-mon[128712]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:17.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:16 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:17.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:16 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:17.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:16 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:17.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:16 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:17.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:16 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:16 smithi141 ceph-mon[131184]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:16 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:16 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:16 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:16 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:16 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:17.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:42:17.559+0000 7f391dffb700 1 -- 172.21.15.2:0/596227716 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3920061c60 con 0x7f39100389e0 2023-11-16T16:42:17.562 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 0 on host 'smithi002' 2023-11-16T16:42:17.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:42:17.563+0000 7f3926bfa700 1 -- 172.21.15.2:0/596227716 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f39100389e0 msgr2=0x7f391003ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:17.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:42:17.563+0000 7f3926bfa700 1 --2- 172.21.15.2:0/596227716 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f39100389e0 0x7f391003ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3914009c60 tx=0x7f3914009260 comp rx=0 tx=0).stop 2023-11-16T16:42:17.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:42:17.563+0000 7f3926bfa700 1 -- 172.21.15.2:0/596227716 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3920074b20 msgr2=0x7f3920198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:17.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:42:17.563+0000 7f3926bfa700 1 --2- 172.21.15.2:0/596227716 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3920074b20 0x7f3920198870 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f390c00e9a0 tx=0x7f390c00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:42:17.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:42:17.563+0000 7f3926bfa700 1 -- 172.21.15.2:0/596227716 shutdown_connections 2023-11-16T16:42:17.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:42:17.563+0000 7f3926bfa700 1 --2- 172.21.15.2:0/596227716 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f39100389e0 0x7f391003ae90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:17.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:42:17.563+0000 7f3926bfa700 1 --2- 172.21.15.2:0/596227716 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3920073550 0x7f3920198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:17.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:42:17.563+0000 7f3926bfa700 1 --2- 172.21.15.2:0/596227716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39201081c0 0x7f392019f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:17.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:42:17.563+0000 7f3926bfa700 1 --2- 172.21.15.2:0/596227716 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3920074b20 0x7f3920198870 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:17.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:42:17.563+0000 7f3926bfa700 1 -- 172.21.15.2:0/596227716 >> 172.21.15.2:0/596227716 conn(0x7f39200fb900 msgr2=0x7f39200fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:42:17.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:42:17.564+0000 7f3926bfa700 1 -- 172.21.15.2:0/596227716 shutdown_connections 2023-11-16T16:42:17.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:42:17.564+0000 7f3926bfa700 1 -- 172.21.15.2:0/596227716 wait complete. 2023-11-16T16:42:17.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:17 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:42:17.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:17 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:42:17.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:17 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:42:17.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:17 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:17.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:17 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:17.895 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:17 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:18.012 DEBUG:teuthology.orchestra.run.smithi002:osd.0> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.0.service 2023-11-16T16:42:18.017 INFO:tasks.cephadm:Deploying osd.1 on smithi141 with /dev/vg_nvme/lv_4... 2023-11-16T16:42:18.017 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-11-16T16:42:18.094 INFO:journalctl@ceph.osd.0.smithi002.stdout:-- Logs begin at Thu 2023-11-16 16:19:53 UTC. -- 2023-11-16T16:42:18.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:17 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:42:18.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:17 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:42:18.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:17 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:42:18.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:17 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:18.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:17 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:18.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:17 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:18.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:17 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:42:18.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:17 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:42:18.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:17 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:42:18.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:17 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:18.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:17 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:18.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:17 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:18.664 INFO:teuthology.orchestra.run.smithi141.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T16:42:19.063 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:18 smithi002 ceph-mon[128712]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:19.063 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:18 smithi002 ceph-mon[128712]: from='osd.0 [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-16T16:42:19.100 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:18 smithi141 ceph-mon[131184]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:19.100 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:18 smithi141 ceph-mon[131184]: from='osd.0 [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-16T16:42:19.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:18 smithi155 ceph-mon[131607]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:19.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:18 smithi155 ceph-mon[131607]: from='osd.0 [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-16T16:42:20.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:19 smithi155 ceph-mon[131607]: from='osd.0 [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-11-16T16:42:20.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:19 smithi155 ceph-mon[131607]: osdmap e6: 1 total, 0 up, 1 in 2023-11-16T16:42:20.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:19 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:20.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:19 smithi155 ceph-mon[131607]: from='osd.0 [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T16:42:20.156 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 16:42:19 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-0[140855]: 2023-11-16T16:42:19.865+0000 7f67908d3700 -1 osd.0 0 waiting for initial osdmap 2023-11-16T16:42:20.156 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 16:42:19 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-0[140855]: 2023-11-16T16:42:19.869+0000 7f678bec7700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:42:20.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:19 smithi002 ceph-mon[128712]: from='osd.0 [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-11-16T16:42:20.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:19 smithi002 ceph-mon[128712]: osdmap e6: 1 total, 0 up, 1 in 2023-11-16T16:42:20.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:19 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:20.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:19 smithi002 ceph-mon[128712]: from='osd.0 [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T16:42:20.196 INFO:teuthology.orchestra.run.smithi141.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-11-16T16:42:20.196 INFO:teuthology.orchestra.run.smithi141.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-11-16T16:42:20.196 INFO:teuthology.orchestra.run.smithi141.stderr: stderr: 10+0 records in 2023-11-16T16:42:20.196 INFO:teuthology.orchestra.run.smithi141.stderr:10+0 records out 2023-11-16T16:42:20.196 INFO:teuthology.orchestra.run.smithi141.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01275 s, 822 MB/s 2023-11-16T16:42:20.197 INFO:teuthology.orchestra.run.smithi141.stderr:--> Zapping successful for: 2023-11-16T16:42:20.229 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:19 smithi141 ceph-mon[131184]: from='osd.0 [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-11-16T16:42:20.229 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:19 smithi141 ceph-mon[131184]: osdmap e6: 1 total, 0 up, 1 in 2023-11-16T16:42:20.229 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:19 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:20.229 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:19 smithi141 ceph-mon[131184]: from='osd.0 [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T16:42:20.606 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi141:vg_nvme/lv_4 2023-11-16T16:42:21.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:20 smithi002 ceph-mon[128712]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:21.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:20 smithi002 ceph-mon[128712]: from='osd.0 [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T16:42:21.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:20 smithi002 ceph-mon[128712]: osdmap e7: 1 total, 0 up, 1 in 2023-11-16T16:42:21.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:20 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:21.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:20 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:21.245 INFO:teuthology.orchestra.run.smithi141.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T16:42:21.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:20 smithi141 ceph-mon[131184]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:20 smithi141 ceph-mon[131184]: from='osd.0 [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T16:42:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:20 smithi141 ceph-mon[131184]: osdmap e7: 1 total, 0 up, 1 in 2023-11-16T16:42:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:20 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:20 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:21.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:20 smithi155 ceph-mon[131607]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-16T16:42:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:20 smithi155 ceph-mon[131607]: from='osd.0 [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T16:42:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:20 smithi155 ceph-mon[131607]: osdmap e7: 1 total, 0 up, 1 in 2023-11-16T16:42:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:20 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:20 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:21.960 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:21 smithi141 ceph-mon[131184]: purged_snaps scrub starts 2023-11-16T16:42:21.960 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:21 smithi141 ceph-mon[131184]: purged_snaps scrub ok 2023-11-16T16:42:21.961 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:21 smithi141 ceph-mon[131184]: osd.0 [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294] boot 2023-11-16T16:42:21.961 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:21 smithi141 ceph-mon[131184]: osdmap e8: 1 total, 1 up, 1 in 2023-11-16T16:42:21.961 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:21 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:21.961 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:21 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:21.961 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:21 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:21.961 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:21 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:42:21.961 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:21 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:21.961 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:21 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:21.961 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:21 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:21.961 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:21 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:22.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:21 smithi002 ceph-mon[128712]: purged_snaps scrub starts 2023-11-16T16:42:22.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:21 smithi002 ceph-mon[128712]: purged_snaps scrub ok 2023-11-16T16:42:22.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:21 smithi002 ceph-mon[128712]: osd.0 [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294] boot 2023-11-16T16:42:22.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:21 smithi002 ceph-mon[128712]: osdmap e8: 1 total, 1 up, 1 in 2023-11-16T16:42:22.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:21 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:22.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:21 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:22.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:21 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:22.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:21 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:42:22.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:21 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:22.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:21 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:22.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:21 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:22.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:21 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:22.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:21 smithi155 ceph-mon[131607]: purged_snaps scrub starts 2023-11-16T16:42:22.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:21 smithi155 ceph-mon[131607]: purged_snaps scrub ok 2023-11-16T16:42:22.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:21 smithi155 ceph-mon[131607]: osd.0 [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294] boot 2023-11-16T16:42:22.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:21 smithi155 ceph-mon[131607]: osdmap e8: 1 total, 1 up, 1 in 2023-11-16T16:42:22.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:21 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:42:22.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:21 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:22.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:21 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:22.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:21 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:42:22.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:21 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:22.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:21 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:22.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:21 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:22.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:21 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:22.369 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.367+0000 7f4451dd2700 1 -- 172.21.15.141:0/3121234795 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f444c1003e0 msgr2=0x7f444c100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:22.369 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.367+0000 7f4451dd2700 1 --2- 172.21.15.141:0/3121234795 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f444c1003e0 0x7f444c100830 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f443c009990 tx=0x7f443c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:42:22.370 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.367+0000 7f4451dd2700 1 -- 172.21.15.141:0/3121234795 shutdown_connections 2023-11-16T16:42:22.370 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.367+0000 7f4451dd2700 1 --2- 172.21.15.141:0/3121234795 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f444c0ff1e0 0x7f444c0ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:22.370 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.367+0000 7f4451dd2700 1 --2- 172.21.15.141:0/3121234795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f444c100d70 0x7f444c107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:22.370 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.367+0000 7f4451dd2700 1 --2- 172.21.15.141:0/3121234795 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f444c1003e0 0x7f444c100830 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:22.370 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.367+0000 7f4451dd2700 1 -- 172.21.15.141:0/3121234795 >> 172.21.15.141:0/3121234795 conn(0x7f444c0fa7f0 msgr2=0x7f444c0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:42:22.370 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.367+0000 7f4451dd2700 1 -- 172.21.15.141:0/3121234795 shutdown_connections 2023-11-16T16:42:22.370 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.367+0000 7f4451dd2700 1 -- 172.21.15.141:0/3121234795 wait complete. 2023-11-16T16:42:22.370 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.368+0000 7f4451dd2700 1 Processor -- start 2023-11-16T16:42:22.372 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.368+0000 7f4451dd2700 1 -- start start 2023-11-16T16:42:22.372 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.368+0000 7f4451dd2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f444c0ff1e0 0x7f444c0699c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:42:22.372 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.368+0000 7f4451dd2700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f444c1003e0 0x7f444c19ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:42:22.373 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.368+0000 7f4451dd2700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f444c100d70 0x7f444c19d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:42:22.373 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.368+0000 7f4451dd2700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444c19db40 con 0x7f444c1003e0 2023-11-16T16:42:22.373 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.368+0000 7f4451dd2700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444c19dcb0 con 0x7f444c100d70 2023-11-16T16:42:22.373 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.368+0000 7f4451dd2700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444c19de70 con 0x7f444c0ff1e0 2023-11-16T16:42:22.373 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.368+0000 7f444bfff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f444c100d70 0x7f444c19d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:42:22.373 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.368+0000 7f444bfff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f444c100d70 0x7f444c19d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59682/0 (socket says 172.21.15.141:59682) 2023-11-16T16:42:22.373 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.368+0000 7f444bfff700 1 -- 172.21.15.141:0/786559414 learned_addr learned my addr 172.21.15.141:0/786559414 (peer_addr_for_me v2:172.21.15.141:0/0) 2023-11-16T16:42:22.373 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.369+0000 7f444b7fe700 1 --2- 172.21.15.141:0/786559414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f444c0ff1e0 0x7f444c0699c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:42:22.373 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.369+0000 7f444affd700 1 --2- 172.21.15.141:0/786559414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f444c1003e0 0x7f444c19ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:42:22.373 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.369+0000 7f444bfff700 1 -- 172.21.15.141:0/786559414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f444c0ff1e0 msgr2=0x7f444c0699c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:22.374 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.369+0000 7f444bfff700 1 --2- 172.21.15.141:0/786559414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f444c0ff1e0 0x7f444c0699c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:22.374 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.369+0000 7f444bfff700 1 -- 172.21.15.141:0/786559414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f444c1003e0 msgr2=0x7f444c19ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:22.374 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.369+0000 7f444bfff700 1 --2- 172.21.15.141:0/786559414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f444c1003e0 0x7f444c19ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:22.374 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.369+0000 7f444bfff700 1 -- 172.21.15.141:0/786559414 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443c0095f0 con 0x7f444c100d70 2023-11-16T16:42:22.374 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.369+0000 7f444affd700 1 --2- 172.21.15.141:0/786559414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f444c1003e0 0x7f444c19ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:42:22.374 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.369+0000 7f444bfff700 1 --2- 172.21.15.141:0/786559414 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f444c100d70 0x7f444c19d3a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f444400b970 tx=0x7f444400bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:42:22.374 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.369+0000 7f4448ff9700 1 -- 172.21.15.141:0/786559414 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444400c9b0 con 0x7f444c100d70 2023-11-16T16:42:22.374 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.369+0000 7f4448ff9700 1 -- 172.21.15.141:0/786559414 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4444014440 con 0x7f444c100d70 2023-11-16T16:42:22.374 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.369+0000 7f4451dd2700 1 -- 172.21.15.141:0/786559414 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f444c10abb0 con 0x7f444c100d70 2023-11-16T16:42:22.374 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.369+0000 7f4448ff9700 1 -- 172.21.15.141:0/786559414 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44440075d0 con 0x7f444c100d70 2023-11-16T16:42:22.374 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.369+0000 7f4451dd2700 1 -- 172.21.15.141:0/786559414 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f444c10aa20 con 0x7f444c100d70 2023-11-16T16:42:22.375 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.370+0000 7f4451dd2700 1 -- 172.21.15.141:0/786559414 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f444c106790 con 0x7f444c100d70 2023-11-16T16:42:22.375 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.371+0000 7f4448ff9700 1 -- 172.21.15.141:0/786559414 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 15) v1 ==== 135658+0+0 (secure 0 0 0) 0x7f44440145b0 con 0x7f444c100d70 2023-11-16T16:42:22.375 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.372+0000 7f4448ff9700 1 --2- 172.21.15.141:0/786559414 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f44380a10c0 0x7f44380a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:42:22.376 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.372+0000 7f4448ff9700 1 -- 172.21.15.141:0/786559414 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f4444006e40 con 0x7f444c100d70 2023-11-16T16:42:22.376 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.373+0000 7f444b7fe700 1 --2- 172.21.15.141:0/786559414 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f44380a10c0 0x7f44380a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:42:22.376 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.373+0000 7f444b7fe700 1 --2- 172.21.15.141:0/786559414 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f44380a10c0 0x7f44380a3570 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f444c100240 tx=0x7f4434009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:42:22.377 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.376+0000 7f4448ff9700 1 -- 172.21.15.141:0/786559414 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4444095c30 con 0x7f444c100d70 2023-11-16T16:42:22.571 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:22.569+0000 7f4451dd2700 1 -- 172.21.15.141:0/786559414 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi141:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f444c061c80 con 0x7f44380a10c0 2023-11-16T16:42:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:22 smithi002 ceph-mon[128712]: Detected new or changed devices on smithi002 2023-11-16T16:42:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:22 smithi002 ceph-mon[128712]: Adjusting osd_memory_target on smithi002 to 17276M 2023-11-16T16:42:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:22 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:42:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:22 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:42:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:22 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:42:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:22 smithi002 ceph-mon[128712]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:22 smithi002 ceph-mon[128712]: osdmap e9: 1 total, 1 up, 1 in 2023-11-16T16:42:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:22 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:42:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:22 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:42:23.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:22 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:23.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:22 smithi141 ceph-mon[131184]: Detected new or changed devices on smithi002 2023-11-16T16:42:23.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:22 smithi141 ceph-mon[131184]: Adjusting osd_memory_target on smithi002 to 17276M 2023-11-16T16:42:23.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:22 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:42:23.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:22 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:42:23.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:22 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:42:23.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:22 smithi141 ceph-mon[131184]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:23.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:22 smithi141 ceph-mon[131184]: osdmap e9: 1 total, 1 up, 1 in 2023-11-16T16:42:23.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:22 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:42:23.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:22 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:42:23.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:22 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:23.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:22 smithi155 ceph-mon[131607]: Detected new or changed devices on smithi002 2023-11-16T16:42:23.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:22 smithi155 ceph-mon[131607]: Adjusting osd_memory_target on smithi002 to 17276M 2023-11-16T16:42:23.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:22 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:42:23.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:22 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:42:23.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:22 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:42:23.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:22 smithi155 ceph-mon[131607]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:23.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:22 smithi155 ceph-mon[131607]: osdmap e9: 1 total, 1 up, 1 in 2023-11-16T16:42:23.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:22 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:42:23.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:22 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:42:23.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:22 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:24.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:23 smithi141 ceph-mon[131184]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi141:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:42:24.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:23 smithi002 ceph-mon[128712]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi141:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:42:24.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:23 smithi155 ceph-mon[131607]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi141:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:42:25.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:24 smithi002 ceph-mon[128712]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:25.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:24 smithi141 ceph-mon[131184]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:25.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:24 smithi155 ceph-mon[131607]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:26.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:25 smithi002 ceph-mon[128712]: from='client.? 172.21.15.141:0/3210387952' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdef1dd6-b1c9-484c-9942-395e0213aa49"}]: dispatch 2023-11-16T16:42:26.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:25 smithi002 ceph-mon[128712]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdef1dd6-b1c9-484c-9942-395e0213aa49"}]: dispatch 2023-11-16T16:42:26.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:25 smithi002 ceph-mon[128712]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fdef1dd6-b1c9-484c-9942-395e0213aa49"}]': finished 2023-11-16T16:42:26.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:25 smithi002 ceph-mon[128712]: osdmap e10: 2 total, 1 up, 2 in 2023-11-16T16:42:26.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:25 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:26.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:25 smithi002 ceph-mon[128712]: from='client.? 172.21.15.141:0/4070436321' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:42:26.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:25 smithi141 ceph-mon[131184]: from='client.? 172.21.15.141:0/3210387952' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdef1dd6-b1c9-484c-9942-395e0213aa49"}]: dispatch 2023-11-16T16:42:26.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:25 smithi141 ceph-mon[131184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdef1dd6-b1c9-484c-9942-395e0213aa49"}]: dispatch 2023-11-16T16:42:26.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:25 smithi141 ceph-mon[131184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fdef1dd6-b1c9-484c-9942-395e0213aa49"}]': finished 2023-11-16T16:42:26.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:25 smithi141 ceph-mon[131184]: osdmap e10: 2 total, 1 up, 2 in 2023-11-16T16:42:26.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:25 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:26.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:25 smithi141 ceph-mon[131184]: from='client.? 172.21.15.141:0/4070436321' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:42:26.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:25 smithi155 ceph-mon[131607]: from='client.? 172.21.15.141:0/3210387952' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdef1dd6-b1c9-484c-9942-395e0213aa49"}]: dispatch 2023-11-16T16:42:26.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:25 smithi155 ceph-mon[131607]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdef1dd6-b1c9-484c-9942-395e0213aa49"}]: dispatch 2023-11-16T16:42:26.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:25 smithi155 ceph-mon[131607]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fdef1dd6-b1c9-484c-9942-395e0213aa49"}]': finished 2023-11-16T16:42:26.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:25 smithi155 ceph-mon[131607]: osdmap e10: 2 total, 1 up, 2 in 2023-11-16T16:42:26.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:25 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:26.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:25 smithi155 ceph-mon[131607]: from='client.? 172.21.15.141:0/4070436321' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:42:27.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:27 smithi155 ceph-mon[131607]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:27.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:27 smithi002 ceph-mon[128712]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:27.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:27 smithi141 ceph-mon[131184]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:29.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:29 smithi155 ceph-mon[131607]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:29.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:29 smithi002 ceph-mon[128712]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:29.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:29 smithi141 ceph-mon[131184]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:31.286 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:31 smithi141 ceph-mon[131184]: pgmap v53: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:31.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:31 smithi155 ceph-mon[131607]: pgmap v53: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:31.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:31 smithi002 ceph-mon[128712]: pgmap v53: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:32.299 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:32 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-11-16T16:42:32.299 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:32 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:32.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:32 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-11-16T16:42:32.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:32 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:32.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:32 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-11-16T16:42:32.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:32 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:33.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:33 smithi155 ceph-mon[131607]: Deploying daemon osd.1 on smithi141 2023-11-16T16:42:33.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:33 smithi155 ceph-mon[131607]: pgmap v54: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:33.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:33 smithi002 ceph-mon[128712]: Deploying daemon osd.1 on smithi141 2023-11-16T16:42:33.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:33 smithi002 ceph-mon[128712]: pgmap v54: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:33.464 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:33 smithi141 ceph-mon[131184]: Deploying daemon osd.1 on smithi141 2023-11-16T16:42:33.464 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:33 smithi141 ceph-mon[131184]: pgmap v54: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:34.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:34 smithi141 ceph-mon[131184]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:34.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:34 smithi155 ceph-mon[131607]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:34.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:34 smithi002 ceph-mon[128712]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:37.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:36 smithi155 ceph-mon[131607]: pgmap v56: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:37.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:36 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:37.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:36 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:37.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:36 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:37.143 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:36 smithi141 ceph-mon[131184]: pgmap v56: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:37.143 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:36 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:37.143 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:36 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:37.143 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:36 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:37.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:36 smithi002 ceph-mon[128712]: pgmap v56: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:37.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:36 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:37.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:36 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:37.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:36 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:38.736 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:38.733+0000 7f4448ff9700 1 -- 172.21.15.141:0/786559414 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f444c061c80 con 0x7f44380a10c0 2023-11-16T16:42:38.736 INFO:teuthology.orchestra.run.smithi141.stdout:Created osd(s) 1 on host 'smithi141' 2023-11-16T16:42:38.738 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:38.736+0000 7f4451dd2700 1 -- 172.21.15.141:0/786559414 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f44380a10c0 msgr2=0x7f44380a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:38.738 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:38.736+0000 7f4451dd2700 1 --2- 172.21.15.141:0/786559414 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f44380a10c0 0x7f44380a3570 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f444c100240 tx=0x7f4434009260 comp rx=0 tx=0).stop 2023-11-16T16:42:38.739 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:38.736+0000 7f4451dd2700 1 -- 172.21.15.141:0/786559414 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f444c100d70 msgr2=0x7f444c19d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:38.739 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:38.736+0000 7f4451dd2700 1 --2- 172.21.15.141:0/786559414 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f444c100d70 0x7f444c19d3a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f444400b970 tx=0x7f444400bd30 comp rx=0 tx=0).stop 2023-11-16T16:42:38.739 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:38.736+0000 7f4451dd2700 1 -- 172.21.15.141:0/786559414 shutdown_connections 2023-11-16T16:42:38.739 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:38.736+0000 7f4451dd2700 1 --2- 172.21.15.141:0/786559414 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f44380a10c0 0x7f44380a3570 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:38.739 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:38.736+0000 7f4451dd2700 1 --2- 172.21.15.141:0/786559414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f444c0ff1e0 0x7f444c0699c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:38.739 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:38.736+0000 7f4451dd2700 1 --2- 172.21.15.141:0/786559414 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f444c100d70 0x7f444c19d3a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:38.739 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:38.736+0000 7f4451dd2700 1 --2- 172.21.15.141:0/786559414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f444c1003e0 0x7f444c19ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:38.739 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:38.736+0000 7f4451dd2700 1 -- 172.21.15.141:0/786559414 >> 172.21.15.141:0/786559414 conn(0x7f444c0fa7f0 msgr2=0x7f444c1038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:42:38.739 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:38.736+0000 7f4451dd2700 1 -- 172.21.15.141:0/786559414 shutdown_connections 2023-11-16T16:42:38.739 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:42:38.736+0000 7f4451dd2700 1 -- 172.21.15.141:0/786559414 wait complete. 2023-11-16T16:42:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:38 smithi141 ceph-mon[131184]: pgmap v57: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:38 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:38 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:38 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:38 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:38 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:38 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:38 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:39.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:38 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:39.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:38 smithi155 ceph-mon[131607]: pgmap v57: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:39.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:38 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:39.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:38 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:39.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:38 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:39.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:38 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:39.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:38 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:39.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:38 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:39.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:38 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:39.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:38 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:39.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:38 smithi002 ceph-mon[128712]: pgmap v57: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:39.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:38 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:39.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:38 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:39.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:38 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:39.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:38 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:39.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:38 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:39.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:38 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:39.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:38 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:39.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:38 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:39.237 DEBUG:teuthology.orchestra.run.smithi141:osd.1> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.1.service 2023-11-16T16:42:39.241 INFO:tasks.cephadm:Deploying osd.2 on smithi155 with /dev/vg_nvme/lv_4... 2023-11-16T16:42:39.241 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-11-16T16:42:39.274 INFO:journalctl@ceph.osd.1.smithi141.stdout:-- Logs begin at Thu 2023-11-16 16:14:35 UTC. -- 2023-11-16T16:42:39.890 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:42:39.973 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:39 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:42:39.973 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:39 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:42:39.973 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:39 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:42:40.062 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:39 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:42:40.063 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:39 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:42:40.063 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:39 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:42:40.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:39 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:42:40.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:39 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:42:40.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:39 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:42:40.265 INFO:journalctl@ceph.osd.1.smithi141.stdout:Nov 16 16:42:39 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-1[136252]: 2023-11-16T16:42:39.969+0000 7f45fd722640 -1 osd.1 0 log_to_monitors true 2023-11-16T16:42:41.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:40 smithi155 ceph-mon[131607]: pgmap v58: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:41.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:40 smithi155 ceph-mon[131607]: from='osd.1 [v2:172.21.15.141:6800/1040978026,v1:172.21.15.141:6801/1040978026]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T16:42:41.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:40 smithi155 ceph-mon[131607]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T16:42:41.150 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:40 smithi141 ceph-mon[131184]: pgmap v58: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:41.150 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:40 smithi141 ceph-mon[131184]: from='osd.1 [v2:172.21.15.141:6800/1040978026,v1:172.21.15.141:6801/1040978026]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T16:42:41.150 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:40 smithi141 ceph-mon[131184]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T16:42:41.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:40 smithi002 ceph-mon[128712]: pgmap v58: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:41.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:40 smithi002 ceph-mon[128712]: from='osd.1 [v2:172.21.15.141:6800/1040978026,v1:172.21.15.141:6801/1040978026]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T16:42:41.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:40 smithi002 ceph-mon[128712]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-16T16:42:41.217 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-11-16T16:42:41.217 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-11-16T16:42:41.217 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2023-11-16T16:42:41.217 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2023-11-16T16:42:41.217 INFO:teuthology.orchestra.run.smithi155.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113836 s, 921 MB/s 2023-11-16T16:42:41.217 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2023-11-16T16:42:41.633 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi155:vg_nvme/lv_4 2023-11-16T16:42:42.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:41 smithi155 ceph-mon[131607]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-11-16T16:42:42.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:41 smithi155 ceph-mon[131607]: osdmap e11: 2 total, 1 up, 2 in 2023-11-16T16:42:42.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:41 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:42.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:41 smithi155 ceph-mon[131607]: from='osd.1 [v2:172.21.15.141:6800/1040978026,v1:172.21.15.141:6801/1040978026]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:42:42.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:41 smithi155 ceph-mon[131607]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:42:42.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:41 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:42.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:41 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:42.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:41 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:42:42.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:41 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:42.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:41 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:42.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:41 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:42.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:41 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:42.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:41 smithi002 ceph-mon[128712]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-11-16T16:42:42.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:41 smithi002 ceph-mon[128712]: osdmap e11: 2 total, 1 up, 2 in 2023-11-16T16:42:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:41 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:41 smithi002 ceph-mon[128712]: from='osd.1 [v2:172.21.15.141:6800/1040978026,v1:172.21.15.141:6801/1040978026]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:42:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:41 smithi002 ceph-mon[128712]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:42:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:41 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:41 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:41 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:42:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:41 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:41 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:41 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:42.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:41 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:42.265 INFO:journalctl@ceph.osd.1.smithi141.stdout:Nov 16 16:42:41 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-1[136252]: 2023-11-16T16:42:41.805+0000 7f45f2d99700 -1 osd.1 0 waiting for initial osdmap 2023-11-16T16:42:42.265 INFO:journalctl@ceph.osd.1.smithi141.stdout:Nov 16 16:42:41 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-1[136252]: 2023-11-16T16:42:41.809+0000 7f45ecb8a700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:42:42.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:41 smithi141 ceph-mon[131184]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-11-16T16:42:42.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:41 smithi141 ceph-mon[131184]: osdmap e11: 2 total, 1 up, 2 in 2023-11-16T16:42:42.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:42.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:41 smithi141 ceph-mon[131184]: from='osd.1 [v2:172.21.15.141:6800/1040978026,v1:172.21.15.141:6801/1040978026]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:42:42.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:41 smithi141 ceph-mon[131184]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:42:42.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:42.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:42.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:42:42.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:42.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:42.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:42.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:42.277 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:42:43.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:42 smithi155 ceph-mon[131607]: Detected new or changed devices on smithi141 2023-11-16T16:42:43.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:42 smithi155 ceph-mon[131607]: Adjusting osd_memory_target on smithi141 to 17168M 2023-11-16T16:42:43.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:42 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:42:43.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:42 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:42:43.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:42 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:42:43.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:42 smithi155 ceph-mon[131607]: pgmap v60: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:43.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:42 smithi155 ceph-mon[131607]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]': finished 2023-11-16T16:42:43.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:42 smithi155 ceph-mon[131607]: osdmap e12: 2 total, 1 up, 2 in 2023-11-16T16:42:43.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:42 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:43.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:42 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:43.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:42 smithi002 ceph-mon[128712]: Detected new or changed devices on smithi141 2023-11-16T16:42:43.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:42 smithi002 ceph-mon[128712]: Adjusting osd_memory_target on smithi141 to 17168M 2023-11-16T16:42:43.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:42 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:42:43.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:42 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:42:43.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:42 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:42:43.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:42 smithi002 ceph-mon[128712]: pgmap v60: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:43.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:42 smithi002 ceph-mon[128712]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]': finished 2023-11-16T16:42:43.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:42 smithi002 ceph-mon[128712]: osdmap e12: 2 total, 1 up, 2 in 2023-11-16T16:42:43.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:42 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:43.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:42 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:43.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:42 smithi141 ceph-mon[131184]: Detected new or changed devices on smithi141 2023-11-16T16:42:43.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:42 smithi141 ceph-mon[131184]: Adjusting osd_memory_target on smithi141 to 17168M 2023-11-16T16:42:43.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:42 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:42:43.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:42 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:42:43.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:42 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:42:43.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:42 smithi141 ceph-mon[131184]: pgmap v60: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:43.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:42 smithi141 ceph-mon[131184]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]': finished 2023-11-16T16:42:43.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:42 smithi141 ceph-mon[131184]: osdmap e12: 2 total, 1 up, 2 in 2023-11-16T16:42:43.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:42 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:43.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:42 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:43.455 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.452+0000 7f625b2ee700 1 -- 172.21.15.155:0/1988547252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62540ff1e0 msgr2=0x7f62540ff5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:43.455 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.452+0000 7f625b2ee700 1 --2- 172.21.15.155:0/1988547252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62540ff1e0 0x7f62540ff5f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6244009990 tx=0x7f6244009ca0 comp rx=0 tx=0).stop 2023-11-16T16:42:43.456 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.452+0000 7f625b2ee700 1 -- 172.21.15.155:0/1988547252 shutdown_connections 2023-11-16T16:42:43.456 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.452+0000 7f625b2ee700 1 --2- 172.21.15.155:0/1988547252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62540ff1e0 0x7f62540ff5f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:43.456 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.452+0000 7f625b2ee700 1 --2- 172.21.15.155:0/1988547252 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6254100d70 0x7f6254107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:43.456 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.452+0000 7f625b2ee700 1 --2- 172.21.15.155:0/1988547252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62541003e0 0x7f6254100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:43.456 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.452+0000 7f625b2ee700 1 -- 172.21.15.155:0/1988547252 >> 172.21.15.155:0/1988547252 conn(0x7f62540fa7f0 msgr2=0x7f62540fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:42:43.456 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.452+0000 7f625b2ee700 1 -- 172.21.15.155:0/1988547252 shutdown_connections 2023-11-16T16:42:43.456 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.452+0000 7f625b2ee700 1 -- 172.21.15.155:0/1988547252 wait complete. 2023-11-16T16:42:43.456 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.452+0000 7f625b2ee700 1 Processor -- start 2023-11-16T16:42:43.457 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.452+0000 7f625b2ee700 1 -- start start 2023-11-16T16:42:43.457 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f625b2ee700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62540ff1e0 0x7f6254195000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:42:43.457 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f625b2ee700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f62541003e0 0x7f6254195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:42:43.457 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f625b2ee700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6254100d70 0x7f625419c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:42:43.457 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f625b2ee700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625419cb90 con 0x7f62540ff1e0 2023-11-16T16:42:43.457 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f625b2ee700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625419ccd0 con 0x7f62541003e0 2023-11-16T16:42:43.457 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f625b2ee700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625419ce60 con 0x7f6254100d70 2023-11-16T16:42:43.457 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f625988b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6254100d70 0x7f625419c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:42:43.457 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f625988b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6254100d70 0x7f625419c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:39024/0 (socket says 172.21.15.155:39024) 2023-11-16T16:42:43.458 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f625988b700 1 -- 172.21.15.155:0/3154725265 learned_addr learned my addr 172.21.15.155:0/3154725265 (peer_addr_for_me v2:172.21.15.155:0/0) 2023-11-16T16:42:43.458 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f6258889700 1 --2- 172.21.15.155:0/3154725265 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f62541003e0 0x7f6254195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:42:43.458 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f625988b700 1 -- 172.21.15.155:0/3154725265 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f62541003e0 msgr2=0x7f6254195540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:43.458 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f625988b700 1 --2- 172.21.15.155:0/3154725265 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f62541003e0 0x7f6254195540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:43.458 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f625988b700 1 -- 172.21.15.155:0/3154725265 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62540ff1e0 msgr2=0x7f6254195000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:43.458 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f625988b700 1 --2- 172.21.15.155:0/3154725265 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62540ff1e0 0x7f6254195000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:43.458 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f625988b700 1 -- 172.21.15.155:0/3154725265 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62440095f0 con 0x7f6254100d70 2023-11-16T16:42:43.459 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f625988b700 1 --2- 172.21.15.155:0/3154725265 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6254100d70 0x7f625419c5c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f624c007f40 tx=0x7f624c00f5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:42:43.459 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f624a7fc700 1 -- 172.21.15.155:0/3154725265 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c00fde0 con 0x7f6254100d70 2023-11-16T16:42:43.459 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f624a7fc700 1 -- 172.21.15.155:0/3154725265 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f624c004bb0 con 0x7f6254100d70 2023-11-16T16:42:43.459 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f624a7fc700 1 -- 172.21.15.155:0/3154725265 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c005020 con 0x7f6254100d70 2023-11-16T16:42:43.459 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f625b2ee700 1 -- 172.21.15.155:0/3154725265 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f625410abb0 con 0x7f6254100d70 2023-11-16T16:42:43.459 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.453+0000 7f625b2ee700 1 -- 172.21.15.155:0/3154725265 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f625410aa20 con 0x7f6254100d70 2023-11-16T16:42:43.459 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.454+0000 7f625b2ee700 1 -- 172.21.15.155:0/3154725265 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62540689c0 con 0x7f6254100d70 2023-11-16T16:42:43.460 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.456+0000 7f624a7fc700 1 -- 172.21.15.155:0/3154725265 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 15) v1 ==== 135658+0+0 (secure 0 0 0) 0x7f624c004d20 con 0x7f6254100d70 2023-11-16T16:42:43.460 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.456+0000 7f624a7fc700 1 --2- 172.21.15.155:0/3154725265 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62400a1140 0x7f62400a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:42:43.460 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.457+0000 7f624a7fc700 1 -- 172.21.15.155:0/3154725265 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f624c0cd760 con 0x7f6254100d70 2023-11-16T16:42:43.461 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.457+0000 7f625908a700 1 --2- 172.21.15.155:0/3154725265 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62400a1140 0x7f62400a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:42:43.461 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.457+0000 7f625908a700 1 --2- 172.21.15.155:0/3154725265 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62400a1140 0x7f62400a35f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f62440098e0 tx=0x7f6244003680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:42:43.461 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.460+0000 7f624a7fc700 1 -- 172.21.15.155:0/3154725265 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f624c098660 con 0x7f6254100d70 2023-11-16T16:42:43.657 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:43.656+0000 7f625b2ee700 1 -- 172.21.15.155:0/3154725265 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6254060f30 con 0x7f62400a1140 2023-11-16T16:42:44.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:43 smithi155 ceph-mon[131607]: purged_snaps scrub starts 2023-11-16T16:42:44.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:43 smithi155 ceph-mon[131607]: purged_snaps scrub ok 2023-11-16T16:42:44.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:43 smithi155 ceph-mon[131607]: osd.1 [v2:172.21.15.141:6800/1040978026,v1:172.21.15.141:6801/1040978026] boot 2023-11-16T16:42:44.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:43 smithi155 ceph-mon[131607]: osdmap e13: 2 total, 2 up, 2 in 2023-11-16T16:42:44.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:44.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:42:44.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:42:44.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:44.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:43 smithi002 ceph-mon[128712]: purged_snaps scrub starts 2023-11-16T16:42:44.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:43 smithi002 ceph-mon[128712]: purged_snaps scrub ok 2023-11-16T16:42:44.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:43 smithi002 ceph-mon[128712]: osd.1 [v2:172.21.15.141:6800/1040978026,v1:172.21.15.141:6801/1040978026] boot 2023-11-16T16:42:44.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:43 smithi002 ceph-mon[128712]: osdmap e13: 2 total, 2 up, 2 in 2023-11-16T16:42:44.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:44.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:42:44.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:42:44.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:44.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:43 smithi141 ceph-mon[131184]: purged_snaps scrub starts 2023-11-16T16:42:44.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:43 smithi141 ceph-mon[131184]: purged_snaps scrub ok 2023-11-16T16:42:44.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:43 smithi141 ceph-mon[131184]: osd.1 [v2:172.21.15.141:6800/1040978026,v1:172.21.15.141:6801/1040978026] boot 2023-11-16T16:42:44.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:43 smithi141 ceph-mon[131184]: osdmap e13: 2 total, 2 up, 2 in 2023-11-16T16:42:44.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:42:44.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:42:44.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:42:44.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:45.089 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:44 smithi155 ceph-mon[131607]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:42:45.089 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:44 smithi155 ceph-mon[131607]: pgmap v63: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:45.089 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:44 smithi155 ceph-mon[131607]: osdmap e14: 2 total, 2 up, 2 in 2023-11-16T16:42:45.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:44 smithi002 ceph-mon[128712]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:42:45.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:44 smithi002 ceph-mon[128712]: pgmap v63: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:45.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:44 smithi002 ceph-mon[128712]: osdmap e14: 2 total, 2 up, 2 in 2023-11-16T16:42:45.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:44 smithi141 ceph-mon[131184]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:42:45.286 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:44 smithi141 ceph-mon[131184]: pgmap v63: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-11-16T16:42:45.286 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:44 smithi141 ceph-mon[131184]: osdmap e14: 2 total, 2 up, 2 in 2023-11-16T16:42:47.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:46 smithi155 ceph-mon[131607]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:47.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:46 smithi155 ceph-mon[131607]: from='client.? 172.21.15.155:0/3244230602' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae"}]: dispatch 2023-11-16T16:42:47.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:46 smithi155 ceph-mon[131607]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae"}]: dispatch 2023-11-16T16:42:47.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:46 smithi155 ceph-mon[131607]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae"}]': finished 2023-11-16T16:42:47.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:46 smithi155 ceph-mon[131607]: osdmap e15: 3 total, 2 up, 3 in 2023-11-16T16:42:47.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:46 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:42:47.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:46 smithi155 ceph-mon[131607]: from='client.? 172.21.15.155:0/3277549561' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:42:47.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:46 smithi002 ceph-mon[128712]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:47.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:46 smithi002 ceph-mon[128712]: from='client.? 172.21.15.155:0/3244230602' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae"}]: dispatch 2023-11-16T16:42:47.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:46 smithi002 ceph-mon[128712]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae"}]: dispatch 2023-11-16T16:42:47.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:46 smithi002 ceph-mon[128712]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae"}]': finished 2023-11-16T16:42:47.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:46 smithi002 ceph-mon[128712]: osdmap e15: 3 total, 2 up, 3 in 2023-11-16T16:42:47.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:46 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:42:47.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:46 smithi002 ceph-mon[128712]: from='client.? 172.21.15.155:0/3277549561' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:42:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:46 smithi141 ceph-mon[131184]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:46 smithi141 ceph-mon[131184]: from='client.? 172.21.15.155:0/3244230602' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae"}]: dispatch 2023-11-16T16:42:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:46 smithi141 ceph-mon[131184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae"}]: dispatch 2023-11-16T16:42:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:46 smithi141 ceph-mon[131184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae"}]': finished 2023-11-16T16:42:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:46 smithi141 ceph-mon[131184]: osdmap e15: 3 total, 2 up, 3 in 2023-11-16T16:42:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:46 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:42:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:46 smithi141 ceph-mon[131184]: from='client.? 172.21.15.155:0/3277549561' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:42:49.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:48 smithi155 ceph-mon[131607]: pgmap v67: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:49.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:48 smithi002 ceph-mon[128712]: pgmap v67: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:49.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:48 smithi141 ceph-mon[131184]: pgmap v67: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:51.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:50 smithi155 ceph-mon[131607]: pgmap v68: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:51.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:50 smithi002 ceph-mon[128712]: pgmap v68: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:51.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:50 smithi141 ceph-mon[131184]: pgmap v68: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:52.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:52 smithi155 ceph-mon[131607]: pgmap v69: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:52.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-11-16T16:42:52.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:53.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:52 smithi002 ceph-mon[128712]: pgmap v69: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:53.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-11-16T16:42:53.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:53.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:52 smithi141 ceph-mon[131184]: pgmap v69: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:53.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-11-16T16:42:53.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:53.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:53 smithi155 ceph-mon[131607]: Deploying daemon osd.2 on smithi155 2023-11-16T16:42:54.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:53 smithi002 ceph-mon[128712]: Deploying daemon osd.2 on smithi155 2023-11-16T16:42:54.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:53 smithi141 ceph-mon[131184]: Deploying daemon osd.2 on smithi155 2023-11-16T16:42:55.109 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:54 smithi155 ceph-mon[131607]: pgmap v70: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:55.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:54 smithi002 ceph-mon[128712]: pgmap v70: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:55.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:54 smithi141 ceph-mon[131184]: pgmap v70: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:57.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:56 smithi155 ceph-mon[131607]: pgmap v71: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:57.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:56 smithi002 ceph-mon[128712]: pgmap v71: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:57.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:56 smithi141 ceph-mon[131184]: pgmap v71: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:58.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:57 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:58.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:57 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:58.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:57 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:58.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:57 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:58.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:57 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:58.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:57 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:58.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:57 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:58.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:57 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:58.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:57 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:42:59.035 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:58 smithi155 ceph-mon[131607]: pgmap v72: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:59.035 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:58 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:59.036 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:58 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:59.036 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:58 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:59.036 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:58 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:59.036 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:58 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:58 smithi141 ceph-mon[131184]: pgmap v72: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:58 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:58 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:58 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:58 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:58 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:59.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:58 smithi002 ceph-mon[128712]: pgmap v72: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:42:59.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:58 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:59.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:58 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:59.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:58 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:42:59.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:58 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:42:59.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:58 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:42:59.876 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:59.873+0000 7f624a7fc700 1 -- 172.21.15.155:0/3154725265 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6254060f30 con 0x7f62400a1140 2023-11-16T16:42:59.877 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 2 on host 'smithi155' 2023-11-16T16:42:59.879 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:59.876+0000 7f625b2ee700 1 -- 172.21.15.155:0/3154725265 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62400a1140 msgr2=0x7f62400a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:59.879 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:59.876+0000 7f625b2ee700 1 --2- 172.21.15.155:0/3154725265 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62400a1140 0x7f62400a35f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f62440098e0 tx=0x7f6244003680 comp rx=0 tx=0).stop 2023-11-16T16:42:59.879 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:59.876+0000 7f625b2ee700 1 -- 172.21.15.155:0/3154725265 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6254100d70 msgr2=0x7f625419c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:42:59.879 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:59.876+0000 7f625b2ee700 1 --2- 172.21.15.155:0/3154725265 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6254100d70 0x7f625419c5c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f624c007f40 tx=0x7f624c00f5c0 comp rx=0 tx=0).stop 2023-11-16T16:42:59.879 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:59.876+0000 7f625b2ee700 1 -- 172.21.15.155:0/3154725265 shutdown_connections 2023-11-16T16:42:59.879 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:59.876+0000 7f625b2ee700 1 --2- 172.21.15.155:0/3154725265 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62400a1140 0x7f62400a35f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:59.879 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:59.876+0000 7f625b2ee700 1 --2- 172.21.15.155:0/3154725265 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6254100d70 0x7f625419c5c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:59.880 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:59.876+0000 7f625b2ee700 1 --2- 172.21.15.155:0/3154725265 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62540ff1e0 0x7f6254195000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:59.880 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:59.876+0000 7f625b2ee700 1 --2- 172.21.15.155:0/3154725265 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f62541003e0 0x7f6254195540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:42:59.880 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:59.876+0000 7f625b2ee700 1 -- 172.21.15.155:0/3154725265 >> 172.21.15.155:0/3154725265 conn(0x7f62540fa7f0 msgr2=0x7f62540fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:42:59.880 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:59.876+0000 7f625b2ee700 1 -- 172.21.15.155:0/3154725265 shutdown_connections 2023-11-16T16:42:59.880 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:42:59.876+0000 7f625b2ee700 1 -- 172.21.15.155:0/3154725265 wait complete. 2023-11-16T16:43:00.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:59 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:43:00.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:59 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:43:00.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:59 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:43:00.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:59 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:00.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:59 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:00.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:42:59 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:00.251 DEBUG:teuthology.orchestra.run.smithi155:osd.2> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.2.service 2023-11-16T16:43:00.253 INFO:tasks.cephadm:Deploying osd.3 on smithi002 with /dev/vg_nvme/lv_3... 2023-11-16T16:43:00.253 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-11-16T16:43:00.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:59 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:43:00.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:59 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:43:00.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:59 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:43:00.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:59 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:00.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:59 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:00.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:42:59 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:00.282 INFO:journalctl@ceph.osd.2.smithi155.stdout:-- Logs begin at Thu 2023-11-16 16:15:48 UTC. -- 2023-11-16T16:43:00.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:59 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:43:00.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:59 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:43:00.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:59 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:43:00.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:59 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:00.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:59 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:00.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:42:59 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:01.118 INFO:journalctl@ceph.osd.2.smithi155.stdout:Nov 16 16:43:00 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-2[137415]: 2023-11-16T16:43:00.799+0000 7f4281976640 -1 osd.2 0 log_to_monitors true 2023-11-16T16:43:01.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:00 smithi155 ceph-mon[131607]: pgmap v73: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:43:01.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:00 smithi155 ceph-mon[131607]: from='osd.2 [v2:172.21.15.155:6800/844890823,v1:172.21.15.155:6801/844890823]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T16:43:01.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:00 smithi155 ceph-mon[131607]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T16:43:01.122 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:43:01.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:00 smithi141 ceph-mon[131184]: pgmap v73: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:43:01.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:00 smithi141 ceph-mon[131184]: from='osd.2 [v2:172.21.15.155:6800/844890823,v1:172.21.15.155:6801/844890823]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T16:43:01.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:00 smithi141 ceph-mon[131184]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T16:43:01.384 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:00 smithi002 ceph-mon[128712]: pgmap v73: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:43:01.384 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:00 smithi002 ceph-mon[128712]: from='osd.2 [v2:172.21.15.155:6800/844890823,v1:172.21.15.155:6801/844890823]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T16:43:01.384 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:00 smithi002 ceph-mon[128712]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-16T16:43:02.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:01 smithi002 ceph-mon[128712]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-11-16T16:43:02.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:01 smithi002 ceph-mon[128712]: osdmap e16: 3 total, 2 up, 3 in 2023-11-16T16:43:02.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:01 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:02.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:01 smithi002 ceph-mon[128712]: from='osd.2 [v2:172.21.15.155:6800/844890823,v1:172.21.15.155:6801/844890823]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:43:02.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:01 smithi002 ceph-mon[128712]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:43:02.206 INFO:journalctl@ceph.osd.2.smithi155.stdout:Nov 16 16:43:01 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-2[137415]: 2023-11-16T16:43:01.937+0000 7f4276fed700 -1 osd.2 0 waiting for initial osdmap 2023-11-16T16:43:02.207 INFO:journalctl@ceph.osd.2.smithi155.stdout:Nov 16 16:43:01 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-2[137415]: 2023-11-16T16:43:01.942+0000 7f42725e1700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:43:02.207 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:01 smithi155 ceph-mon[131607]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-11-16T16:43:02.207 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:01 smithi155 ceph-mon[131607]: osdmap e16: 3 total, 2 up, 3 in 2023-11-16T16:43:02.207 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:01 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:02.207 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:01 smithi155 ceph-mon[131607]: from='osd.2 [v2:172.21.15.155:6800/844890823,v1:172.21.15.155:6801/844890823]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:43:02.207 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:01 smithi155 ceph-mon[131607]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:43:02.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:01 smithi141 ceph-mon[131184]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-11-16T16:43:02.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:01 smithi141 ceph-mon[131184]: osdmap e16: 3 total, 2 up, 3 in 2023-11-16T16:43:02.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:01 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:02.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:01 smithi141 ceph-mon[131184]: from='osd.2 [v2:172.21.15.155:6800/844890823,v1:172.21.15.155:6801/844890823]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:43:02.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:01 smithi141 ceph-mon[131184]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:43:03.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:02 smithi141 ceph-mon[131184]: pgmap v75: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:43:03.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:02 smithi141 ceph-mon[131184]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2023-11-16T16:43:03.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:02 smithi141 ceph-mon[131184]: osdmap e17: 3 total, 2 up, 3 in 2023-11-16T16:43:03.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:02 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:03.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:02 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:03.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:02 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:03.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:02 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:03.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:02 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:03.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:02 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:03.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:02 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:03.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:02 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:03.267 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:02 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:03.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:02 smithi155 ceph-mon[131607]: pgmap v75: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:43:03.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:02 smithi155 ceph-mon[131607]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2023-11-16T16:43:03.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:02 smithi155 ceph-mon[131607]: osdmap e17: 3 total, 2 up, 3 in 2023-11-16T16:43:03.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:02 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:03.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:02 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:03.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:02 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:03.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:02 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:03.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:02 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:03.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:02 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:03.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:02 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:03.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:02 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:03.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:02 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:03.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:02 smithi002 ceph-mon[128712]: pgmap v75: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:43:03.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:02 smithi002 ceph-mon[128712]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2023-11-16T16:43:03.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:02 smithi002 ceph-mon[128712]: osdmap e17: 3 total, 2 up, 3 in 2023-11-16T16:43:03.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:02 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:03.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:02 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:03.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:02 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:03.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:02 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:03.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:02 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:03.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:02 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:03.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:02 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:03.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:02 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:03.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:02 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:03.441 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-11-16T16:43:03.442 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-11-16T16:43:03.442 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2023-11-16T16:43:03.442 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2023-11-16T16:43:03.442 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0312531 s, 336 MB/s 2023-11-16T16:43:03.442 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2023-11-16T16:43:04.005 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi002:vg_nvme/lv_3 2023-11-16T16:43:04.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:04 smithi155 ceph-mon[131607]: Detected new or changed devices on smithi155 2023-11-16T16:43:04.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:04 smithi155 ceph-mon[131607]: Adjusting osd_memory_target on smithi155 to 17168M 2023-11-16T16:43:04.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:04 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:43:04.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:04 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:43:04.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:04 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:43:04.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:04 smithi155 ceph-mon[131607]: osd.2 [v2:172.21.15.155:6800/844890823,v1:172.21.15.155:6801/844890823] boot 2023-11-16T16:43:04.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:04 smithi155 ceph-mon[131607]: osdmap e18: 3 total, 3 up, 3 in 2023-11-16T16:43:04.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:04 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:04.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:04 smithi002 ceph-mon[128712]: Detected new or changed devices on smithi155 2023-11-16T16:43:04.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:04 smithi002 ceph-mon[128712]: Adjusting osd_memory_target on smithi155 to 17168M 2023-11-16T16:43:04.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:04 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:43:04.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:04 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:43:04.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:04 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:43:04.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:04 smithi002 ceph-mon[128712]: osd.2 [v2:172.21.15.155:6800/844890823,v1:172.21.15.155:6801/844890823] boot 2023-11-16T16:43:04.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:04 smithi002 ceph-mon[128712]: osdmap e18: 3 total, 3 up, 3 in 2023-11-16T16:43:04.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:04 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:04.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:04 smithi141 ceph-mon[131184]: Detected new or changed devices on smithi155 2023-11-16T16:43:04.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:04 smithi141 ceph-mon[131184]: Adjusting osd_memory_target on smithi155 to 17168M 2023-11-16T16:43:04.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:04 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:43:04.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:04 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:43:04.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:04 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:43:04.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:04 smithi141 ceph-mon[131184]: osd.2 [v2:172.21.15.155:6800/844890823,v1:172.21.15.155:6801/844890823] boot 2023-11-16T16:43:04.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:04 smithi141 ceph-mon[131184]: osdmap e18: 3 total, 3 up, 3 in 2023-11-16T16:43:04.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:04 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:43:04.867 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:43:05.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:05 smithi002 ceph-mon[128712]: purged_snaps scrub starts 2023-11-16T16:43:05.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:05 smithi002 ceph-mon[128712]: purged_snaps scrub ok 2023-11-16T16:43:05.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:05 smithi002 ceph-mon[128712]: pgmap v78: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:43:05.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:05 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-11-16T16:43:05.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:05 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-11-16T16:43:05.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:05 smithi002 ceph-mon[128712]: osdmap e19: 3 total, 3 up, 3 in 2023-11-16T16:43:05.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:05 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T16:43:05.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:05 smithi155 ceph-mon[131607]: purged_snaps scrub starts 2023-11-16T16:43:05.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:05 smithi155 ceph-mon[131607]: purged_snaps scrub ok 2023-11-16T16:43:05.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:05 smithi155 ceph-mon[131607]: pgmap v78: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:43:05.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:05 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-11-16T16:43:05.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:05 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-11-16T16:43:05.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:05 smithi155 ceph-mon[131607]: osdmap e19: 3 total, 3 up, 3 in 2023-11-16T16:43:05.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:05 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T16:43:05.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:05 smithi141 ceph-mon[131184]: purged_snaps scrub starts 2023-11-16T16:43:05.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:05 smithi141 ceph-mon[131184]: purged_snaps scrub ok 2023-11-16T16:43:05.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:05 smithi141 ceph-mon[131184]: pgmap v78: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-11-16T16:43:05.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:05 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-11-16T16:43:05.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:05 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-11-16T16:43:05.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:05 smithi141 ceph-mon[131184]: osdmap e19: 3 total, 3 up, 3 in 2023-11-16T16:43:05.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:05 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T16:43:06.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.438+0000 7ffa0a93e700 1 -- 172.21.15.2:0/1458914761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa04101f00 msgr2=0x7ffa041087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:06.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.438+0000 7ffa0a93e700 1 --2- 172.21.15.2:0/1458914761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa04101f00 0x7ffa041087c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00b0a0 tx=0x7ff9fc00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:43:06.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.439+0000 7ffa0a93e700 1 -- 172.21.15.2:0/1458914761 shutdown_connections 2023-11-16T16:43:06.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.439+0000 7ffa0a93e700 1 --2- 172.21.15.2:0/1458914761 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa04100370 0x7ffa04100780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:06.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.439+0000 7ffa0a93e700 1 --2- 172.21.15.2:0/1458914761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa04101f00 0x7ffa041087c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:06.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.439+0000 7ffa0a93e700 1 --2- 172.21.15.2:0/1458914761 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffa04101570 0x7ffa041019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:06.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.439+0000 7ffa0a93e700 1 -- 172.21.15.2:0/1458914761 >> 172.21.15.2:0/1458914761 conn(0x7ffa040fb9a0 msgr2=0x7ffa040fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:43:06.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.439+0000 7ffa0a93e700 1 -- 172.21.15.2:0/1458914761 shutdown_connections 2023-11-16T16:43:06.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.439+0000 7ffa0a93e700 1 -- 172.21.15.2:0/1458914761 wait complete. 2023-11-16T16:43:06.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.440+0000 7ffa0a93e700 1 Processor -- start 2023-11-16T16:43:06.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.440+0000 7ffa0a93e700 1 -- start start 2023-11-16T16:43:06.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.441+0000 7ffa0a93e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa04100370 0x7ffa04074890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:06.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.441+0000 7ffa0a93e700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffa04101570 0x7ffa04072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:06.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.441+0000 7ffa0a93e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa04101f00 0x7ffa04073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:06.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.441+0000 7ffa0a93e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa04073ba0 con 0x7ffa04101f00 2023-11-16T16:43:06.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.441+0000 7ffa0a93e700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa04073d10 con 0x7ffa04101570 2023-11-16T16:43:06.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.441+0000 7ffa0a93e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa041a01a0 con 0x7ffa04100370 2023-11-16T16:43:06.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.441+0000 7ffa03fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa04100370 0x7ffa04074890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:06.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.441+0000 7ffa03fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa04100370 0x7ffa04074890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:50654/0 (socket says 172.21.15.2:50654) 2023-11-16T16:43:06.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.441+0000 7ffa08edb700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa04101f00 0x7ffa04073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:06.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.441+0000 7ffa03fff700 1 -- 172.21.15.2:0/2961516947 learned_addr learned my addr 172.21.15.2:0/2961516947 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:43:06.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.441+0000 7ffa037fe700 1 --2- 172.21.15.2:0/2961516947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffa04101570 0x7ffa04072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:06.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.442+0000 7ffa03fff700 1 -- 172.21.15.2:0/2961516947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffa04101570 msgr2=0x7ffa04072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:06.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.442+0000 7ffa03fff700 1 --2- 172.21.15.2:0/2961516947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffa04101570 0x7ffa04072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:06.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.442+0000 7ffa03fff700 1 -- 172.21.15.2:0/2961516947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa04101f00 msgr2=0x7ffa04073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:06.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.442+0000 7ffa03fff700 1 --2- 172.21.15.2:0/2961516947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa04101f00 0x7ffa04073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:06.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.442+0000 7ffa03fff700 1 -- 172.21.15.2:0/2961516947 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f8009520 con 0x7ffa04100370 2023-11-16T16:43:06.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.442+0000 7ffa03fff700 1 --2- 172.21.15.2:0/2961516947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa04100370 0x7ffa04074890 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff9f400e970 tx=0x7ff9f400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:43:06.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.442+0000 7ffa017fa700 1 -- 172.21.15.2:0/2961516947 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4009790 con 0x7ffa04100370 2023-11-16T16:43:06.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.442+0000 7ffa017fa700 1 -- 172.21.15.2:0/2961516947 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9f400eec0 con 0x7ffa04100370 2023-11-16T16:43:06.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.442+0000 7ffa0a93e700 1 -- 172.21.15.2:0/2961516947 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9fc009d00 con 0x7ffa04100370 2023-11-16T16:43:06.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.442+0000 7ffa017fa700 1 -- 172.21.15.2:0/2961516947 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4003cf0 con 0x7ffa04100370 2023-11-16T16:43:06.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.442+0000 7ffa0a93e700 1 -- 172.21.15.2:0/2961516947 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa0410bd10 con 0x7ffa04100370 2023-11-16T16:43:06.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.443+0000 7ffa0a93e700 1 -- 172.21.15.2:0/2961516947 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa0410bea0 con 0x7ffa04100370 2023-11-16T16:43:06.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.446+0000 7ffa017fa700 1 -- 172.21.15.2:0/2961516947 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 15) v1 ==== 135658+0+0 (secure 0 0 0) 0x7ff9f400ccd0 con 0x7ffa04100370 2023-11-16T16:43:06.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.447+0000 7ffa017fa700 1 --2- 172.21.15.2:0/2961516947 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff9ec0a1140 0x7ff9ec0a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:06.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.447+0000 7ffa017fa700 1 -- 172.21.15.2:0/2961516947 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3237+0+0 (secure 0 0 0) 0x7ff9f4014070 con 0x7ffa04100370 2023-11-16T16:43:06.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.447+0000 7ffa037fe700 1 --2- 172.21.15.2:0/2961516947 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff9ec0a1140 0x7ff9ec0a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:06.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.448+0000 7ffa037fe700 1 --2- 172.21.15.2:0/2961516947 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff9ec0a1140 0x7ff9ec0a35f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8009d80 tx=0x7ff9f8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:43:06.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.449+0000 7ffa017fa700 1 -- 172.21.15.2:0/2961516947 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff9f4096390 con 0x7ffa04100370 2023-11-16T16:43:06.504 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 16:43:06 smithi002 sudo[143825]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T16:43:06.504 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 16:43:06 smithi002 sudo[143825]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:06.504 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 16:43:06 smithi002 sudo[143825]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:06.504 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 16:43:06 smithi002 sudo[143825]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:06.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.510+0000 7ffa017fa700 1 -- 172.21.15.2:0/2961516947 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7ff9f4096760 con 0x7ffa04100370 2023-11-16T16:43:06.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:06.668+0000 7ffa0a93e700 1 -- 172.21.15.2:0/2961516947 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa04060f30 con 0x7ff9ec0a1140 2023-11-16T16:43:06.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:06 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-11-16T16:43:06.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:06 smithi141 ceph-mon[131184]: osdmap e20: 3 total, 3 up, 3 in 2023-11-16T16:43:06.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:06 smithi141 ceph-mon[131184]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:06.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:06 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-11-16T16:43:06.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:06 smithi155 ceph-mon[131607]: osdmap e20: 3 total, 3 up, 3 in 2023-11-16T16:43:06.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:06 smithi155 ceph-mon[131607]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:06.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:06 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-11-16T16:43:06.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:06 smithi002 ceph-mon[128712]: osdmap e20: 3 total, 3 up, 3 in 2023-11-16T16:43:06.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:06 smithi002 ceph-mon[128712]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:06.906 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 16:43:06 smithi002 sudo[143893]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T16:43:06.906 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 16:43:06 smithi002 sudo[143893]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:06.906 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 16:43:06 smithi002 sudo[143893]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:06.907 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 16:43:06 smithi002 sudo[143893]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:07.257 INFO:journalctl@ceph.osd.1.smithi141.stdout:Nov 16 16:43:07 smithi141 sudo[138486]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T16:43:07.257 INFO:journalctl@ceph.osd.1.smithi141.stdout:Nov 16 16:43:07 smithi141 sudo[138486]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:07.257 INFO:journalctl@ceph.osd.1.smithi141.stdout:Nov 16 16:43:07 smithi141 sudo[138486]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:07.257 INFO:journalctl@ceph.osd.1.smithi141.stdout:Nov 16 16:43:07 smithi141 sudo[138486]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:07.515 INFO:journalctl@ceph.osd.1.smithi141.stdout:Nov 16 16:43:07 smithi141 sudo[138489]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T16:43:07.515 INFO:journalctl@ceph.osd.1.smithi141.stdout:Nov 16 16:43:07 smithi141 sudo[138489]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:07.515 INFO:journalctl@ceph.osd.1.smithi141.stdout:Nov 16 16:43:07 smithi141 sudo[138489]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:07.515 INFO:journalctl@ceph.osd.1.smithi141.stdout:Nov 16 16:43:07 smithi141 sudo[138489]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:07.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:07 smithi141 ceph-mon[131184]: osdmap e21: 3 total, 3 up, 3 in 2023-11-16T16:43:07.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:07 smithi141 ceph-mon[131184]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-11-16T16:43:07.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:07 smithi141 ceph-mon[131184]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:43:07.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:07 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:43:07.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:07 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:43:07.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:07 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:07.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:07 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:07.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:07 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:07.777 INFO:journalctl@ceph.osd.2.smithi155.stdout:Nov 16 16:43:07 smithi155 sudo[139549]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T16:43:07.777 INFO:journalctl@ceph.osd.2.smithi155.stdout:Nov 16 16:43:07 smithi155 sudo[139549]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:07.777 INFO:journalctl@ceph.osd.2.smithi155.stdout:Nov 16 16:43:07 smithi155 sudo[139549]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:07.778 INFO:journalctl@ceph.osd.2.smithi155.stdout:Nov 16 16:43:07 smithi155 sudo[139549]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:07.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:07 smithi155 ceph-mon[131607]: osdmap e21: 3 total, 3 up, 3 in 2023-11-16T16:43:07.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:07 smithi155 ceph-mon[131607]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-11-16T16:43:07.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:07 smithi155 ceph-mon[131607]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:43:07.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:07 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:43:07.778 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:07 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:43:07.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:07 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:07.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:07 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:07.779 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:07 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:07.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:07 smithi002 ceph-mon[128712]: osdmap e21: 3 total, 3 up, 3 in 2023-11-16T16:43:07.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:07 smithi002 ceph-mon[128712]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-11-16T16:43:07.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:07 smithi002 ceph-mon[128712]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:43:07.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:07 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:43:07.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:07 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:43:07.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:07 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:07.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:07 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:07.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:07 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:08.117 INFO:journalctl@ceph.osd.2.smithi155.stdout:Nov 16 16:43:07 smithi155 sudo[139552]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T16:43:08.118 INFO:journalctl@ceph.osd.2.smithi155.stdout:Nov 16 16:43:07 smithi155 sudo[139552]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:08.118 INFO:journalctl@ceph.osd.2.smithi155.stdout:Nov 16 16:43:07 smithi155 sudo[139552]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:08.118 INFO:journalctl@ceph.osd.2.smithi155.stdout:Nov 16 16:43:07 smithi155 sudo[139552]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:08.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:08 smithi002 sudo[144049]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T16:43:08.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:08 smithi002 sudo[144049]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:08.567 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:08 smithi002 sudo[144049]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:08.568 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:08 smithi002 sudo[144049]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:09.180 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:08 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:09.180 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:08 smithi155 ceph-mon[131607]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T16:43:09.180 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 sudo[139555]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T16:43:09.180 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 sudo[139555]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:09.180 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 sudo[139555]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:09.180 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 sudo[139555]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:09.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:08 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:09.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:08 smithi141 ceph-mon[131184]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T16:43:09.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:08 smithi002 sudo[144076]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T16:43:09.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:08 smithi002 sudo[144076]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:09.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:08 smithi002 sudo[144076]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:09.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:08 smithi002 sudo[144076]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:09.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:08 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:09.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:08 smithi002 ceph-mon[128712]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T16:43:09.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 sudo[139558]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T16:43:09.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 sudo[139558]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:09.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 sudo[139558]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:09.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 sudo[139558]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:09.909 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 sudo[138492]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-16T16:43:09.909 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 sudo[138492]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:09.910 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 sudo[138492]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:09.910 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 sudo[138492]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:10.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 sudo[138495]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-16T16:43:10.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 sudo[138495]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-16T16:43:10.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 sudo[138495]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-16T16:43:10.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 sudo[138495]: pam_unix(sudo:session): session closed for user root 2023-11-16T16:43:10.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 ceph-mon[131184]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:10.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 ceph-mon[131184]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T16:43:10.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:43:10.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:43:10.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:43:10.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 ceph-mon[131184]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T16:43:10.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:43:10.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:43:10.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:43:10.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 ceph-mon[131184]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T16:43:10.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 ceph-mon[131184]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T16:43:10.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:43:10.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:43:10.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:43:10.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:09 smithi141 ceph-mon[131184]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T16:43:10.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 ceph-mon[131607]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:10.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 ceph-mon[131607]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T16:43:10.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:43:10.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:43:10.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:43:10.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 ceph-mon[131607]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T16:43:10.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:43:10.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:43:10.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:43:10.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 ceph-mon[131607]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T16:43:10.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 ceph-mon[131607]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T16:43:10.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:43:10.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:43:10.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:43:10.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:09 smithi155 ceph-mon[131607]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T16:43:10.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:09 smithi002 ceph-mon[128712]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:10.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:09 smithi002 ceph-mon[128712]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T16:43:10.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:09 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:43:10.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:09 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:43:10.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:09 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:43:10.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:09 smithi002 ceph-mon[128712]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T16:43:10.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:09 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:43:10.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:09 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:43:10.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:09 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:43:10.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:09 smithi002 ceph-mon[128712]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T16:43:10.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:09 smithi002 ceph-mon[128712]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-16T16:43:10.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:09 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:43:10.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:09 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:43:10.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:09 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:43:10.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:09 smithi002 ceph-mon[128712]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-16T16:43:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:10 smithi141 ceph-mon[131184]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:10 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1186898731' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "374aea7d-57ed-49b1-921a-c78d31478c37"}]: dispatch 2023-11-16T16:43:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:10 smithi141 ceph-mon[131184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "374aea7d-57ed-49b1-921a-c78d31478c37"}]: dispatch 2023-11-16T16:43:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:10 smithi141 ceph-mon[131184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "374aea7d-57ed-49b1-921a-c78d31478c37"}]': finished 2023-11-16T16:43:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:10 smithi141 ceph-mon[131184]: osdmap e22: 4 total, 3 up, 4 in 2023-11-16T16:43:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:10 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:11.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:10 smithi155 ceph-mon[131607]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:11.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:10 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1186898731' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "374aea7d-57ed-49b1-921a-c78d31478c37"}]: dispatch 2023-11-16T16:43:11.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:10 smithi155 ceph-mon[131607]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "374aea7d-57ed-49b1-921a-c78d31478c37"}]: dispatch 2023-11-16T16:43:11.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:10 smithi155 ceph-mon[131607]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "374aea7d-57ed-49b1-921a-c78d31478c37"}]': finished 2023-11-16T16:43:11.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:10 smithi155 ceph-mon[131607]: osdmap e22: 4 total, 3 up, 4 in 2023-11-16T16:43:11.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:10 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:11.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:10 smithi002 ceph-mon[128712]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:11.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:10 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1186898731' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "374aea7d-57ed-49b1-921a-c78d31478c37"}]: dispatch 2023-11-16T16:43:11.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:10 smithi002 ceph-mon[128712]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "374aea7d-57ed-49b1-921a-c78d31478c37"}]: dispatch 2023-11-16T16:43:11.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:10 smithi002 ceph-mon[128712]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "374aea7d-57ed-49b1-921a-c78d31478c37"}]': finished 2023-11-16T16:43:11.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:10 smithi002 ceph-mon[128712]: osdmap e22: 4 total, 3 up, 4 in 2023-11-16T16:43:11.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:12.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:11 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/313021763' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:43:12.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:11 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/313021763' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:43:12.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:11 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/313021763' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:43:13.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:12 smithi141 ceph-mon[131184]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:13.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:12 smithi155 ceph-mon[131607]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:13.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:12 smithi002 ceph-mon[128712]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:14 smithi141 ceph-mon[131184]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:15.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:14 smithi155 ceph-mon[131607]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:15.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:14 smithi002 ceph-mon[128712]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:17.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:16 smithi002 ceph-mon[128712]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2023-11-16T16:43:17.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:16 smithi141 ceph-mon[131184]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2023-11-16T16:43:17.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:16 smithi155 ceph-mon[131607]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2023-11-16T16:43:18.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:17 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-11-16T16:43:18.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:17 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:18.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:17 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-11-16T16:43:18.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:17 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:18.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:17 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-11-16T16:43:18.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:17 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:18 smithi141 ceph-mon[131184]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2023-11-16T16:43:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:18 smithi141 ceph-mon[131184]: Deploying daemon osd.3 on smithi002 2023-11-16T16:43:19.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:18 smithi155 ceph-mon[131607]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2023-11-16T16:43:19.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:18 smithi155 ceph-mon[131607]: Deploying daemon osd.3 on smithi002 2023-11-16T16:43:19.379 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:18 smithi002 ceph-mon[128712]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2023-11-16T16:43:19.379 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:18 smithi002 ceph-mon[128712]: Deploying daemon osd.3 on smithi002 2023-11-16T16:43:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:20 smithi141 ceph-mon[131184]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2023-11-16T16:43:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:20 smithi155 ceph-mon[131607]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2023-11-16T16:43:21.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:20 smithi002 ceph-mon[128712]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2023-11-16T16:43:23.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:22 smithi141 ceph-mon[131184]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 883 B/s rd, 23 KiB/s wr, 2 op/s 2023-11-16T16:43:23.362 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:22 smithi002 ceph-mon[128712]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 883 B/s rd, 23 KiB/s wr, 2 op/s 2023-11-16T16:43:23.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:22 smithi155 ceph-mon[131607]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 883 B/s rd, 23 KiB/s wr, 2 op/s 2023-11-16T16:43:25.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:24 smithi002 ceph-mon[128712]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-11-16T16:43:25.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:24 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:25.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:24 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:25.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:24 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:25.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:24 smithi141 ceph-mon[131184]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-11-16T16:43:25.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:24 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:25.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:24 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:25.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:24 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:25.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:24 smithi155 ceph-mon[131607]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-11-16T16:43:25.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:24 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:25.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:24 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:25.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:24 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:27.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:26 smithi002 ceph-mon[128712]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-11-16T16:43:27.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:26 smithi141 ceph-mon[131184]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-11-16T16:43:27.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:26 smithi155 ceph-mon[131607]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-11-16T16:43:28.038 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:28.035+0000 7ffa017fa700 1 -- 172.21.15.2:0/2961516947 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ffa04060f30 con 0x7ff9ec0a1140 2023-11-16T16:43:28.038 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 3 on host 'smithi002' 2023-11-16T16:43:28.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:28.038+0000 7ffa0a93e700 1 -- 172.21.15.2:0/2961516947 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff9ec0a1140 msgr2=0x7ff9ec0a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:28.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:28.038+0000 7ffa0a93e700 1 --2- 172.21.15.2:0/2961516947 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff9ec0a1140 0x7ff9ec0a35f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8009d80 tx=0x7ff9f8009260 comp rx=0 tx=0).stop 2023-11-16T16:43:28.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:28.038+0000 7ffa0a93e700 1 -- 172.21.15.2:0/2961516947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa04100370 msgr2=0x7ffa04074890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:28.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:28.038+0000 7ffa0a93e700 1 --2- 172.21.15.2:0/2961516947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa04100370 0x7ffa04074890 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff9f400e970 tx=0x7ff9f400ec80 comp rx=0 tx=0).stop 2023-11-16T16:43:28.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:28.038+0000 7ffa0a93e700 1 -- 172.21.15.2:0/2961516947 shutdown_connections 2023-11-16T16:43:28.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:28.038+0000 7ffa0a93e700 1 --2- 172.21.15.2:0/2961516947 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff9ec0a1140 0x7ff9ec0a35f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:28.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:28.038+0000 7ffa0a93e700 1 --2- 172.21.15.2:0/2961516947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa04100370 0x7ffa04074890 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:28.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:28.038+0000 7ffa0a93e700 1 --2- 172.21.15.2:0/2961516947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa04101f00 0x7ffa04073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:28.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:28.038+0000 7ffa0a93e700 1 --2- 172.21.15.2:0/2961516947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffa04101570 0x7ffa04072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:28.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:28.038+0000 7ffa0a93e700 1 -- 172.21.15.2:0/2961516947 >> 172.21.15.2:0/2961516947 conn(0x7ffa040fb9a0 msgr2=0x7ffa040fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:43:28.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:28.038+0000 7ffa0a93e700 1 -- 172.21.15.2:0/2961516947 shutdown_connections 2023-11-16T16:43:28.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:43:28.038+0000 7ffa0a93e700 1 -- 172.21.15.2:0/2961516947 wait complete. 2023-11-16T16:43:28.514 DEBUG:teuthology.orchestra.run.smithi002:osd.3> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.3.service 2023-11-16T16:43:28.521 INFO:tasks.cephadm:Deploying osd.4 on smithi141 with /dev/vg_nvme/lv_3... 2023-11-16T16:43:28.522 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-11-16T16:43:28.599 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:28 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:28.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:28 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:28.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:28 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:28.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:28 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:28.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:28 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:43:28.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:28 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:28.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:28 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:43:28.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:28 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:43:28.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:28 smithi002 ceph-mon[128712]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:28.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:28 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:28.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:28 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:28.601 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:28 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:28.613 INFO:journalctl@ceph.osd.3.smithi002.stdout:-- Logs begin at Thu 2023-11-16 16:19:53 UTC. -- 2023-11-16T16:43:28.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:28 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:28.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:28 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:28.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:28 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:28.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:28 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:28.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:28 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:43:28.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:28 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:28.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:28 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:43:28.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:28 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:43:28.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:28 smithi155 ceph-mon[131607]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:28.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:28 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:28.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:28 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:28.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:28 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:28.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:28 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:28.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:28 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:28.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:28 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:28.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:28 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:28.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:28 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:43:28.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:28 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:28.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:28 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:43:28.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:28 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:43:28.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:28 smithi141 ceph-mon[131184]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:28.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:28 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:28.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:28 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:28.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:28 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:29.179 INFO:teuthology.orchestra.run.smithi141.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T16:43:29.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:29 smithi002 ceph-mon[128712]: from='osd.3 [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-16T16:43:29.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:29 smithi155 ceph-mon[131607]: from='osd.3 [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-16T16:43:29.692 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:29 smithi141 ceph-mon[131184]: from='osd.3 [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-16T16:43:30.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:30 smithi141 ceph-mon[131184]: from='osd.3 [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-11-16T16:43:30.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:30 smithi141 ceph-mon[131184]: osdmap e23: 4 total, 3 up, 4 in 2023-11-16T16:43:30.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:30 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:30.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:30 smithi141 ceph-mon[131184]: from='osd.3 [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T16:43:30.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:30 smithi141 ceph-mon[131184]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:30.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:30 smithi155 ceph-mon[131607]: from='osd.3 [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-11-16T16:43:30.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:30 smithi155 ceph-mon[131607]: osdmap e23: 4 total, 3 up, 4 in 2023-11-16T16:43:30.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:30 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:30.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:30 smithi155 ceph-mon[131607]: from='osd.3 [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T16:43:30.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:30 smithi155 ceph-mon[131607]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:30.656 INFO:journalctl@ceph.osd.3.smithi002.stdout:Nov 16 16:43:30 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-3[146408]: 2023-11-16T16:43:30.368+0000 7fbd7517b700 -1 osd.3 0 waiting for initial osdmap 2023-11-16T16:43:30.656 INFO:journalctl@ceph.osd.3.smithi002.stdout:Nov 16 16:43:30 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-3[146408]: 2023-11-16T16:43:30.375+0000 7fbd6ff6e700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:43:30.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:30 smithi002 ceph-mon[128712]: from='osd.3 [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-11-16T16:43:30.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:30 smithi002 ceph-mon[128712]: osdmap e23: 4 total, 3 up, 4 in 2023-11-16T16:43:30.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:30 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:30.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:30 smithi002 ceph-mon[128712]: from='osd.3 [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T16:43:30.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:30 smithi002 ceph-mon[128712]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-11-16T16:43:30.715 INFO:teuthology.orchestra.run.smithi141.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-11-16T16:43:30.715 INFO:teuthology.orchestra.run.smithi141.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-11-16T16:43:30.715 INFO:teuthology.orchestra.run.smithi141.stderr: stderr: 10+0 records in 2023-11-16T16:43:30.715 INFO:teuthology.orchestra.run.smithi141.stderr:10+0 records out 2023-11-16T16:43:30.716 INFO:teuthology.orchestra.run.smithi141.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0146462 s, 716 MB/s 2023-11-16T16:43:30.716 INFO:teuthology.orchestra.run.smithi141.stderr:--> Zapping successful for: 2023-11-16T16:43:31.134 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi141:vg_nvme/lv_3 2023-11-16T16:43:31.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:31 smithi141 ceph-mon[131184]: from='osd.3 [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T16:43:31.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:31 smithi141 ceph-mon[131184]: osdmap e24: 4 total, 3 up, 4 in 2023-11-16T16:43:31.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:31 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:31.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:31 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:31.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:31 smithi002 ceph-mon[128712]: from='osd.3 [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T16:43:31.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:31 smithi002 ceph-mon[128712]: osdmap e24: 4 total, 3 up, 4 in 2023-11-16T16:43:31.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:31 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:31.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:31 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:31.778 INFO:teuthology.orchestra.run.smithi141.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T16:43:31.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:31 smithi155 ceph-mon[131607]: from='osd.3 [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T16:43:31.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:31 smithi155 ceph-mon[131607]: osdmap e24: 4 total, 3 up, 4 in 2023-11-16T16:43:31.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:31 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:31.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:31 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:32.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: purged_snaps scrub starts 2023-11-16T16:43:32.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: purged_snaps scrub ok 2023-11-16T16:43:32.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: osd.3 [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480] boot 2023-11-16T16:43:32.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: osdmap e25: 4 total, 4 up, 4 in 2023-11-16T16:43:32.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:32.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: Detected new or changed devices on smithi002 2023-11-16T16:43:32.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:32.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:32.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:32.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:32.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: Adjusting osd_memory_target on smithi002 to 8638M 2023-11-16T16:43:32.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:32.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:32.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:32.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:43:32.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:32.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:43:32.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:43:32.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:32.658 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:32 smithi002 ceph-mon[128712]: osdmap e26: 4 total, 4 up, 4 in 2023-11-16T16:43:32.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: purged_snaps scrub starts 2023-11-16T16:43:32.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: purged_snaps scrub ok 2023-11-16T16:43:32.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: osd.3 [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480] boot 2023-11-16T16:43:32.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: osdmap e25: 4 total, 4 up, 4 in 2023-11-16T16:43:32.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:32.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: Detected new or changed devices on smithi002 2023-11-16T16:43:32.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:32.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:32.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:32.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:32.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: Adjusting osd_memory_target on smithi002 to 8638M 2023-11-16T16:43:32.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:32.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:32.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:32.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:43:32.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:32.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:43:32.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:43:32.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:32.768 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:32 smithi141 ceph-mon[131184]: osdmap e26: 4 total, 4 up, 4 in 2023-11-16T16:43:32.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: purged_snaps scrub starts 2023-11-16T16:43:32.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: purged_snaps scrub ok 2023-11-16T16:43:32.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: osd.3 [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480] boot 2023-11-16T16:43:32.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: osdmap e25: 4 total, 4 up, 4 in 2023-11-16T16:43:32.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:43:32.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: Detected new or changed devices on smithi002 2023-11-16T16:43:32.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:32.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:32.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:32.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:32.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: Adjusting osd_memory_target on smithi002 to 8638M 2023-11-16T16:43:32.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:32.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:32.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:32.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:43:32.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:32.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:43:32.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:43:32.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:32.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:32 smithi155 ceph-mon[131607]: osdmap e26: 4 total, 4 up, 4 in 2023-11-16T16:43:32.959 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.954+0000 7f9f302c6700 1 -- 172.21.15.141:0/3760199784 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f281007c0 msgr2=0x7f9f28100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:32.959 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.954+0000 7f9f302c6700 1 --2- 172.21.15.141:0/3760199784 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f281007c0 0x7f9f28100bd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9f18009990 tx=0x7f9f18009ca0 comp rx=0 tx=0).stop 2023-11-16T16:43:32.959 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.954+0000 7f9f302c6700 1 -- 172.21.15.141:0/3760199784 shutdown_connections 2023-11-16T16:43:32.960 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.954+0000 7f9f302c6700 1 --2- 172.21.15.141:0/3760199784 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f280ffe90 0x7f9f28107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:32.960 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.954+0000 7f9f302c6700 1 --2- 172.21.15.141:0/3760199784 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f281007c0 0x7f9f28100bd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:32.960 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.954+0000 7f9f302c6700 1 --2- 172.21.15.141:0/3760199784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f280ff500 0x7f9f280ff950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:32.961 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.954+0000 7f9f302c6700 1 -- 172.21.15.141:0/3760199784 >> 172.21.15.141:0/3760199784 conn(0x7f9f280fa000 msgr2=0x7f9f280fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:43:32.961 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.954+0000 7f9f302c6700 1 -- 172.21.15.141:0/3760199784 shutdown_connections 2023-11-16T16:43:32.961 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.954+0000 7f9f302c6700 1 -- 172.21.15.141:0/3760199784 wait complete. 2023-11-16T16:43:32.961 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.955+0000 7f9f302c6700 1 Processor -- start 2023-11-16T16:43:32.962 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.955+0000 7f9f302c6700 1 -- start start 2023-11-16T16:43:32.962 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.955+0000 7f9f302c6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f280ff500 0x7f9f280fef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:32.962 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.955+0000 7f9f302c6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f280ffe90 0x7f9f2819ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:32.962 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.955+0000 7f9f302c6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f281007c0 0x7f9f2819d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:32.963 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.955+0000 7f9f302c6700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2819db70 con 0x7f9f280ff500 2023-11-16T16:43:32.963 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.955+0000 7f9f302c6700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2819dce0 con 0x7f9f280ffe90 2023-11-16T16:43:32.963 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.955+0000 7f9f302c6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2819dea0 con 0x7f9f281007c0 2023-11-16T16:43:32.963 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.955+0000 7f9f2d861700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f280ffe90 0x7f9f2819ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:32.963 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.955+0000 7f9f2d861700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f280ffe90 0x7f9f2819ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59754/0 (socket says 172.21.15.141:59754) 2023-11-16T16:43:32.964 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.955+0000 7f9f2d861700 1 -- 172.21.15.141:0/1708975442 learned_addr learned my addr 172.21.15.141:0/1708975442 (peer_addr_for_me v2:172.21.15.141:0/0) 2023-11-16T16:43:32.964 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.955+0000 7f9f2e863700 1 --2- 172.21.15.141:0/1708975442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f281007c0 0x7f9f2819d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:32.964 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.955+0000 7f9f2d861700 1 -- 172.21.15.141:0/1708975442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f281007c0 msgr2=0x7f9f2819d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:32.964 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.955+0000 7f9f2d861700 1 --2- 172.21.15.141:0/1708975442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f281007c0 0x7f9f2819d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:32.965 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.955+0000 7f9f2d861700 1 -- 172.21.15.141:0/1708975442 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f280ff500 msgr2=0x7f9f280fef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:32.965 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.955+0000 7f9f2d861700 1 --2- 172.21.15.141:0/1708975442 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f280ff500 0x7f9f280fef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:32.965 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.955+0000 7f9f2d861700 1 -- 172.21.15.141:0/1708975442 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f180095f0 con 0x7f9f280ffe90 2023-11-16T16:43:32.965 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.955+0000 7f9f2d861700 1 --2- 172.21.15.141:0/1708975442 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f280ffe90 0x7f9f2819ad80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9f2400b530 tx=0x7f9f2400b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:43:32.966 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.956+0000 7f9f1f7fe700 1 -- 172.21.15.141:0/1708975442 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f24011830 con 0x7f9f280ffe90 2023-11-16T16:43:32.966 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.956+0000 7f9f1f7fe700 1 -- 172.21.15.141:0/1708975442 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9f24011e50 con 0x7f9f280ffe90 2023-11-16T16:43:32.966 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.956+0000 7f9f302c6700 1 -- 172.21.15.141:0/1708975442 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f2810ac10 con 0x7f9f280ffe90 2023-11-16T16:43:32.966 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.956+0000 7f9f1f7fe700 1 -- 172.21.15.141:0/1708975442 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2400f2e0 con 0x7f9f280ffe90 2023-11-16T16:43:32.967 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.956+0000 7f9f302c6700 1 -- 172.21.15.141:0/1708975442 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f2810aa80 con 0x7f9f280ffe90 2023-11-16T16:43:32.967 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.957+0000 7f9f302c6700 1 -- 172.21.15.141:0/1708975442 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f28106870 con 0x7f9f280ffe90 2023-11-16T16:43:32.968 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.958+0000 7f9f1f7fe700 1 -- 172.21.15.141:0/1708975442 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f9f24010380 con 0x7f9f280ffe90 2023-11-16T16:43:32.968 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.958+0000 7f9f1f7fe700 1 --2- 172.21.15.141:0/1708975442 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9f140a11d0 0x7f9f140a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:32.969 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.959+0000 7f9f1f7fe700 1 -- 172.21.15.141:0/1708975442 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3698+0+0 (secure 0 0 0) 0x7f9f240cb1e0 con 0x7f9f280ffe90 2023-11-16T16:43:32.969 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.959+0000 7f9f2e062700 1 --2- 172.21.15.141:0/1708975442 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9f140a11d0 0x7f9f140a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:32.969 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.959+0000 7f9f2e062700 1 --2- 172.21.15.141:0/1708975442 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9f140a11d0 0x7f9f140a3680 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9f18005b40 tx=0x7f9f18005ad0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:43:32.969 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:32.963+0000 7f9f1f7fe700 1 -- 172.21.15.141:0/1708975442 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9f24095b30 con 0x7f9f280ffe90 2023-11-16T16:43:33.157 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:33.155+0000 7f9f302c6700 1 -- 172.21.15.141:0/1708975442 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi141:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f28061b90 con 0x7f9f140a11d0 2023-11-16T16:43:33.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:33 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:43:33.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:33 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:43:33.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:33 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:33.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:33 smithi155 ceph-mon[131607]: osdmap e27: 4 total, 4 up, 4 in 2023-11-16T16:43:33.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:33 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:43:33.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:33 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:43:33.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:33 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:33.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:33 smithi002 ceph-mon[128712]: osdmap e27: 4 total, 4 up, 4 in 2023-11-16T16:43:34.004 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:33 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:43:34.005 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:33 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:43:34.005 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:33 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:34.005 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:33 smithi141 ceph-mon[131184]: osdmap e27: 4 total, 4 up, 4 in 2023-11-16T16:43:35.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:34 smithi141 ceph-mon[131184]: from='client.24233 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi141:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:43:35.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:34 smithi141 ceph-mon[131184]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:35.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:34 smithi141 ceph-mon[131184]: osdmap e28: 4 total, 4 up, 4 in 2023-11-16T16:43:35.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:34 smithi155 ceph-mon[131607]: from='client.24233 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi141:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:43:35.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:34 smithi155 ceph-mon[131607]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:35.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:34 smithi155 ceph-mon[131607]: osdmap e28: 4 total, 4 up, 4 in 2023-11-16T16:43:35.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:34 smithi002 ceph-mon[128712]: from='client.24233 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi141:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:43:35.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:34 smithi002 ceph-mon[128712]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:35.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:34 smithi002 ceph-mon[128712]: osdmap e28: 4 total, 4 up, 4 in 2023-11-16T16:43:36.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:35 smithi141 ceph-mon[131184]: osdmap e29: 4 total, 4 up, 4 in 2023-11-16T16:43:36.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:35 smithi141 ceph-mon[131184]: from='client.? 172.21.15.141:0/2811753583' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a4b549e-4eb4-4f67-829c-c37004403b76"}]: dispatch 2023-11-16T16:43:36.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:35 smithi141 ceph-mon[131184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a4b549e-4eb4-4f67-829c-c37004403b76"}]: dispatch 2023-11-16T16:43:36.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:35 smithi141 ceph-mon[131184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a4b549e-4eb4-4f67-829c-c37004403b76"}]': finished 2023-11-16T16:43:36.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:35 smithi141 ceph-mon[131184]: osdmap e30: 5 total, 4 up, 5 in 2023-11-16T16:43:36.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:35 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:36.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:35 smithi155 ceph-mon[131607]: osdmap e29: 4 total, 4 up, 4 in 2023-11-16T16:43:36.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:35 smithi155 ceph-mon[131607]: from='client.? 172.21.15.141:0/2811753583' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a4b549e-4eb4-4f67-829c-c37004403b76"}]: dispatch 2023-11-16T16:43:36.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:35 smithi155 ceph-mon[131607]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a4b549e-4eb4-4f67-829c-c37004403b76"}]: dispatch 2023-11-16T16:43:36.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:35 smithi155 ceph-mon[131607]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a4b549e-4eb4-4f67-829c-c37004403b76"}]': finished 2023-11-16T16:43:36.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:35 smithi155 ceph-mon[131607]: osdmap e30: 5 total, 4 up, 5 in 2023-11-16T16:43:36.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:35 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:36.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:35 smithi002 ceph-mon[128712]: osdmap e29: 4 total, 4 up, 4 in 2023-11-16T16:43:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:35 smithi002 ceph-mon[128712]: from='client.? 172.21.15.141:0/2811753583' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a4b549e-4eb4-4f67-829c-c37004403b76"}]: dispatch 2023-11-16T16:43:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:35 smithi002 ceph-mon[128712]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a4b549e-4eb4-4f67-829c-c37004403b76"}]: dispatch 2023-11-16T16:43:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:35 smithi002 ceph-mon[128712]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a4b549e-4eb4-4f67-829c-c37004403b76"}]': finished 2023-11-16T16:43:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:35 smithi002 ceph-mon[128712]: osdmap e30: 5 total, 4 up, 5 in 2023-11-16T16:43:36.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:35 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:37.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:36 smithi141 ceph-mon[131184]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 171 KiB/s, 0 objects/s recovering 2023-11-16T16:43:37.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:36 smithi141 ceph-mon[131184]: from='client.? 172.21.15.141:0/56579411' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:43:37.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:36 smithi155 ceph-mon[131607]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 171 KiB/s, 0 objects/s recovering 2023-11-16T16:43:37.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:36 smithi155 ceph-mon[131607]: from='client.? 172.21.15.141:0/56579411' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:43:37.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:36 smithi002 ceph-mon[128712]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 171 KiB/s, 0 objects/s recovering 2023-11-16T16:43:37.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:36 smithi002 ceph-mon[128712]: from='client.? 172.21.15.141:0/56579411' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:43:39.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:38 smithi141 ceph-mon[131184]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 132 KiB/s, 0 objects/s recovering 2023-11-16T16:43:39.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:38 smithi155 ceph-mon[131607]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 132 KiB/s, 0 objects/s recovering 2023-11-16T16:43:39.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:38 smithi002 ceph-mon[128712]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 132 KiB/s, 0 objects/s recovering 2023-11-16T16:43:41.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:40 smithi141 ceph-mon[131184]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-16T16:43:41.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:40 smithi155 ceph-mon[131607]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-16T16:43:41.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:40 smithi002 ceph-mon[128712]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-16T16:43:43.206 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:42 smithi141 ceph-mon[131184]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:43.207 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:42 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-11-16T16:43:43.207 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:42 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:43.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:42 smithi155 ceph-mon[131607]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:43.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:42 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-11-16T16:43:43.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:42 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:43.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:42 smithi002 ceph-mon[128712]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:43.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:42 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-11-16T16:43:43.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:42 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:44.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:43 smithi141 ceph-mon[131184]: Deploying daemon osd.4 on smithi141 2023-11-16T16:43:44.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:43 smithi155 ceph-mon[131607]: Deploying daemon osd.4 on smithi141 2023-11-16T16:43:44.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:43 smithi002 ceph-mon[128712]: Deploying daemon osd.4 on smithi141 2023-11-16T16:43:45.149 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:44 smithi141 ceph-mon[131184]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:45.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:44 smithi155 ceph-mon[131607]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:45.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:44 smithi002 ceph-mon[128712]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:46 smithi141 ceph-mon[131184]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:46 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:46 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:46 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:47.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:46 smithi155 ceph-mon[131607]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:47.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:46 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:47.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:46 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:47.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:46 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:47.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:46 smithi002 ceph-mon[128712]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:47.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:46 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:47.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:46 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:47.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:46 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:49.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:48 smithi141 ceph-mon[131184]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:49.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:48 smithi155 ceph-mon[131607]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:49.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:48 smithi002 ceph-mon[128712]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:49.472 INFO:teuthology.orchestra.run.smithi141.stdout:Created osd(s) 4 on host 'smithi141' 2023-11-16T16:43:49.472 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:49.471+0000 7f9f1f7fe700 1 -- 172.21.15.141:0/1708975442 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9f28061b90 con 0x7f9f140a11d0 2023-11-16T16:43:49.474 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:49.474+0000 7f9f302c6700 1 -- 172.21.15.141:0/1708975442 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9f140a11d0 msgr2=0x7f9f140a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:49.474 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:49.474+0000 7f9f302c6700 1 --2- 172.21.15.141:0/1708975442 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9f140a11d0 0x7f9f140a3680 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9f18005b40 tx=0x7f9f18005ad0 comp rx=0 tx=0).stop 2023-11-16T16:43:49.475 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:49.474+0000 7f9f302c6700 1 -- 172.21.15.141:0/1708975442 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f280ffe90 msgr2=0x7f9f2819ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:49.475 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:49.474+0000 7f9f302c6700 1 --2- 172.21.15.141:0/1708975442 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f280ffe90 0x7f9f2819ad80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9f2400b530 tx=0x7f9f2400b840 comp rx=0 tx=0).stop 2023-11-16T16:43:49.475 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:49.474+0000 7f9f302c6700 1 -- 172.21.15.141:0/1708975442 shutdown_connections 2023-11-16T16:43:49.475 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:49.474+0000 7f9f302c6700 1 --2- 172.21.15.141:0/1708975442 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9f140a11d0 0x7f9f140a3680 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:49.475 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:49.474+0000 7f9f302c6700 1 --2- 172.21.15.141:0/1708975442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f281007c0 0x7f9f2819d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:49.475 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:49.474+0000 7f9f302c6700 1 --2- 172.21.15.141:0/1708975442 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f280ff500 0x7f9f280fef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:49.475 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:49.474+0000 7f9f302c6700 1 --2- 172.21.15.141:0/1708975442 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9f280ffe90 0x7f9f2819ad80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:49.475 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:49.474+0000 7f9f302c6700 1 -- 172.21.15.141:0/1708975442 >> 172.21.15.141:0/1708975442 conn(0x7f9f280fa000 msgr2=0x7f9f281039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:43:49.475 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:49.474+0000 7f9f302c6700 1 -- 172.21.15.141:0/1708975442 shutdown_connections 2023-11-16T16:43:49.475 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:43:49.474+0000 7f9f302c6700 1 -- 172.21.15.141:0/1708975442 wait complete. 2023-11-16T16:43:49.863 DEBUG:teuthology.orchestra.run.smithi141:osd.4> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.4.service 2023-11-16T16:43:49.865 INFO:tasks.cephadm:Deploying osd.5 on smithi155 with /dev/vg_nvme/lv_3... 2023-11-16T16:43:49.865 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-11-16T16:43:49.899 INFO:journalctl@ceph.osd.4.smithi141.stdout:-- Logs begin at Thu 2023-11-16 16:14:35 UTC. -- 2023-11-16T16:43:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:50 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:43:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:50 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:43:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:50 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:43:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:50.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:50.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:50.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:50.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:50.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:50 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:43:50.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:50.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:50 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:43:50.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:50 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:43:50.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:50.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:50.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:50.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:50.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:50.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:50.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:50.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:50 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:43:50.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:50.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:50 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:43:50.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:50 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:43:50.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:50.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:43:50.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:50.527 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:43:50.730 INFO:journalctl@ceph.osd.4.smithi141.stdout:Nov 16 16:43:50 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-4[141721]: 2023-11-16T16:43:50.611+0000 7f2589240640 -1 osd.4 0 log_to_monitors true 2023-11-16T16:43:51.243 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:51 smithi141 ceph-mon[131184]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:51.243 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:51 smithi141 ceph-mon[131184]: from='osd.4 [v2:172.21.15.141:6808/1025476846,v1:172.21.15.141:6809/1025476846]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T16:43:51.243 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:51 smithi141 ceph-mon[131184]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T16:43:51.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:51 smithi155 ceph-mon[131607]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:51.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:51 smithi155 ceph-mon[131607]: from='osd.4 [v2:172.21.15.141:6808/1025476846,v1:172.21.15.141:6809/1025476846]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T16:43:51.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:51 smithi155 ceph-mon[131607]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T16:43:51.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:51 smithi002 ceph-mon[128712]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:51.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:51 smithi002 ceph-mon[128712]: from='osd.4 [v2:172.21.15.141:6808/1025476846,v1:172.21.15.141:6809/1025476846]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T16:43:51.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:51 smithi002 ceph-mon[128712]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-16T16:43:52.038 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-11-16T16:43:52.038 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-11-16T16:43:52.038 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2023-11-16T16:43:52.038 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2023-11-16T16:43:52.038 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109136 s, 961 MB/s 2023-11-16T16:43:52.038 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2023-11-16T16:43:52.381 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi155:vg_nvme/lv_3 2023-11-16T16:43:52.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:52 smithi155 ceph-mon[131607]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-11-16T16:43:52.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:52 smithi155 ceph-mon[131607]: osdmap e31: 5 total, 4 up, 5 in 2023-11-16T16:43:52.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:52.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:52 smithi155 ceph-mon[131607]: from='osd.4 [v2:172.21.15.141:6808/1025476846,v1:172.21.15.141:6809/1025476846]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:43:52.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:52 smithi155 ceph-mon[131607]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:43:52.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:52 smithi155 ceph-mon[131607]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:52.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:52.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:52.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:52.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:52.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:52.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:52.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:52.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:52.765 INFO:journalctl@ceph.osd.4.smithi141.stdout:Nov 16 16:43:52 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-4[141721]: 2023-11-16T16:43:52.454+0000 7f257e8b7700 -1 osd.4 0 waiting for initial osdmap 2023-11-16T16:43:52.765 INFO:journalctl@ceph.osd.4.smithi141.stdout:Nov 16 16:43:52 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-4[141721]: 2023-11-16T16:43:52.462+0000 7f25776a6700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:43:52.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:52 smithi141 ceph-mon[131184]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-11-16T16:43:52.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:52 smithi141 ceph-mon[131184]: osdmap e31: 5 total, 4 up, 5 in 2023-11-16T16:43:52.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:52.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:52 smithi141 ceph-mon[131184]: from='osd.4 [v2:172.21.15.141:6808/1025476846,v1:172.21.15.141:6809/1025476846]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:43:52.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:52 smithi141 ceph-mon[131184]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:43:52.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:52 smithi141 ceph-mon[131184]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:52.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:52.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:52.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:52.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:52.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:52.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:52.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:52.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:52.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:52 smithi002 ceph-mon[128712]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-11-16T16:43:52.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:52 smithi002 ceph-mon[128712]: osdmap e31: 5 total, 4 up, 5 in 2023-11-16T16:43:52.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:52.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:52 smithi002 ceph-mon[128712]: from='osd.4 [v2:172.21.15.141:6808/1025476846,v1:172.21.15.141:6809/1025476846]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:43:52.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:52 smithi002 ceph-mon[128712]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:43:52.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:52 smithi002 ceph-mon[128712]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:52.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:52.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:52.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:52.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:43:52.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:52.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:52.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:43:52.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:43:53.026 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:43:53.510 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:53 smithi155 ceph-mon[131607]: Detected new or changed devices on smithi141 2023-11-16T16:43:53.511 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:53 smithi155 ceph-mon[131607]: Adjusting osd_memory_target on smithi141 to 8584M 2023-11-16T16:43:53.511 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:53 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:43:53.511 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:53 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:43:53.511 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:53 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:43:53.511 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:53 smithi155 ceph-mon[131607]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]': finished 2023-11-16T16:43:53.511 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:53 smithi155 ceph-mon[131607]: osdmap e32: 5 total, 4 up, 5 in 2023-11-16T16:43:53.511 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:53 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:53.511 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:53 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:53.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:53 smithi141 ceph-mon[131184]: Detected new or changed devices on smithi141 2023-11-16T16:43:53.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:53 smithi141 ceph-mon[131184]: Adjusting osd_memory_target on smithi141 to 8584M 2023-11-16T16:43:53.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:53 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:43:53.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:53 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:43:53.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:53 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:43:53.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:53 smithi141 ceph-mon[131184]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]': finished 2023-11-16T16:43:53.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:53 smithi141 ceph-mon[131184]: osdmap e32: 5 total, 4 up, 5 in 2023-11-16T16:43:53.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:53 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:53.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:53 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:53.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:53 smithi002 ceph-mon[128712]: Detected new or changed devices on smithi141 2023-11-16T16:43:53.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:53 smithi002 ceph-mon[128712]: Adjusting osd_memory_target on smithi141 to 8584M 2023-11-16T16:43:53.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:53 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:43:53.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:53 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:43:53.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:53 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:43:53.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:53 smithi002 ceph-mon[128712]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]': finished 2023-11-16T16:43:53.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:53 smithi002 ceph-mon[128712]: osdmap e32: 5 total, 4 up, 5 in 2023-11-16T16:43:53.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:53 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:53.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:53 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:54.263 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.258+0000 7f71f82cd700 1 -- 172.21.15.155:0/178595600 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71f0100860 msgr2=0x7f71f0100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:54.263 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.258+0000 7f71f82cd700 1 --2- 172.21.15.155:0/178595600 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71f0100860 0x7f71f0100c70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f71e0009990 tx=0x7f71e0009ca0 comp rx=0 tx=0).stop 2023-11-16T16:43:54.263 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.258+0000 7f71f82cd700 1 -- 172.21.15.155:0/178595600 shutdown_connections 2023-11-16T16:43:54.263 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.258+0000 7f71f82cd700 1 --2- 172.21.15.155:0/178595600 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71f0100860 0x7f71f0100c70 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:54.263 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.258+0000 7f71f82cd700 1 --2- 172.21.15.155:0/178595600 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71f00ffde0 0x7f71f0107690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:54.264 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.258+0000 7f71f82cd700 1 --2- 172.21.15.155:0/178595600 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71f00ff430 0x7f71f00ff8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:54.264 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.258+0000 7f71f82cd700 1 -- 172.21.15.155:0/178595600 >> 172.21.15.155:0/178595600 conn(0x7f71f00f9f70 msgr2=0x7f71f00fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:43:54.264 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.258+0000 7f71f82cd700 1 -- 172.21.15.155:0/178595600 shutdown_connections 2023-11-16T16:43:54.264 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.258+0000 7f71f82cd700 1 -- 172.21.15.155:0/178595600 wait complete. 2023-11-16T16:43:54.264 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.259+0000 7f71f82cd700 1 Processor -- start 2023-11-16T16:43:54.264 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.259+0000 7f71f82cd700 1 -- start start 2023-11-16T16:43:54.264 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.259+0000 7f71f82cd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71f00ff430 0x7f71f00feea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:54.264 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.259+0000 7f71f82cd700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71f00ffde0 0x7f71f019ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:54.265 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.259+0000 7f71f82cd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71f0100860 0x7f71f019d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:54.265 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.259+0000 7f71f82cd700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f019db40 con 0x7f71f0100860 2023-11-16T16:43:54.265 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.259+0000 7f71f82cd700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f019dcb0 con 0x7f71f00ffde0 2023-11-16T16:43:54.265 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.259+0000 7f71f82cd700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f019de70 con 0x7f71f00ff430 2023-11-16T16:43:54.265 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.259+0000 7f71f6069700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71f00ff430 0x7f71f00feea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:54.265 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.259+0000 7f71f6069700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71f00ff430 0x7f71f00feea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:39102/0 (socket says 172.21.15.155:39102) 2023-11-16T16:43:54.265 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.259+0000 7f71f6069700 1 -- 172.21.15.155:0/793290412 learned_addr learned my addr 172.21.15.155:0/793290412 (peer_addr_for_me v2:172.21.15.155:0/0) 2023-11-16T16:43:54.265 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.259+0000 7f71f5868700 1 --2- 172.21.15.155:0/793290412 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71f00ffde0 0x7f71f019ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:54.265 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.259+0000 7f71f6069700 1 -- 172.21.15.155:0/793290412 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71f00ffde0 msgr2=0x7f71f019ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:54.266 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.259+0000 7f71f6069700 1 --2- 172.21.15.155:0/793290412 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71f00ffde0 0x7f71f019ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:54.266 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.259+0000 7f71f6069700 1 -- 172.21.15.155:0/793290412 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71f0100860 msgr2=0x7f71f019d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:43:54.266 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.259+0000 7f71f6069700 1 --2- 172.21.15.155:0/793290412 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71f0100860 0x7f71f019d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:43:54.266 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.259+0000 7f71f6069700 1 -- 172.21.15.155:0/793290412 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e00095f0 con 0x7f71f00ff430 2023-11-16T16:43:54.266 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.260+0000 7f71f6069700 1 --2- 172.21.15.155:0/793290412 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71f00ff430 0x7f71f00feea0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f71e00059d0 tx=0x7f71e00047a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:43:54.267 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.260+0000 7f71e77fe700 1 -- 172.21.15.155:0/793290412 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e001d070 con 0x7f71f00ff430 2023-11-16T16:43:54.267 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.260+0000 7f71e77fe700 1 -- 172.21.15.155:0/793290412 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71e000fbc0 con 0x7f71f00ff430 2023-11-16T16:43:54.267 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.260+0000 7f71f82cd700 1 -- 172.21.15.155:0/793290412 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f010abc0 con 0x7f71f00ff430 2023-11-16T16:43:54.267 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.260+0000 7f71e77fe700 1 -- 172.21.15.155:0/793290412 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e0017650 con 0x7f71f00ff430 2023-11-16T16:43:54.267 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.260+0000 7f71f82cd700 1 -- 172.21.15.155:0/793290412 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f010aa30 con 0x7f71f00ff430 2023-11-16T16:43:54.268 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.261+0000 7f71f82cd700 1 -- 172.21.15.155:0/793290412 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f01067f0 con 0x7f71f00ff430 2023-11-16T16:43:54.268 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.262+0000 7f71e77fe700 1 -- 172.21.15.155:0/793290412 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f71e000fd30 con 0x7f71f00ff430 2023-11-16T16:43:54.269 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.263+0000 7f71e77fe700 1 --2- 172.21.15.155:0/793290412 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f71dc0a1150 0x7f71dc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:43:54.269 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.263+0000 7f71e77fe700 1 -- 172.21.15.155:0/793290412 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f71e00cd280 con 0x7f71f00ff430 2023-11-16T16:43:54.269 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.263+0000 7f71f5868700 1 --2- 172.21.15.155:0/793290412 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f71dc0a1150 0x7f71dc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:43:54.269 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.264+0000 7f71f5868700 1 --2- 172.21.15.155:0/793290412 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f71dc0a1150 0x7f71dc0a3600 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f71f00ff400 tx=0x7f71ec00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:43:54.269 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.267+0000 7f71e77fe700 1 -- 172.21.15.155:0/793290412 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f71e0097ad0 con 0x7f71f00ff430 2023-11-16T16:43:54.462 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:43:54.459+0000 7f71f82cd700 1 -- 172.21.15.155:0/793290412 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f71f0061c30 con 0x7f71dc0a1150 2023-11-16T16:43:54.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:54 smithi141 ceph-mon[131184]: purged_snaps scrub starts 2023-11-16T16:43:54.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:54 smithi141 ceph-mon[131184]: purged_snaps scrub ok 2023-11-16T16:43:54.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:54 smithi141 ceph-mon[131184]: osd.4 [v2:172.21.15.141:6808/1025476846,v1:172.21.15.141:6809/1025476846] boot 2023-11-16T16:43:54.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:54 smithi141 ceph-mon[131184]: osdmap e33: 5 total, 5 up, 5 in 2023-11-16T16:43:54.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:54.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:54 smithi141 ceph-mon[131184]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:54.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:54 smithi155 ceph-mon[131607]: purged_snaps scrub starts 2023-11-16T16:43:54.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:54 smithi155 ceph-mon[131607]: purged_snaps scrub ok 2023-11-16T16:43:54.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:54 smithi155 ceph-mon[131607]: osd.4 [v2:172.21.15.141:6808/1025476846,v1:172.21.15.141:6809/1025476846] boot 2023-11-16T16:43:54.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:54 smithi155 ceph-mon[131607]: osdmap e33: 5 total, 5 up, 5 in 2023-11-16T16:43:54.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:54.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:54 smithi155 ceph-mon[131607]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:54.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:54 smithi002 ceph-mon[128712]: purged_snaps scrub starts 2023-11-16T16:43:54.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:54 smithi002 ceph-mon[128712]: purged_snaps scrub ok 2023-11-16T16:43:54.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:54 smithi002 ceph-mon[128712]: osd.4 [v2:172.21.15.141:6808/1025476846,v1:172.21.15.141:6809/1025476846] boot 2023-11-16T16:43:54.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:54 smithi002 ceph-mon[128712]: osdmap e33: 5 total, 5 up, 5 in 2023-11-16T16:43:54.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:43:54.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:54 smithi002 ceph-mon[128712]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-11-16T16:43:55.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:55 smithi155 ceph-mon[131607]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:43:55.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:55 smithi155 ceph-mon[131607]: osdmap e34: 5 total, 5 up, 5 in 2023-11-16T16:43:55.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:55 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:43:55.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:55 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:43:55.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:55 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:55.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:55 smithi141 ceph-mon[131184]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:43:55.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:55 smithi141 ceph-mon[131184]: osdmap e34: 5 total, 5 up, 5 in 2023-11-16T16:43:55.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:55 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:43:55.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:55 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:43:55.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:55 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:55.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:55 smithi002 ceph-mon[128712]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:43:55.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:55 smithi002 ceph-mon[128712]: osdmap e34: 5 total, 5 up, 5 in 2023-11-16T16:43:55.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:55 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:43:55.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:55 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:43:55.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:55 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:43:56.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:56 smithi141 ceph-mon[131184]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:43:56.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:56 smithi155 ceph-mon[131607]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:43:56.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:56 smithi002 ceph-mon[128712]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:43:57.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:57 smithi141 ceph-mon[131184]: from='client.? 172.21.15.155:0/4200138266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd"}]: dispatch 2023-11-16T16:43:57.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:57 smithi141 ceph-mon[131184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd"}]: dispatch 2023-11-16T16:43:57.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:57 smithi141 ceph-mon[131184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd"}]': finished 2023-11-16T16:43:57.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:57 smithi141 ceph-mon[131184]: osdmap e35: 6 total, 5 up, 6 in 2023-11-16T16:43:57.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:57 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:43:57.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:57 smithi155 ceph-mon[131607]: from='client.? 172.21.15.155:0/4200138266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd"}]: dispatch 2023-11-16T16:43:57.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:57 smithi155 ceph-mon[131607]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd"}]: dispatch 2023-11-16T16:43:57.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:57 smithi155 ceph-mon[131607]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd"}]': finished 2023-11-16T16:43:57.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:57 smithi155 ceph-mon[131607]: osdmap e35: 6 total, 5 up, 6 in 2023-11-16T16:43:57.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:57 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:43:57.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:57 smithi002 ceph-mon[128712]: from='client.? 172.21.15.155:0/4200138266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd"}]: dispatch 2023-11-16T16:43:57.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:57 smithi002 ceph-mon[128712]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd"}]: dispatch 2023-11-16T16:43:57.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:57 smithi002 ceph-mon[128712]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd"}]': finished 2023-11-16T16:43:57.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:57 smithi002 ceph-mon[128712]: osdmap e35: 6 total, 5 up, 6 in 2023-11-16T16:43:57.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:57 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:43:58.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:58 smithi141 ceph-mon[131184]: from='client.? 172.21.15.155:0/3092977238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:43:58.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:43:58 smithi141 ceph-mon[131184]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:43:58.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:58 smithi155 ceph-mon[131607]: from='client.? 172.21.15.155:0/3092977238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:43:58.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:43:58 smithi155 ceph-mon[131607]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:43:58.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:58 smithi002 ceph-mon[128712]: from='client.? 172.21.15.155:0/3092977238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:43:58.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:43:58 smithi002 ceph-mon[128712]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:01.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:00 smithi141 ceph-mon[131184]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:01.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:00 smithi155 ceph-mon[131607]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:01.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:00 smithi002 ceph-mon[128712]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:03.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:02 smithi141 ceph-mon[131184]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:03.310 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:02 smithi155 ceph-mon[131607]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:03.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:02 smithi002 ceph-mon[128712]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:04.209 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:03 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-11-16T16:44:04.209 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:03 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:04.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:03 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-11-16T16:44:04.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:03 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:04.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:03 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-11-16T16:44:04.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:03 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:05.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:04 smithi155 ceph-mon[131607]: Deploying daemon osd.5 on smithi155 2023-11-16T16:44:05.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:04 smithi155 ceph-mon[131607]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:05.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:04 smithi141 ceph-mon[131184]: Deploying daemon osd.5 on smithi155 2023-11-16T16:44:05.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:04 smithi141 ceph-mon[131184]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:05.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:04 smithi002 ceph-mon[128712]: Deploying daemon osd.5 on smithi155 2023-11-16T16:44:05.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:04 smithi002 ceph-mon[128712]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:07.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:06 smithi141 ceph-mon[131184]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:07.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:06 smithi155 ceph-mon[131607]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:07.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:06 smithi002 ceph-mon[128712]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:09.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:08 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:09.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:08 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:09.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:08 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:09.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:08 smithi141 ceph-mon[131184]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:09.067 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:08 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:09.067 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:08 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:09.067 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:08 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:09.067 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:08 smithi155 ceph-mon[131607]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:09.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:08 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:09.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:08 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:09.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:08 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:09.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:08 smithi002 ceph-mon[128712]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:10.680 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 5 on host 'smithi155' 2023-11-16T16:44:10.680 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:44:10.678+0000 7f71e77fe700 1 -- 172.21.15.155:0/793290412 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f71f0061c30 con 0x7f71dc0a1150 2023-11-16T16:44:10.682 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:44:10.681+0000 7f71f82cd700 1 -- 172.21.15.155:0/793290412 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f71dc0a1150 msgr2=0x7f71dc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:10.682 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:44:10.681+0000 7f71f82cd700 1 --2- 172.21.15.155:0/793290412 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f71dc0a1150 0x7f71dc0a3600 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f71f00ff400 tx=0x7f71ec00b410 comp rx=0 tx=0).stop 2023-11-16T16:44:10.683 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:44:10.681+0000 7f71f82cd700 1 -- 172.21.15.155:0/793290412 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71f00ff430 msgr2=0x7f71f00feea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:10.683 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:44:10.681+0000 7f71f82cd700 1 --2- 172.21.15.155:0/793290412 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71f00ff430 0x7f71f00feea0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f71e00059d0 tx=0x7f71e00047a0 comp rx=0 tx=0).stop 2023-11-16T16:44:10.683 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:44:10.681+0000 7f71f82cd700 1 -- 172.21.15.155:0/793290412 shutdown_connections 2023-11-16T16:44:10.683 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:44:10.681+0000 7f71f82cd700 1 --2- 172.21.15.155:0/793290412 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f71dc0a1150 0x7f71dc0a3600 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:10.683 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:44:10.681+0000 7f71f82cd700 1 --2- 172.21.15.155:0/793290412 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71f00ff430 0x7f71f00feea0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:10.683 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:44:10.681+0000 7f71f82cd700 1 --2- 172.21.15.155:0/793290412 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71f0100860 0x7f71f019d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:10.683 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:44:10.681+0000 7f71f82cd700 1 --2- 172.21.15.155:0/793290412 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f71f00ffde0 0x7f71f019ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:10.684 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:44:10.681+0000 7f71f82cd700 1 -- 172.21.15.155:0/793290412 >> 172.21.15.155:0/793290412 conn(0x7f71f00f9f70 msgr2=0x7f71f0103930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:44:10.684 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:44:10.681+0000 7f71f82cd700 1 -- 172.21.15.155:0/793290412 shutdown_connections 2023-11-16T16:44:10.684 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:44:10.681+0000 7f71f82cd700 1 -- 172.21.15.155:0/793290412 wait complete. 2023-11-16T16:44:11.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:10 smithi155 ceph-mon[131607]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:11.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:10 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:11.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:10 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:11.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:10 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:11.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:10 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:11.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:10 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:11.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:10 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:11.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:10 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:11.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:10 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:11.120 DEBUG:teuthology.orchestra.run.smithi155:osd.5> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.5.service 2023-11-16T16:44:11.122 INFO:tasks.cephadm:Deploying osd.6 on smithi002 with /dev/vg_nvme/lv_2... 2023-11-16T16:44:11.122 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-11-16T16:44:11.153 INFO:journalctl@ceph.osd.5.smithi155.stdout:-- Logs begin at Thu 2023-11-16 16:15:48 UTC. -- 2023-11-16T16:44:11.192 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:10 smithi002 ceph-mon[128712]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:11.193 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:11.193 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:11.193 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:11.193 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:11.193 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:11.193 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:11.194 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:11.194 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:11.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:10 smithi141 ceph-mon[131184]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:10 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:10 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:10 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:10 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:10 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:10 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:10 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:11.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:10 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:11.833 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 16:44:11 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T16:44:11.613+0000 7fbe433eb640 -1 osd.5 0 log_to_monitors true 2023-11-16T16:44:11.990 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:44:12.110 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:11 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:44:12.110 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:11 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:44:12.110 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:11 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:44:12.110 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:11 smithi155 ceph-mon[131607]: from='osd.5 [v2:172.21.15.155:6808/2988032938,v1:172.21.15.155:6809/2988032938]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T16:44:12.110 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:11 smithi155 ceph-mon[131607]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T16:44:12.250 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:11 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:44:12.250 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:11 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:44:12.251 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:11 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:44:12.251 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:11 smithi002 ceph-mon[128712]: from='osd.5 [v2:172.21.15.155:6808/2988032938,v1:172.21.15.155:6809/2988032938]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T16:44:12.251 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:11 smithi002 ceph-mon[128712]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T16:44:12.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:11 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:44:12.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:11 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:44:12.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:11 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:44:12.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:11 smithi141 ceph-mon[131184]: from='osd.5 [v2:172.21.15.155:6808/2988032938,v1:172.21.15.155:6809/2988032938]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T16:44:12.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:11 smithi141 ceph-mon[131184]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-16T16:44:13.105 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:12 smithi155 ceph-mon[131607]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:13.105 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:12 smithi155 ceph-mon[131607]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-11-16T16:44:13.106 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:12 smithi155 ceph-mon[131607]: osdmap e36: 6 total, 5 up, 6 in 2023-11-16T16:44:13.106 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:12 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:13.106 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:12 smithi155 ceph-mon[131607]: from='osd.5 [v2:172.21.15.155:6808/2988032938,v1:172.21.15.155:6809/2988032938]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:44:13.106 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:12 smithi155 ceph-mon[131607]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:44:13.106 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 16:44:12 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T16:44:12.958+0000 7fbe38a62700 -1 osd.5 0 waiting for initial osdmap 2023-11-16T16:44:13.106 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 16:44:12 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T16:44:12.967+0000 7fbe33855700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:44:13.238 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:12 smithi002 ceph-mon[128712]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:13.238 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:12 smithi002 ceph-mon[128712]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-11-16T16:44:13.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:12 smithi002 ceph-mon[128712]: osdmap e36: 6 total, 5 up, 6 in 2023-11-16T16:44:13.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:12 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:13.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:12 smithi002 ceph-mon[128712]: from='osd.5 [v2:172.21.15.155:6808/2988032938,v1:172.21.15.155:6809/2988032938]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:44:13.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:12 smithi002 ceph-mon[128712]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:44:13.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:12 smithi141 ceph-mon[131184]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:13.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:12 smithi141 ceph-mon[131184]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-11-16T16:44:13.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:12 smithi141 ceph-mon[131184]: osdmap e36: 6 total, 5 up, 6 in 2023-11-16T16:44:13.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:12 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:13.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:12 smithi141 ceph-mon[131184]: from='osd.5 [v2:172.21.15.155:6808/2988032938,v1:172.21.15.155:6809/2988032938]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:44:13.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:12 smithi141 ceph-mon[131184]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:44:13.974 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-11-16T16:44:13.974 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-11-16T16:44:13.974 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2023-11-16T16:44:13.975 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2023-11-16T16:44:13.975 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0350134 s, 299 MB/s 2023-11-16T16:44:13.975 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2023-11-16T16:44:14.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:13 smithi141 ceph-mon[131184]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2023-11-16T16:44:14.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:13 smithi141 ceph-mon[131184]: osdmap e37: 6 total, 5 up, 6 in 2023-11-16T16:44:14.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:14.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:14.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:14.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:14.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:14.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:14.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:14.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:14.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:14.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:14.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:13 smithi141 ceph-mon[131184]: osd.5 [v2:172.21.15.155:6808/2988032938,v1:172.21.15.155:6809/2988032938] boot 2023-11-16T16:44:14.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:13 smithi141 ceph-mon[131184]: osdmap e38: 6 total, 6 up, 6 in 2023-11-16T16:44:14.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:14.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:13 smithi155 ceph-mon[131607]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2023-11-16T16:44:14.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:13 smithi155 ceph-mon[131607]: osdmap e37: 6 total, 5 up, 6 in 2023-11-16T16:44:14.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:14.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:14.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:14.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:14.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:14.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:14.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:14.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:14.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:14.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:14.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:13 smithi155 ceph-mon[131607]: osd.5 [v2:172.21.15.155:6808/2988032938,v1:172.21.15.155:6809/2988032938] boot 2023-11-16T16:44:14.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:13 smithi155 ceph-mon[131607]: osdmap e38: 6 total, 6 up, 6 in 2023-11-16T16:44:14.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:13 smithi002 ceph-mon[128712]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2023-11-16T16:44:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:13 smithi002 ceph-mon[128712]: osdmap e37: 6 total, 5 up, 6 in 2023-11-16T16:44:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:14.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:14.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:14.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:14.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:14.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:13 smithi002 ceph-mon[128712]: osd.5 [v2:172.21.15.155:6808/2988032938,v1:172.21.15.155:6809/2988032938] boot 2023-11-16T16:44:14.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:13 smithi002 ceph-mon[128712]: osdmap e38: 6 total, 6 up, 6 in 2023-11-16T16:44:14.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:44:14.479 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi002:vg_nvme/lv_2 2023-11-16T16:44:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:14 smithi141 ceph-mon[131184]: purged_snaps scrub starts 2023-11-16T16:44:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:14 smithi141 ceph-mon[131184]: purged_snaps scrub ok 2023-11-16T16:44:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:14 smithi141 ceph-mon[131184]: Detected new or changed devices on smithi155 2023-11-16T16:44:15.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:14 smithi141 ceph-mon[131184]: Adjusting osd_memory_target on smithi155 to 8584M 2023-11-16T16:44:15.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:14 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:44:15.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:14 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:44:15.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:14 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:44:15.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:14 smithi141 ceph-mon[131184]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:15.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:14 smithi141 ceph-mon[131184]: osdmap e39: 6 total, 6 up, 6 in 2023-11-16T16:44:15.340 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:44:15.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:14 smithi155 ceph-mon[131607]: purged_snaps scrub starts 2023-11-16T16:44:15.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:14 smithi155 ceph-mon[131607]: purged_snaps scrub ok 2023-11-16T16:44:15.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:14 smithi155 ceph-mon[131607]: Detected new or changed devices on smithi155 2023-11-16T16:44:15.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:14 smithi155 ceph-mon[131607]: Adjusting osd_memory_target on smithi155 to 8584M 2023-11-16T16:44:15.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:14 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:44:15.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:14 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:44:15.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:14 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:44:15.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:14 smithi155 ceph-mon[131607]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:15.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:14 smithi155 ceph-mon[131607]: osdmap e39: 6 total, 6 up, 6 in 2023-11-16T16:44:15.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:14 smithi002 ceph-mon[128712]: purged_snaps scrub starts 2023-11-16T16:44:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:14 smithi002 ceph-mon[128712]: purged_snaps scrub ok 2023-11-16T16:44:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:14 smithi002 ceph-mon[128712]: Detected new or changed devices on smithi155 2023-11-16T16:44:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:14 smithi002 ceph-mon[128712]: Adjusting osd_memory_target on smithi155 to 8584M 2023-11-16T16:44:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:14 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:44:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:14 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:44:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:14 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:44:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:14 smithi002 ceph-mon[128712]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-11-16T16:44:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:14 smithi002 ceph-mon[128712]: osdmap e39: 6 total, 6 up, 6 in 2023-11-16T16:44:16.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.694+0000 7f7a6ddaa700 1 -- 172.21.15.2:0/1525661254 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a681014f0 msgr2=0x7f7a68101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:16.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.694+0000 7f7a6ddaa700 1 --2- 172.21.15.2:0/1525661254 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a681014f0 0x7f7a68101940 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009990 tx=0x7f7a5c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:44:16.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.694+0000 7f7a6ddaa700 1 -- 172.21.15.2:0/1525661254 shutdown_connections 2023-11-16T16:44:16.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.694+0000 7f7a6ddaa700 1 --2- 172.21.15.2:0/1525661254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7a681002f0 0x7f7a68100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:16.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.694+0000 7f7a6ddaa700 1 --2- 172.21.15.2:0/1525661254 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a68101e80 0x7f7a68108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:16.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.694+0000 7f7a6ddaa700 1 --2- 172.21.15.2:0/1525661254 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a681014f0 0x7f7a68101940 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:16.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.694+0000 7f7a6ddaa700 1 -- 172.21.15.2:0/1525661254 >> 172.21.15.2:0/1525661254 conn(0x7f7a680fb900 msgr2=0x7f7a680fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:44:16.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.694+0000 7f7a6ddaa700 1 -- 172.21.15.2:0/1525661254 shutdown_connections 2023-11-16T16:44:16.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.694+0000 7f7a6ddaa700 1 -- 172.21.15.2:0/1525661254 wait complete. 2023-11-16T16:44:16.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.695+0000 7f7a6ddaa700 1 Processor -- start 2023-11-16T16:44:16.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.695+0000 7f7a6ddaa700 1 -- start start 2023-11-16T16:44:16.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.696+0000 7f7a6ddaa700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a681002f0 0x7f7a681981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:44:16.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.696+0000 7f7a6ddaa700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a681014f0 0x7f7a68198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:44:16.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.696+0000 7f7a6ddaa700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7a68101e80 0x7f7a6819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:44:16.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.696+0000 7f7a6ddaa700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6819fd80 con 0x7f7a681014f0 2023-11-16T16:44:16.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.696+0000 7f7a6ddaa700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6819fec0 con 0x7f7a681002f0 2023-11-16T16:44:16.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.696+0000 7f7a6ddaa700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a681a0050 con 0x7f7a68101e80 2023-11-16T16:44:16.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.696+0000 7f7a66ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a681014f0 0x7f7a68198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:44:16.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.696+0000 7f7a66ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a681014f0 0x7f7a68198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42054/0 (socket says 172.21.15.2:42054) 2023-11-16T16:44:16.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.696+0000 7f7a66ffd700 1 -- 172.21.15.2:0/695274429 learned_addr learned my addr 172.21.15.2:0/695274429 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:44:16.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.696+0000 7f7a677fe700 1 --2- 172.21.15.2:0/695274429 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a681002f0 0x7f7a681981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:44:16.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.696+0000 7f7a67fff700 1 --2- 172.21.15.2:0/695274429 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7a68101e80 0x7f7a6819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:44:16.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.696+0000 7f7a66ffd700 1 -- 172.21.15.2:0/695274429 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7a68101e80 msgr2=0x7f7a6819f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:16.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.696+0000 7f7a66ffd700 1 --2- 172.21.15.2:0/695274429 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7a68101e80 0x7f7a6819f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:16.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.697+0000 7f7a66ffd700 1 -- 172.21.15.2:0/695274429 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a681002f0 msgr2=0x7f7a681981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:16.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.697+0000 7f7a66ffd700 1 --2- 172.21.15.2:0/695274429 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a681002f0 0x7f7a681981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:16.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.697+0000 7f7a66ffd700 1 -- 172.21.15.2:0/695274429 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5c0095f0 con 0x7f7a681014f0 2023-11-16T16:44:16.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.697+0000 7f7a677fe700 1 --2- 172.21.15.2:0/695274429 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a681002f0 0x7f7a681981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:44:16.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.697+0000 7f7a66ffd700 1 --2- 172.21.15.2:0/695274429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a681014f0 0x7f7a68198730 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c006010 tx=0x7f7a5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:44:16.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.697+0000 7f7a64ff9700 1 -- 172.21.15.2:0/695274429 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c01d070 con 0x7f7a681014f0 2023-11-16T16:44:16.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.697+0000 7f7a6ddaa700 1 -- 172.21.15.2:0/695274429 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6810bc50 con 0x7f7a681014f0 2023-11-16T16:44:16.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.697+0000 7f7a64ff9700 1 -- 172.21.15.2:0/695274429 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a5c022470 con 0x7f7a681014f0 2023-11-16T16:44:16.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.697+0000 7f7a64ff9700 1 -- 172.21.15.2:0/695274429 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c017650 con 0x7f7a681014f0 2023-11-16T16:44:16.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.697+0000 7f7a6ddaa700 1 -- 172.21.15.2:0/695274429 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6810bac0 con 0x7f7a681014f0 2023-11-16T16:44:16.702 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.698+0000 7f7a6ddaa700 1 -- 172.21.15.2:0/695274429 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a68066ea0 con 0x7f7a681014f0 2023-11-16T16:44:16.702 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.701+0000 7f7a64ff9700 1 -- 172.21.15.2:0/695274429 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f7a5c022850 con 0x7f7a681014f0 2023-11-16T16:44:16.705 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.703+0000 7f7a64ff9700 1 --2- 172.21.15.2:0/695274429 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f7a500a1150 0x7f7a500a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:44:16.705 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.703+0000 7f7a64ff9700 1 -- 172.21.15.2:0/695274429 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4562+0+0 (secure 0 0 0) 0x7f7a5c0cd160 con 0x7f7a681014f0 2023-11-16T16:44:16.705 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.703+0000 7f7a677fe700 1 --2- 172.21.15.2:0/695274429 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f7a500a1150 0x7f7a500a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:44:16.706 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.704+0000 7f7a677fe700 1 --2- 172.21.15.2:0/695274429 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f7a500a1150 0x7f7a500a3600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7a68101350 tx=0x7f7a58009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:44:16.707 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.706+0000 7f7a64ff9700 1 -- 172.21.15.2:0/695274429 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7a5c097740 con 0x7f7a681014f0 2023-11-16T16:44:16.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:16.922+0000 7f7a6ddaa700 1 -- 172.21.15.2:0/695274429 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a681a0b20 con 0x7f7a500a1150 2023-11-16T16:44:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:16 smithi141 ceph-mon[131184]: pgmap v135: 1 pgs: 1 peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:16 smithi141 ceph-mon[131184]: osdmap e40: 6 total, 6 up, 6 in 2023-11-16T16:44:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:16 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:44:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:16 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:44:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:16 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:17.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:16 smithi155 ceph-mon[131607]: pgmap v135: 1 pgs: 1 peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:17.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:16 smithi155 ceph-mon[131607]: osdmap e40: 6 total, 6 up, 6 in 2023-11-16T16:44:17.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:16 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:44:17.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:16 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:44:17.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:16 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:17.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:16 smithi002 ceph-mon[128712]: pgmap v135: 1 pgs: 1 peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:17.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:16 smithi002 ceph-mon[128712]: osdmap e40: 6 total, 6 up, 6 in 2023-11-16T16:44:17.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:16 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:44:17.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:16 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:44:17.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:16 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:18.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:17 smithi002 ceph-mon[128712]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:44:18.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:17 smithi002 ceph-mon[128712]: osdmap e41: 6 total, 6 up, 6 in 2023-11-16T16:44:18.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:17 smithi141 ceph-mon[131184]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:44:18.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:17 smithi141 ceph-mon[131184]: osdmap e41: 6 total, 6 up, 6 in 2023-11-16T16:44:18.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:17 smithi155 ceph-mon[131607]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:44:18.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:17 smithi155 ceph-mon[131607]: osdmap e41: 6 total, 6 up, 6 in 2023-11-16T16:44:19.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:18 smithi002 ceph-mon[128712]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:19.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:18 smithi002 ceph-mon[128712]: osdmap e42: 6 total, 6 up, 6 in 2023-11-16T16:44:19.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:18 smithi141 ceph-mon[131184]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:18 smithi141 ceph-mon[131184]: osdmap e42: 6 total, 6 up, 6 in 2023-11-16T16:44:19.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:18 smithi155 ceph-mon[131607]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:19.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:18 smithi155 ceph-mon[131607]: osdmap e42: 6 total, 6 up, 6 in 2023-11-16T16:44:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:20 smithi141 ceph-mon[131184]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2023-11-16T16:44:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:20 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2141052153' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e574f450-ac2d-445f-bcef-61ccdc6f6985"}]: dispatch 2023-11-16T16:44:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:20 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2141052153' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e574f450-ac2d-445f-bcef-61ccdc6f6985"}]': finished 2023-11-16T16:44:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:20 smithi141 ceph-mon[131184]: osdmap e43: 7 total, 6 up, 7 in 2023-11-16T16:44:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:20 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:20 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/449656571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:44:21.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:20 smithi155 ceph-mon[131607]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2023-11-16T16:44:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:20 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2141052153' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e574f450-ac2d-445f-bcef-61ccdc6f6985"}]: dispatch 2023-11-16T16:44:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:20 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2141052153' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e574f450-ac2d-445f-bcef-61ccdc6f6985"}]': finished 2023-11-16T16:44:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:20 smithi155 ceph-mon[131607]: osdmap e43: 7 total, 6 up, 7 in 2023-11-16T16:44:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:20 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:20 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/449656571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:44:21.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:20 smithi002 ceph-mon[128712]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2023-11-16T16:44:21.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:20 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2141052153' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e574f450-ac2d-445f-bcef-61ccdc6f6985"}]: dispatch 2023-11-16T16:44:21.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:20 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2141052153' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e574f450-ac2d-445f-bcef-61ccdc6f6985"}]': finished 2023-11-16T16:44:21.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:20 smithi002 ceph-mon[128712]: osdmap e43: 7 total, 6 up, 7 in 2023-11-16T16:44:21.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:20 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:21.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:20 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/449656571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:44:23.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:22 smithi141 ceph-mon[131184]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 99 KiB/s, 0 objects/s recovering 2023-11-16T16:44:23.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:22 smithi155 ceph-mon[131607]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 99 KiB/s, 0 objects/s recovering 2023-11-16T16:44:23.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:22 smithi002 ceph-mon[128712]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 99 KiB/s, 0 objects/s recovering 2023-11-16T16:44:25.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:24 smithi141 ceph-mon[131184]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2023-11-16T16:44:25.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:24 smithi155 ceph-mon[131607]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2023-11-16T16:44:25.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:24 smithi002 ceph-mon[128712]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2023-11-16T16:44:27.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:26 smithi141 ceph-mon[131184]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T16:44:27.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:26 smithi155 ceph-mon[131607]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T16:44:27.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:26 smithi002 ceph-mon[128712]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T16:44:28.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:27 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-11-16T16:44:28.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:27 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:28.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:27 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-11-16T16:44:28.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:27 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:28.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:27 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-11-16T16:44:28.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:27 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:29.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:28 smithi141 ceph-mon[131184]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2023-11-16T16:44:29.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:28 smithi141 ceph-mon[131184]: Deploying daemon osd.6 on smithi002 2023-11-16T16:44:29.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:28 smithi155 ceph-mon[131607]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2023-11-16T16:44:29.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:28 smithi155 ceph-mon[131607]: Deploying daemon osd.6 on smithi002 2023-11-16T16:44:29.402 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:28 smithi002 ceph-mon[128712]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2023-11-16T16:44:29.402 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:28 smithi002 ceph-mon[128712]: Deploying daemon osd.6 on smithi002 2023-11-16T16:44:31.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:30 smithi141 ceph-mon[131184]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:31.306 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:30 smithi002 ceph-mon[128712]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:31.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:30 smithi155 ceph-mon[131607]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:33.262 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:32 smithi002 ceph-mon[128712]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:33.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:32 smithi141 ceph-mon[131184]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:33.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:32 smithi155 ceph-mon[131607]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:35.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:34 smithi141 ceph-mon[131184]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:35.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:35.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:35.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:34 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:35.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:34 smithi155 ceph-mon[131607]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:35.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:34 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:35.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:34 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:35.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:34 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:35.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:34 smithi002 ceph-mon[128712]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:35.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:34 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:35.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:34 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:35.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:34 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:37.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:36 smithi002 ceph-mon[128712]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:37.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:36 smithi141 ceph-mon[131184]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:37.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:36 smithi155 ceph-mon[131607]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:38.187 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:38.185+0000 7f7a64ff9700 1 -- 172.21.15.2:0/695274429 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7a681a0b20 con 0x7f7a500a1150 2023-11-16T16:44:38.187 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 6 on host 'smithi002' 2023-11-16T16:44:38.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:38.187+0000 7f7a6ddaa700 1 -- 172.21.15.2:0/695274429 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f7a500a1150 msgr2=0x7f7a500a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:38.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:38.187+0000 7f7a6ddaa700 1 --2- 172.21.15.2:0/695274429 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f7a500a1150 0x7f7a500a3600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7a68101350 tx=0x7f7a58009260 comp rx=0 tx=0).stop 2023-11-16T16:44:38.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:38.187+0000 7f7a6ddaa700 1 -- 172.21.15.2:0/695274429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a681014f0 msgr2=0x7f7a68198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:38.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:38.187+0000 7f7a6ddaa700 1 --2- 172.21.15.2:0/695274429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a681014f0 0x7f7a68198730 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c006010 tx=0x7f7a5c004730 comp rx=0 tx=0).stop 2023-11-16T16:44:38.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:38.187+0000 7f7a6ddaa700 1 -- 172.21.15.2:0/695274429 shutdown_connections 2023-11-16T16:44:38.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:38.187+0000 7f7a6ddaa700 1 --2- 172.21.15.2:0/695274429 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f7a500a1150 0x7f7a500a3600 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:38.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:38.187+0000 7f7a6ddaa700 1 --2- 172.21.15.2:0/695274429 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7a68101e80 0x7f7a6819f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:38.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:38.187+0000 7f7a6ddaa700 1 --2- 172.21.15.2:0/695274429 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7a681002f0 0x7f7a681981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:38.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:38.187+0000 7f7a6ddaa700 1 --2- 172.21.15.2:0/695274429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a681014f0 0x7f7a68198730 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:38.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:38.187+0000 7f7a6ddaa700 1 -- 172.21.15.2:0/695274429 >> 172.21.15.2:0/695274429 conn(0x7f7a680fb900 msgr2=0x7f7a68106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:44:38.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:38.187+0000 7f7a6ddaa700 1 -- 172.21.15.2:0/695274429 shutdown_connections 2023-11-16T16:44:38.191 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:44:38.187+0000 7f7a6ddaa700 1 -- 172.21.15.2:0/695274429 wait complete. 2023-11-16T16:44:38.684 DEBUG:teuthology.orchestra.run.smithi002:osd.6> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.6.service 2023-11-16T16:44:38.689 INFO:tasks.cephadm:Deploying osd.7 on smithi141 with /dev/vg_nvme/lv_2... 2023-11-16T16:44:38.689 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-11-16T16:44:38.767 INFO:journalctl@ceph.osd.6.smithi002.stdout:-- Logs begin at Thu 2023-11-16 16:19:53 UTC. -- 2023-11-16T16:44:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:38 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:38 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:38 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:38 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:38 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:44:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:38 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:38 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:44:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:38 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:44:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:38 smithi141 ceph-mon[131184]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:39.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:38 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:39.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:38 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:39.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:38 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:39.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:38 smithi141 ceph-mon[131184]: from='osd.6 [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-16T16:44:39.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:38 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:39.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:38 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:39.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:38 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:39.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:38 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:38 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:38 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:38 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:38 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:38 smithi155 ceph-mon[131607]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:38 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:38 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:38 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:39.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:38 smithi155 ceph-mon[131607]: from='osd.6 [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-16T16:44:39.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:38 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:39.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:38 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:39.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:38 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:39.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:38 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:39.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:38 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:44:39.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:38 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:39.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:38 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:44:39.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:38 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:44:39.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:38 smithi002 ceph-mon[128712]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:39.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:38 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:39.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:38 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:39.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:38 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:39.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:38 smithi002 ceph-mon[128712]: from='osd.6 [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-16T16:44:39.340 INFO:teuthology.orchestra.run.smithi141.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T16:44:40.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:40 smithi002 ceph-mon[128712]: from='osd.6 [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-11-16T16:44:40.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:40 smithi002 ceph-mon[128712]: osdmap e44: 7 total, 6 up, 7 in 2023-11-16T16:44:40.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:40 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:40.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:40 smithi002 ceph-mon[128712]: from='osd.6 [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T16:44:40.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:40 smithi002 ceph-mon[128712]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:40.407 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 16:44:40 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T16:44:40.187+0000 7f69353a4700 -1 osd.6 0 waiting for initial osdmap 2023-11-16T16:44:40.407 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 16:44:40 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T16:44:40.200+0000 7f692e193700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:44:40.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:40 smithi141 ceph-mon[131184]: from='osd.6 [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-11-16T16:44:40.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:40 smithi141 ceph-mon[131184]: osdmap e44: 7 total, 6 up, 7 in 2023-11-16T16:44:40.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:40 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:40.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:40 smithi141 ceph-mon[131184]: from='osd.6 [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T16:44:40.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:40 smithi141 ceph-mon[131184]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:40.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:40 smithi155 ceph-mon[131607]: from='osd.6 [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-11-16T16:44:40.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:40 smithi155 ceph-mon[131607]: osdmap e44: 7 total, 6 up, 7 in 2023-11-16T16:44:40.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:40 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:40.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:40 smithi155 ceph-mon[131607]: from='osd.6 [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T16:44:40.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:40 smithi155 ceph-mon[131607]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-11-16T16:44:40.851 INFO:teuthology.orchestra.run.smithi141.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-11-16T16:44:40.851 INFO:teuthology.orchestra.run.smithi141.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-11-16T16:44:40.851 INFO:teuthology.orchestra.run.smithi141.stderr: stderr: 10+0 records in 2023-11-16T16:44:40.851 INFO:teuthology.orchestra.run.smithi141.stderr:10+0 records out 2023-11-16T16:44:40.851 INFO:teuthology.orchestra.run.smithi141.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111006 s, 945 MB/s 2023-11-16T16:44:40.851 INFO:teuthology.orchestra.run.smithi141.stderr:--> Zapping successful for: 2023-11-16T16:44:41.295 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi141:vg_nvme/lv_2 2023-11-16T16:44:41.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:41 smithi141 ceph-mon[131184]: from='osd.6 [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T16:44:41.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:41 smithi141 ceph-mon[131184]: osdmap e45: 7 total, 6 up, 7 in 2023-11-16T16:44:41.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:41.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:41 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:41.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:41 smithi155 ceph-mon[131607]: from='osd.6 [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T16:44:41.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:41 smithi155 ceph-mon[131607]: osdmap e45: 7 total, 6 up, 7 in 2023-11-16T16:44:41.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:41 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:41.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:41 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:41.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:41 smithi002 ceph-mon[128712]: from='osd.6 [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T16:44:41.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:41 smithi002 ceph-mon[128712]: osdmap e45: 7 total, 6 up, 7 in 2023-11-16T16:44:41.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:41 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:41.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:41 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:41.938 INFO:teuthology.orchestra.run.smithi141.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T16:44:42.490 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: purged_snaps scrub starts 2023-11-16T16:44:42.490 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: purged_snaps scrub ok 2023-11-16T16:44:42.490 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: osd.6 [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640] boot 2023-11-16T16:44:42.490 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: osdmap e46: 7 total, 7 up, 7 in 2023-11-16T16:44:42.490 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:42.490 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: Detected new or changed devices on smithi002 2023-11-16T16:44:42.491 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:42.491 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:42.491 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:42.491 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:42.491 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:42.491 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: Adjusting osd_memory_target on smithi002 to 5758M 2023-11-16T16:44:42.491 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:42.491 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:42.491 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:42.491 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:44:42.491 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:42.491 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:44:42.491 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:44:42.491 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:42 smithi141 ceph-mon[131184]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:42.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: purged_snaps scrub starts 2023-11-16T16:44:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: purged_snaps scrub ok 2023-11-16T16:44:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: osd.6 [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640] boot 2023-11-16T16:44:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: osdmap e46: 7 total, 7 up, 7 in 2023-11-16T16:44:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: Detected new or changed devices on smithi002 2023-11-16T16:44:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: Adjusting osd_memory_target on smithi002 to 5758M 2023-11-16T16:44:42.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:42.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:42.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:42.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:44:42.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:42.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:44:42.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:44:42.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:42 smithi155 ceph-mon[131607]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:42.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: purged_snaps scrub starts 2023-11-16T16:44:42.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: purged_snaps scrub ok 2023-11-16T16:44:42.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: osd.6 [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640] boot 2023-11-16T16:44:42.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: osdmap e46: 7 total, 7 up, 7 in 2023-11-16T16:44:42.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:44:42.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: Detected new or changed devices on smithi002 2023-11-16T16:44:42.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:42.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:42.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:42.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:42.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:44:42.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: Adjusting osd_memory_target on smithi002 to 5758M 2023-11-16T16:44:42.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:42.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:42.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:44:42.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:44:42.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:42.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:44:42.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:44:42.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:42 smithi002 ceph-mon[128712]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:43.109 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.107+0000 7fc225ee6700 1 -- 172.21.15.141:0/2282782339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc220101e80 msgr2=0x7fc220108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:43.109 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.107+0000 7fc225ee6700 1 --2- 172.21.15.141:0/2282782339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc220101e80 0x7fc220108740 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc21800b0a0 tx=0x7fc21800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:44:43.109 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.107+0000 7fc225ee6700 1 -- 172.21.15.141:0/2282782339 shutdown_connections 2023-11-16T16:44:43.109 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.107+0000 7fc225ee6700 1 --2- 172.21.15.141:0/2282782339 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2201002f0 0x7fc220100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:43.109 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.107+0000 7fc225ee6700 1 --2- 172.21.15.141:0/2282782339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc220101e80 0x7fc220108740 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:43.109 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.107+0000 7fc225ee6700 1 --2- 172.21.15.141:0/2282782339 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2201014f0 0x7fc220101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:43.110 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.107+0000 7fc225ee6700 1 -- 172.21.15.141:0/2282782339 >> 172.21.15.141:0/2282782339 conn(0x7fc2200fb900 msgr2=0x7fc2200fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:44:43.110 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.107+0000 7fc225ee6700 1 -- 172.21.15.141:0/2282782339 shutdown_connections 2023-11-16T16:44:43.110 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.107+0000 7fc225ee6700 1 -- 172.21.15.141:0/2282782339 wait complete. 2023-11-16T16:44:43.112 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.108+0000 7fc225ee6700 1 Processor -- start 2023-11-16T16:44:43.112 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.108+0000 7fc225ee6700 1 -- start start 2023-11-16T16:44:43.112 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc225ee6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2201002f0 0x7fc220196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:44:43.112 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc225ee6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2201014f0 0x7fc2201967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:44:43.112 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc225ee6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc220101e80 0x7fc22019d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:44:43.112 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc225ee6700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22019de00 con 0x7fc220101e80 2023-11-16T16:44:43.112 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc225ee6700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22019df40 con 0x7fc2201014f0 2023-11-16T16:44:43.112 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc225ee6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22019e0d0 con 0x7fc2201002f0 2023-11-16T16:44:43.112 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc21effd700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2201014f0 0x7fc2201967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:44:43.112 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc21effd700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2201014f0 0x7fc2201967b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:59870/0 (socket says 172.21.15.141:59870) 2023-11-16T16:44:43.112 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc21effd700 1 -- 172.21.15.141:0/1061738684 learned_addr learned my addr 172.21.15.141:0/1061738684 (peer_addr_for_me v2:172.21.15.141:0/0) 2023-11-16T16:44:43.113 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc21ffff700 1 --2- 172.21.15.141:0/1061738684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc220101e80 0x7fc22019d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:44:43.113 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc21f7fe700 1 --2- 172.21.15.141:0/1061738684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2201002f0 0x7fc220196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:44:43.113 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc21effd700 1 -- 172.21.15.141:0/1061738684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2201002f0 msgr2=0x7fc220196270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:43.113 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc21effd700 1 --2- 172.21.15.141:0/1061738684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2201002f0 0x7fc220196270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:43.113 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc21effd700 1 -- 172.21.15.141:0/1061738684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc220101e80 msgr2=0x7fc22019d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:44:43.113 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc21effd700 1 --2- 172.21.15.141:0/1061738684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc220101e80 0x7fc22019d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:44:43.113 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc21effd700 1 -- 172.21.15.141:0/1061738684 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc218009d00 con 0x7fc2201014f0 2023-11-16T16:44:43.113 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc21effd700 1 --2- 172.21.15.141:0/1061738684 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2201014f0 0x7fc2201967b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc21400b530 tx=0x7fc21400b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:44:43.113 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc21cff9700 1 -- 172.21.15.141:0/1061738684 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc214011830 con 0x7fc2201014f0 2023-11-16T16:44:43.113 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc21cff9700 1 -- 172.21.15.141:0/1061738684 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc214011e50 con 0x7fc2201014f0 2023-11-16T16:44:43.113 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc225ee6700 1 -- 172.21.15.141:0/1061738684 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc22010bd60 con 0x7fc2201014f0 2023-11-16T16:44:43.113 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.109+0000 7fc21cff9700 1 -- 172.21.15.141:0/1061738684 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21400f2e0 con 0x7fc2201014f0 2023-11-16T16:44:43.113 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.110+0000 7fc225ee6700 1 -- 172.21.15.141:0/1061738684 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc22010bbd0 con 0x7fc2201014f0 2023-11-16T16:44:43.114 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.111+0000 7fc225ee6700 1 -- 172.21.15.141:0/1061738684 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2201078e0 con 0x7fc2201014f0 2023-11-16T16:44:43.114 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.112+0000 7fc21cff9700 1 -- 172.21.15.141:0/1061738684 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fc214010580 con 0x7fc2201014f0 2023-11-16T16:44:43.115 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.112+0000 7fc21cff9700 1 --2- 172.21.15.141:0/1061738684 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2080a1150 0x7fc2080a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:44:43.115 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.112+0000 7fc21cff9700 1 -- 172.21.15.141:0/1061738684 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7fc2140cb6f0 con 0x7fc2201014f0 2023-11-16T16:44:43.115 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.112+0000 7fc21f7fe700 1 --2- 172.21.15.141:0/1061738684 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2080a1150 0x7fc2080a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:44:43.115 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.113+0000 7fc21f7fe700 1 --2- 172.21.15.141:0/1061738684 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2080a1150 0x7fc2080a3600 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc210006fd0 tx=0x7fc210008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:44:43.118 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.116+0000 7fc21cff9700 1 -- 172.21.15.141:0/1061738684 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc214095b50 con 0x7fc2201014f0 2023-11-16T16:44:43.313 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:44:43.311+0000 7fc225ee6700 1 -- 172.21.15.141:0/1061738684 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi141:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fc220061c40 con 0x7fc2080a1150 2023-11-16T16:44:43.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:43 smithi141 ceph-mon[131184]: osdmap e47: 7 total, 7 up, 7 in 2023-11-16T16:44:43.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:43 smithi155 ceph-mon[131607]: osdmap e47: 7 total, 7 up, 7 in 2023-11-16T16:44:43.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:43 smithi002 ceph-mon[128712]: osdmap e47: 7 total, 7 up, 7 in 2023-11-16T16:44:44.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:44 smithi141 ceph-mon[131184]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi141:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:44:44.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:44:44.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:44:44.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:44.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:44 smithi141 ceph-mon[131184]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:44.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:44 smithi155 ceph-mon[131607]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi141:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:44:44.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:44:44.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:44:44.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:44.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:44 smithi155 ceph-mon[131607]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:44.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:44 smithi002 ceph-mon[128712]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi141:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:44:44.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:44:44.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:44:44.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:44.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:44 smithi002 ceph-mon[128712]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:47.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:46 smithi155 ceph-mon[131607]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:47.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:46 smithi155 ceph-mon[131607]: from='client.? 172.21.15.141:0/3890072457' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46"}]: dispatch 2023-11-16T16:44:47.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:46 smithi155 ceph-mon[131607]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46"}]: dispatch 2023-11-16T16:44:47.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:46 smithi155 ceph-mon[131607]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46"}]': finished 2023-11-16T16:44:47.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:46 smithi155 ceph-mon[131607]: osdmap e48: 8 total, 7 up, 8 in 2023-11-16T16:44:47.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:46 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:44:47.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:46 smithi155 ceph-mon[131607]: from='client.? 172.21.15.141:0/1259908970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:44:47.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:46 smithi002 ceph-mon[128712]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:47.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:46 smithi002 ceph-mon[128712]: from='client.? 172.21.15.141:0/3890072457' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46"}]: dispatch 2023-11-16T16:44:47.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:46 smithi002 ceph-mon[128712]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46"}]: dispatch 2023-11-16T16:44:47.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:46 smithi002 ceph-mon[128712]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46"}]': finished 2023-11-16T16:44:47.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:46 smithi002 ceph-mon[128712]: osdmap e48: 8 total, 7 up, 8 in 2023-11-16T16:44:47.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:46 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:44:47.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:46 smithi002 ceph-mon[128712]: from='client.? 172.21.15.141:0/1259908970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:44:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:46 smithi141 ceph-mon[131184]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:46 smithi141 ceph-mon[131184]: from='client.? 172.21.15.141:0/3890072457' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46"}]: dispatch 2023-11-16T16:44:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:46 smithi141 ceph-mon[131184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46"}]: dispatch 2023-11-16T16:44:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:46 smithi141 ceph-mon[131184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46"}]': finished 2023-11-16T16:44:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:46 smithi141 ceph-mon[131184]: osdmap e48: 8 total, 7 up, 8 in 2023-11-16T16:44:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:46 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:44:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:46 smithi141 ceph-mon[131184]: from='client.? 172.21.15.141:0/1259908970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:44:49.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:48 smithi155 ceph-mon[131607]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:49.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:48 smithi002 ceph-mon[128712]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:49.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:48 smithi141 ceph-mon[131184]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:51.135 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:50 smithi141 ceph-mon[131184]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:51.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:50 smithi155 ceph-mon[131607]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:51.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:50 smithi002 ceph-mon[128712]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:53.223 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:52 smithi141 ceph-mon[131184]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:53.223 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-11-16T16:44:53.223 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:52 smithi155 ceph-mon[131607]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-11-16T16:44:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:53.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:52 smithi002 ceph-mon[128712]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:53.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-11-16T16:44:53.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:44:54.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:53 smithi141 ceph-mon[131184]: Deploying daemon osd.7 on smithi141 2023-11-16T16:44:54.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:53 smithi155 ceph-mon[131607]: Deploying daemon osd.7 on smithi141 2023-11-16T16:44:54.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:53 smithi002 ceph-mon[128712]: Deploying daemon osd.7 on smithi141 2023-11-16T16:44:55.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:54 smithi141 ceph-mon[131184]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:55.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:54 smithi155 ceph-mon[131607]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:55.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:54 smithi002 ceph-mon[128712]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:57.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:56 smithi141 ceph-mon[131184]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:57.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:56 smithi155 ceph-mon[131607]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:57.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:56 smithi002 ceph-mon[128712]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:58.333 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:58 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:58.333 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:58 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:58.333 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:58 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:58.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:58 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:58.397 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:58 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:58.397 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:58 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:58.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:58 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:58.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:58 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:44:58.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:58 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:44:59.208 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:44:59 smithi141 ceph-mon[131184]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:59.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:44:59 smithi155 ceph-mon[131607]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:44:59.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:44:59 smithi002 ceph-mon[128712]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:45:00.230 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:00.228+0000 7fc21cff9700 1 -- 172.21.15.141:0/1061738684 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc220061c40 con 0x7fc2080a1150 2023-11-16T16:45:00.231 INFO:teuthology.orchestra.run.smithi141.stdout:Created osd(s) 7 on host 'smithi141' 2023-11-16T16:45:00.232 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:00.231+0000 7fc225ee6700 1 -- 172.21.15.141:0/1061738684 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2080a1150 msgr2=0x7fc2080a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:00.232 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:00.231+0000 7fc225ee6700 1 --2- 172.21.15.141:0/1061738684 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2080a1150 0x7fc2080a3600 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc210006fd0 tx=0x7fc210008040 comp rx=0 tx=0).stop 2023-11-16T16:45:00.233 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:00.231+0000 7fc225ee6700 1 -- 172.21.15.141:0/1061738684 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2201014f0 msgr2=0x7fc2201967b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:00.233 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:00.231+0000 7fc225ee6700 1 --2- 172.21.15.141:0/1061738684 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2201014f0 0x7fc2201967b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc21400b530 tx=0x7fc21400b8f0 comp rx=0 tx=0).stop 2023-11-16T16:45:00.233 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:00.231+0000 7fc225ee6700 1 -- 172.21.15.141:0/1061738684 shutdown_connections 2023-11-16T16:45:00.233 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:00.231+0000 7fc225ee6700 1 --2- 172.21.15.141:0/1061738684 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2080a1150 0x7fc2080a3600 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:00.233 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:00.231+0000 7fc225ee6700 1 --2- 172.21.15.141:0/1061738684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2201002f0 0x7fc220196270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:00.234 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:00.231+0000 7fc225ee6700 1 --2- 172.21.15.141:0/1061738684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc220101e80 0x7fc22019d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:00.234 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:00.231+0000 7fc225ee6700 1 --2- 172.21.15.141:0/1061738684 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2201014f0 0x7fc2201967b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:00.234 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:00.231+0000 7fc225ee6700 1 -- 172.21.15.141:0/1061738684 >> 172.21.15.141:0/1061738684 conn(0x7fc2200fb900 msgr2=0x7fc220104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:45:00.235 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:00.232+0000 7fc225ee6700 1 -- 172.21.15.141:0/1061738684 shutdown_connections 2023-11-16T16:45:00.235 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:00.232+0000 7fc225ee6700 1 -- 172.21.15.141:0/1061738684 wait complete. 2023-11-16T16:45:00.615 DEBUG:teuthology.orchestra.run.smithi141:osd.7> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.7.service 2023-11-16T16:45:00.617 INFO:tasks.cephadm:Deploying osd.8 on smithi155 with /dev/vg_nvme/lv_2... 2023-11-16T16:45:00.618 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-11-16T16:45:00.647 INFO:journalctl@ceph.osd.7.smithi141.stdout:-- Logs begin at Thu 2023-11-16 16:14:35 UTC. -- 2023-11-16T16:45:00.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:00 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:00.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:00 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:00.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:00 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:00.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:00 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:00.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:00 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:45:00.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:00 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:00.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:00 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:45:00.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:00 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:45:00.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:00 smithi155 ceph-mon[131607]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:45:00.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:00 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:00.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:00 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:00.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:00 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:00.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:00 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:00.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:00 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:00.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:00 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:00.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:00 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:00.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:00 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:45:00.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:00 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:00.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:00 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:45:00.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:00 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:45:00.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:00 smithi002 ceph-mon[128712]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:45:00.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:00 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:00.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:00 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:00.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:00 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:00.989 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:00 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:00.989 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:00 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:00.989 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:00 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:00.989 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:00 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:00.989 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:00 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:45:00.989 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:00 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:00.989 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:00 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:45:00.989 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:00 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:45:00.989 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:00 smithi141 ceph-mon[131184]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:45:00.989 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:00 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:00.990 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:00 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:00.990 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:00 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:01.264 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 16:45:00 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T16:45:00.985+0000 7fd9ac348640 -1 osd.7 0 log_to_monitors true 2023-11-16T16:45:01.272 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:45:01.725 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:01 smithi155 ceph-mon[131607]: from='osd.7 [v2:172.21.15.141:6816/372138580,v1:172.21.15.141:6817/372138580]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T16:45:01.725 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:01 smithi155 ceph-mon[131607]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T16:45:01.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:01 smithi002 ceph-mon[128712]: from='osd.7 [v2:172.21.15.141:6816/372138580,v1:172.21.15.141:6817/372138580]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T16:45:01.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:01 smithi002 ceph-mon[128712]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T16:45:01.906 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:01 smithi141 ceph-mon[131184]: from='osd.7 [v2:172.21.15.141:6816/372138580,v1:172.21.15.141:6817/372138580]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T16:45:01.906 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:01 smithi141 ceph-mon[131184]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-16T16:45:02.703 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 16:45:02 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T16:45:02.591+0000 7fd9a19bf700 -1 osd.7 0 waiting for initial osdmap 2023-11-16T16:45:02.703 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 16:45:02 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T16:45:02.604+0000 7fd99b7b0700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:45:02.703 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:02 smithi141 ceph-mon[131184]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-11-16T16:45:02.703 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:02 smithi141 ceph-mon[131184]: osdmap e49: 8 total, 7 up, 8 in 2023-11-16T16:45:02.703 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:02 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:45:02.704 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:02 smithi141 ceph-mon[131184]: from='osd.7 [v2:172.21.15.141:6816/372138580,v1:172.21.15.141:6817/372138580]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:45:02.704 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:02 smithi141 ceph-mon[131184]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:45:02.704 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:02 smithi141 ceph-mon[131184]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:45:02.719 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-11-16T16:45:02.719 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-11-16T16:45:02.719 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2023-11-16T16:45:02.719 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2023-11-16T16:45:02.719 INFO:teuthology.orchestra.run.smithi155.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125485 s, 836 MB/s 2023-11-16T16:45:02.719 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2023-11-16T16:45:02.841 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:02 smithi155 ceph-mon[131607]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-11-16T16:45:02.841 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:02 smithi155 ceph-mon[131607]: osdmap e49: 8 total, 7 up, 8 in 2023-11-16T16:45:02.842 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:02 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:45:02.842 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:02 smithi155 ceph-mon[131607]: from='osd.7 [v2:172.21.15.141:6816/372138580,v1:172.21.15.141:6817/372138580]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:45:02.842 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:02 smithi155 ceph-mon[131607]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:45:02.842 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:02 smithi155 ceph-mon[131607]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:45:02.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:02 smithi002 ceph-mon[128712]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-11-16T16:45:02.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:02 smithi002 ceph-mon[128712]: osdmap e49: 8 total, 7 up, 8 in 2023-11-16T16:45:02.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:02 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:45:02.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:02 smithi002 ceph-mon[128712]: from='osd.7 [v2:172.21.15.141:6816/372138580,v1:172.21.15.141:6817/372138580]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:45:02.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:02 smithi002 ceph-mon[128712]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:45:02.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:02 smithi002 ceph-mon[128712]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-11-16T16:45:03.112 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi155:vg_nvme/lv_2 2023-11-16T16:45:03.750 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:45:03.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]': finished 2023-11-16T16:45:03.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: osdmap e50: 8 total, 7 up, 8 in 2023-11-16T16:45:03.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:45:03.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:45:03.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: Detected new or changed devices on smithi141 2023-11-16T16:45:03.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:03.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:03.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:03.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:03.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:03.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: Adjusting osd_memory_target on smithi141 to 5722M 2023-11-16T16:45:03.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:03.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:03.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:03.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:45:03.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:03.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:45:03.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:45:03.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: osd.7 [v2:172.21.15.141:6816/372138580,v1:172.21.15.141:6817/372138580] boot 2023-11-16T16:45:03.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: osdmap e51: 8 total, 8 up, 8 in 2023-11-16T16:45:03.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:03 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:45:03.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]': finished 2023-11-16T16:45:03.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: osdmap e50: 8 total, 7 up, 8 in 2023-11-16T16:45:03.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:45:03.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:45:03.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: Detected new or changed devices on smithi141 2023-11-16T16:45:03.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:03.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:03.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:03.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:03.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:03.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: Adjusting osd_memory_target on smithi141 to 5722M 2023-11-16T16:45:03.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:03.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:03.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:03.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:45:03.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:03.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:45:03.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:45:03.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: osd.7 [v2:172.21.15.141:6816/372138580,v1:172.21.15.141:6817/372138580] boot 2023-11-16T16:45:03.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: osdmap e51: 8 total, 8 up, 8 in 2023-11-16T16:45:03.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:03 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:45:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]': finished 2023-11-16T16:45:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: osdmap e50: 8 total, 7 up, 8 in 2023-11-16T16:45:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:45:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:45:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: Detected new or changed devices on smithi141 2023-11-16T16:45:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:04.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: Adjusting osd_memory_target on smithi141 to 5722M 2023-11-16T16:45:04.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:04.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:04.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:04.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:45:04.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:04.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:45:04.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:45:04.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: osd.7 [v2:172.21.15.141:6816/372138580,v1:172.21.15.141:6817/372138580] boot 2023-11-16T16:45:04.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: osdmap e51: 8 total, 8 up, 8 in 2023-11-16T16:45:04.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:03 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:45:04.932 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.930+0000 7febbeaa1700 1 -- 172.21.15.155:0/1220921987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febb8101510 msgr2=0x7febb8101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:04.932 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.930+0000 7febbeaa1700 1 --2- 172.21.15.155:0/1220921987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febb8101510 0x7febb8101960 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7feba4009990 tx=0x7feba4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:45:04.932 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.930+0000 7febbeaa1700 1 -- 172.21.15.155:0/1220921987 shutdown_connections 2023-11-16T16:45:04.932 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.930+0000 7febbeaa1700 1 --2- 172.21.15.155:0/1220921987 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7febb8101ea0 0x7febb8108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:04.932 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.930+0000 7febbeaa1700 1 --2- 172.21.15.155:0/1220921987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febb8100310 0x7febb8100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:04.932 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.930+0000 7febbeaa1700 1 --2- 172.21.15.155:0/1220921987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febb8101510 0x7febb8101960 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:04.933 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.930+0000 7febbeaa1700 1 -- 172.21.15.155:0/1220921987 >> 172.21.15.155:0/1220921987 conn(0x7febb80fb960 msgr2=0x7febb80fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:45:04.933 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.930+0000 7febbeaa1700 1 -- 172.21.15.155:0/1220921987 shutdown_connections 2023-11-16T16:45:04.933 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.930+0000 7febbeaa1700 1 -- 172.21.15.155:0/1220921987 wait complete. 2023-11-16T16:45:04.935 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.931+0000 7febbeaa1700 1 Processor -- start 2023-11-16T16:45:04.935 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.931+0000 7febbeaa1700 1 -- start start 2023-11-16T16:45:04.935 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.931+0000 7febbeaa1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febb8100310 0x7febb81960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:04.935 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.931+0000 7febbeaa1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febb8101510 0x7febb8196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:04.935 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.931+0000 7febbeaa1700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7febb8101ea0 0x7febb819d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:04.935 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.932+0000 7febbeaa1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb819dc50 con 0x7febb8100310 2023-11-16T16:45:04.935 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.932+0000 7febbeaa1700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb819dd90 con 0x7febb8101ea0 2023-11-16T16:45:04.935 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.932+0000 7febbeaa1700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb819df20 con 0x7febb8101510 2023-11-16T16:45:04.935 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.932+0000 7febb7fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febb8101510 0x7febb8196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:04.936 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.932+0000 7febb7fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febb8101510 0x7febb8196600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:39220/0 (socket says 172.21.15.155:39220) 2023-11-16T16:45:04.936 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.932+0000 7febb7fff700 1 -- 172.21.15.155:0/3878531453 learned_addr learned my addr 172.21.15.155:0/3878531453 (peer_addr_for_me v2:172.21.15.155:0/0) 2023-11-16T16:45:04.936 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.932+0000 7febbd03e700 1 --2- 172.21.15.155:0/3878531453 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7febb8101ea0 0x7febb819d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:04.936 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.932+0000 7febb7fff700 1 -- 172.21.15.155:0/3878531453 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7febb8101ea0 msgr2=0x7febb819d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:04.936 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.932+0000 7febb7fff700 1 --2- 172.21.15.155:0/3878531453 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7febb8101ea0 0x7febb819d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:04.936 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.932+0000 7febb7fff700 1 -- 172.21.15.155:0/3878531453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febb8100310 msgr2=0x7febb81960c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:04.936 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.932+0000 7febb7fff700 1 --2- 172.21.15.155:0/3878531453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febb8100310 0x7febb81960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:04.936 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.932+0000 7febb7fff700 1 -- 172.21.15.155:0/3878531453 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba40095f0 con 0x7febb8101510 2023-11-16T16:45:04.936 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.932+0000 7febb7fff700 1 --2- 172.21.15.155:0/3878531453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febb8101510 0x7febb8196600 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7feba4005910 tx=0x7feba400f830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:45:04.936 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.932+0000 7febb5ffb700 1 -- 172.21.15.155:0/3878531453 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba401d070 con 0x7febb8101510 2023-11-16T16:45:04.936 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.932+0000 7febbeaa1700 1 -- 172.21.15.155:0/3878531453 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb810bd10 con 0x7febb8101510 2023-11-16T16:45:04.936 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.932+0000 7febb5ffb700 1 -- 172.21.15.155:0/3878531453 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feba400fd10 con 0x7febb8101510 2023-11-16T16:45:04.937 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.932+0000 7febb5ffb700 1 -- 172.21.15.155:0/3878531453 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba40176e0 con 0x7febb8101510 2023-11-16T16:45:04.937 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.933+0000 7febbeaa1700 1 -- 172.21.15.155:0/3878531453 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb810bb80 con 0x7febb8101510 2023-11-16T16:45:04.937 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.934+0000 7febbeaa1700 1 -- 172.21.15.155:0/3878531453 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb81078e0 con 0x7febb8101510 2023-11-16T16:45:04.938 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.935+0000 7febb5ffb700 1 -- 172.21.15.155:0/3878531453 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7feba4022470 con 0x7febb8101510 2023-11-16T16:45:04.938 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.935+0000 7febb5ffb700 1 --2- 172.21.15.155:0/3878531453 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7feba80a1150 0x7feba80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:04.938 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.935+0000 7febb5ffb700 1 -- 172.21.15.155:0/3878531453 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5426+0+0 (secure 0 0 0) 0x7feba40cd210 con 0x7febb8101510 2023-11-16T16:45:04.938 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.936+0000 7febbc83d700 1 --2- 172.21.15.155:0/3878531453 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7feba80a1150 0x7feba80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:04.938 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.936+0000 7febbc83d700 1 --2- 172.21.15.155:0/3878531453 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7feba80a1150 0x7feba80a3600 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7febb8101370 tx=0x7febac006d20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:45:04.941 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:04.940+0000 7febb5ffb700 1 -- 172.21.15.155:0/3878531453 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7feba405c730 con 0x7febb8101510 2023-11-16T16:45:05.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:04 smithi155 ceph-mon[131607]: purged_snaps scrub starts 2023-11-16T16:45:05.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:04 smithi155 ceph-mon[131607]: purged_snaps scrub ok 2023-11-16T16:45:05.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:04 smithi155 ceph-mon[131607]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:05.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:04 smithi155 ceph-mon[131607]: osdmap e52: 8 total, 8 up, 8 in 2023-11-16T16:45:05.137 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:05.136+0000 7febbeaa1700 1 -- 172.21.15.155:0/3878531453 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7febb8060f30 con 0x7feba80a1150 2023-11-16T16:45:05.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:04 smithi141 ceph-mon[131184]: purged_snaps scrub starts 2023-11-16T16:45:05.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:04 smithi141 ceph-mon[131184]: purged_snaps scrub ok 2023-11-16T16:45:05.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:04 smithi141 ceph-mon[131184]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:05.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:04 smithi141 ceph-mon[131184]: osdmap e52: 8 total, 8 up, 8 in 2023-11-16T16:45:05.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:04 smithi002 ceph-mon[128712]: purged_snaps scrub starts 2023-11-16T16:45:05.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:04 smithi002 ceph-mon[128712]: purged_snaps scrub ok 2023-11-16T16:45:05.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:04 smithi002 ceph-mon[128712]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:05.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:04 smithi002 ceph-mon[128712]: osdmap e52: 8 total, 8 up, 8 in 2023-11-16T16:45:06.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:05 smithi141 ceph-mon[131184]: osdmap e53: 8 total, 8 up, 8 in 2023-11-16T16:45:06.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:05 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:45:06.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:05 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:45:06.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:05 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:06.307 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:05 smithi155 ceph-mon[131607]: osdmap e53: 8 total, 8 up, 8 in 2023-11-16T16:45:06.307 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:05 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:45:06.307 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:05 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:45:06.308 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:05 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:06.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:05 smithi002 ceph-mon[128712]: osdmap e53: 8 total, 8 up, 8 in 2023-11-16T16:45:06.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:05 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:45:06.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:05 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:45:06.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:05 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:07.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:06 smithi155 ceph-mon[131607]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:45:07.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:06 smithi155 ceph-mon[131607]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:07.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:06 smithi155 ceph-mon[131607]: osdmap e54: 8 total, 8 up, 8 in 2023-11-16T16:45:07.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:06 smithi141 ceph-mon[131184]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:45:07.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:06 smithi141 ceph-mon[131184]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:07.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:06 smithi141 ceph-mon[131184]: osdmap e54: 8 total, 8 up, 8 in 2023-11-16T16:45:07.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:06 smithi002 ceph-mon[128712]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:45:07.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:06 smithi002 ceph-mon[128712]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 45 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:07.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:06 smithi002 ceph-mon[128712]: osdmap e54: 8 total, 8 up, 8 in 2023-11-16T16:45:08.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:07 smithi141 ceph-mon[131184]: osdmap e55: 8 total, 8 up, 8 in 2023-11-16T16:45:08.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:07 smithi141 ceph-mon[131184]: from='client.? 172.21.15.155:0/2972157114' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd6d5d76-9af2-49f8-a769-3ef0fb0599dc"}]: dispatch 2023-11-16T16:45:08.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:07 smithi141 ceph-mon[131184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd6d5d76-9af2-49f8-a769-3ef0fb0599dc"}]: dispatch 2023-11-16T16:45:08.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:07 smithi141 ceph-mon[131184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd6d5d76-9af2-49f8-a769-3ef0fb0599dc"}]': finished 2023-11-16T16:45:08.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:07 smithi141 ceph-mon[131184]: osdmap e56: 9 total, 8 up, 9 in 2023-11-16T16:45:08.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:07 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:08.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:07 smithi155 ceph-mon[131607]: osdmap e55: 8 total, 8 up, 8 in 2023-11-16T16:45:08.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:07 smithi155 ceph-mon[131607]: from='client.? 172.21.15.155:0/2972157114' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd6d5d76-9af2-49f8-a769-3ef0fb0599dc"}]: dispatch 2023-11-16T16:45:08.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:07 smithi155 ceph-mon[131607]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd6d5d76-9af2-49f8-a769-3ef0fb0599dc"}]: dispatch 2023-11-16T16:45:08.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:07 smithi155 ceph-mon[131607]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd6d5d76-9af2-49f8-a769-3ef0fb0599dc"}]': finished 2023-11-16T16:45:08.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:07 smithi155 ceph-mon[131607]: osdmap e56: 9 total, 8 up, 9 in 2023-11-16T16:45:08.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:07 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:08.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:07 smithi002 ceph-mon[128712]: osdmap e55: 8 total, 8 up, 8 in 2023-11-16T16:45:08.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:07 smithi002 ceph-mon[128712]: from='client.? 172.21.15.155:0/2972157114' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd6d5d76-9af2-49f8-a769-3ef0fb0599dc"}]: dispatch 2023-11-16T16:45:08.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:07 smithi002 ceph-mon[128712]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd6d5d76-9af2-49f8-a769-3ef0fb0599dc"}]: dispatch 2023-11-16T16:45:08.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:07 smithi002 ceph-mon[128712]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd6d5d76-9af2-49f8-a769-3ef0fb0599dc"}]': finished 2023-11-16T16:45:08.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:07 smithi002 ceph-mon[128712]: osdmap e56: 9 total, 8 up, 9 in 2023-11-16T16:45:08.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:07 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:09.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:08 smithi141 ceph-mon[131184]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:09.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:08 smithi141 ceph-mon[131184]: from='client.? 172.21.15.155:0/5913599' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:45:09.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:08 smithi155 ceph-mon[131607]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:09.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:08 smithi155 ceph-mon[131607]: from='client.? 172.21.15.155:0/5913599' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:45:09.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:08 smithi002 ceph-mon[128712]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:09.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:08 smithi002 ceph-mon[128712]: from='client.? 172.21.15.155:0/5913599' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:45:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:10 smithi141 ceph-mon[131184]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:11.278 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:10 smithi155 ceph-mon[131607]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:11.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:10 smithi002 ceph-mon[128712]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:13.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:12 smithi155 ceph-mon[131607]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:13.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:12 smithi141 ceph-mon[131184]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:13.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:12 smithi002 ceph-mon[128712]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:14 smithi141 ceph-mon[131184]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:14 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-11-16T16:45:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:14 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:14 smithi141 ceph-mon[131184]: Deploying daemon osd.8 on smithi155 2023-11-16T16:45:15.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:14 smithi155 ceph-mon[131607]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:15.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:14 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-11-16T16:45:15.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:14 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:15.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:14 smithi155 ceph-mon[131607]: Deploying daemon osd.8 on smithi155 2023-11-16T16:45:15.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:14 smithi002 ceph-mon[128712]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:15.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:14 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-11-16T16:45:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:14 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:14 smithi002 ceph-mon[128712]: Deploying daemon osd.8 on smithi155 2023-11-16T16:45:17.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:16 smithi155 ceph-mon[131607]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:16 smithi141 ceph-mon[131184]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:17.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:16 smithi002 ceph-mon[128712]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:19.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:18 smithi155 ceph-mon[131607]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:19.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:18 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:19.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:18 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:19.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:18 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:18 smithi141 ceph-mon[131184]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:18 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:18 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:18 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:19.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:18 smithi002 ceph-mon[128712]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:19.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:18 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:19.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:18 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:19.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:18 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:21.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:20 smithi155 ceph-mon[131607]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:21.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:20 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:21.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:20 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:21.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:20 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:21.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:20 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:21.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:20 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:21.199 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:21.198+0000 7febb5ffb700 1 -- 172.21.15.155:0/3878531453 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7febb8060f30 con 0x7feba80a1150 2023-11-16T16:45:21.199 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 8 on host 'smithi155' 2023-11-16T16:45:21.201 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:21.200+0000 7febbeaa1700 1 -- 172.21.15.155:0/3878531453 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7feba80a1150 msgr2=0x7feba80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:21.201 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:21.200+0000 7febbeaa1700 1 --2- 172.21.15.155:0/3878531453 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7feba80a1150 0x7feba80a3600 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7febb8101370 tx=0x7febac006d20 comp rx=0 tx=0).stop 2023-11-16T16:45:21.201 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:21.200+0000 7febbeaa1700 1 -- 172.21.15.155:0/3878531453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febb8101510 msgr2=0x7febb8196600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:21.201 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:21.200+0000 7febbeaa1700 1 --2- 172.21.15.155:0/3878531453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febb8101510 0x7febb8196600 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7feba4005910 tx=0x7feba400f830 comp rx=0 tx=0).stop 2023-11-16T16:45:21.202 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:21.200+0000 7febbeaa1700 1 -- 172.21.15.155:0/3878531453 shutdown_connections 2023-11-16T16:45:21.202 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:21.200+0000 7febbeaa1700 1 --2- 172.21.15.155:0/3878531453 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7feba80a1150 0x7feba80a3600 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:21.202 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:21.200+0000 7febbeaa1700 1 --2- 172.21.15.155:0/3878531453 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7febb8101ea0 0x7febb819d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:21.202 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:21.200+0000 7febbeaa1700 1 --2- 172.21.15.155:0/3878531453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7febb8100310 0x7febb81960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:21.202 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:21.200+0000 7febbeaa1700 1 --2- 172.21.15.155:0/3878531453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febb8101510 0x7febb8196600 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:21.202 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:21.200+0000 7febbeaa1700 1 -- 172.21.15.155:0/3878531453 >> 172.21.15.155:0/3878531453 conn(0x7febb80fb960 msgr2=0x7febb8104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:45:21.202 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:21.200+0000 7febbeaa1700 1 -- 172.21.15.155:0/3878531453 shutdown_connections 2023-11-16T16:45:21.202 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:45:21.200+0000 7febbeaa1700 1 -- 172.21.15.155:0/3878531453 wait complete. 2023-11-16T16:45:21.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:20 smithi141 ceph-mon[131184]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:20 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:20 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:20 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:20 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:20 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:21.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:20 smithi002 ceph-mon[128712]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:21.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:20 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:21.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:20 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:21.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:20 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:21.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:20 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:21.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:20 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:21.649 DEBUG:teuthology.orchestra.run.smithi155:osd.8> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.8.service 2023-11-16T16:45:21.651 INFO:tasks.cephadm:Deploying osd.9 on smithi002 with /dev/vg_nvme/lv_1... 2023-11-16T16:45:21.652 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-11-16T16:45:21.681 INFO:journalctl@ceph.osd.8.smithi155.stdout:-- Logs begin at Thu 2023-11-16 16:15:48 UTC. -- 2023-11-16T16:45:22.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:21 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:45:22.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:21 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:45:22.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:21 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:45:22.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:21 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:22.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:21 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:22.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:21 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:22.295 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:21 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:45:22.295 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:21 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:45:22.295 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:21 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:45:22.296 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:21 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:22.296 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:21 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:22.296 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:21 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:22.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:21 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:45:22.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:21 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:45:22.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:21 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:45:22.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:21 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:22.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:21 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:22.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:21 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:22.525 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:45:22.617 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 16:45:22 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T16:45:22.358+0000 7f8d503d0640 -1 osd.8 0 log_to_monitors true 2023-11-16T16:45:23.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:22 smithi002 ceph-mon[128712]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:23.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:22 smithi002 ceph-mon[128712]: from='osd.8 [v2:172.21.15.155:6816/3808747539,v1:172.21.15.155:6817/3808747539]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T16:45:23.072 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:22 smithi002 ceph-mon[128712]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T16:45:23.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:22 smithi141 ceph-mon[131184]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:23.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:22 smithi141 ceph-mon[131184]: from='osd.8 [v2:172.21.15.155:6816/3808747539,v1:172.21.15.155:6817/3808747539]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T16:45:23.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:22 smithi141 ceph-mon[131184]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T16:45:23.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:22 smithi155 ceph-mon[131607]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:23.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:22 smithi155 ceph-mon[131607]: from='osd.8 [v2:172.21.15.155:6816/3808747539,v1:172.21.15.155:6817/3808747539]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T16:45:23.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:22 smithi155 ceph-mon[131607]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-11-16T16:45:24.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-11-16T16:45:24.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: osdmap e57: 9 total, 8 up, 9 in 2023-11-16T16:45:24.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:24.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: from='osd.8 [v2:172.21.15.155:6816/3808747539,v1:172.21.15.155:6817/3808747539]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:45:24.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:45:24.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:24.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: Detected new or changed devices on smithi155 2023-11-16T16:45:24.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:24.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:24.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:24.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:24.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:24.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: Adjusting osd_memory_target on smithi155 to 5722M 2023-11-16T16:45:24.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:24.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:24.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:24.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:45:24.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:24.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:45:24.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:24 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:45:24.554 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-11-16T16:45:24.554 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-11-16T16:45:24.554 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2023-11-16T16:45:24.554 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2023-11-16T16:45:24.555 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0429839 s, 244 MB/s 2023-11-16T16:45:24.555 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2023-11-16T16:45:24.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-11-16T16:45:24.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: osdmap e57: 9 total, 8 up, 9 in 2023-11-16T16:45:24.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:24.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: from='osd.8 [v2:172.21.15.155:6816/3808747539,v1:172.21.15.155:6817/3808747539]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:45:24.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:45:24.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:24.591 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: Detected new or changed devices on smithi155 2023-11-16T16:45:24.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:24.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:24.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:24.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:24.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:24.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: Adjusting osd_memory_target on smithi155 to 5722M 2023-11-16T16:45:24.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:24.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:24.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:24.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:45:24.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:24.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:45:24.592 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:24 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:45:24.617 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T16:45:24.160+0000 7f8d45a47700 -1 osd.8 0 waiting for initial osdmap 2023-11-16T16:45:24.618 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T16:45:24.172+0000 7f8d4183c700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:45:24.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-11-16T16:45:24.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: osdmap e57: 9 total, 8 up, 9 in 2023-11-16T16:45:24.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:24.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: from='osd.8 [v2:172.21.15.155:6816/3808747539,v1:172.21.15.155:6817/3808747539]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:45:24.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:45:24.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:24.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: Detected new or changed devices on smithi155 2023-11-16T16:45:24.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:24.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:24.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:24.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:24.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:24.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: Adjusting osd_memory_target on smithi155 to 5722M 2023-11-16T16:45:24.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:24.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:24.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:24.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:45:24.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:24.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:45:24.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:24 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:45:25.010 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi002:vg_nvme/lv_1 2023-11-16T16:45:25.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:25 smithi002 ceph-mon[128712]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2023-11-16T16:45:25.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:25 smithi002 ceph-mon[128712]: osdmap e58: 9 total, 8 up, 9 in 2023-11-16T16:45:25.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:25 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:25.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:25 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:25.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:25 smithi141 ceph-mon[131184]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2023-11-16T16:45:25.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:25 smithi141 ceph-mon[131184]: osdmap e58: 9 total, 8 up, 9 in 2023-11-16T16:45:25.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:25 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:25.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:25 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:25.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:25 smithi155 ceph-mon[131607]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2023-11-16T16:45:25.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:25 smithi155 ceph-mon[131607]: osdmap e58: 9 total, 8 up, 9 in 2023-11-16T16:45:25.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:25 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:25.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:25 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:25.873 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:45:26.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:26 smithi002 ceph-mon[128712]: purged_snaps scrub starts 2023-11-16T16:45:26.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:26 smithi002 ceph-mon[128712]: purged_snaps scrub ok 2023-11-16T16:45:26.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:26 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:26.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:26 smithi002 ceph-mon[128712]: osd.8 [v2:172.21.15.155:6816/3808747539,v1:172.21.15.155:6817/3808747539] boot 2023-11-16T16:45:26.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:26 smithi002 ceph-mon[128712]: osdmap e59: 9 total, 9 up, 9 in 2023-11-16T16:45:26.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:26 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:26.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:26 smithi002 ceph-mon[128712]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:26.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:26 smithi141 ceph-mon[131184]: purged_snaps scrub starts 2023-11-16T16:45:26.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:26 smithi141 ceph-mon[131184]: purged_snaps scrub ok 2023-11-16T16:45:26.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:26 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:26.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:26 smithi141 ceph-mon[131184]: osd.8 [v2:172.21.15.155:6816/3808747539,v1:172.21.15.155:6817/3808747539] boot 2023-11-16T16:45:26.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:26 smithi141 ceph-mon[131184]: osdmap e59: 9 total, 9 up, 9 in 2023-11-16T16:45:26.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:26 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:26.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:26 smithi141 ceph-mon[131184]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:26.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:26 smithi155 ceph-mon[131607]: purged_snaps scrub starts 2023-11-16T16:45:26.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:26 smithi155 ceph-mon[131607]: purged_snaps scrub ok 2023-11-16T16:45:26.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:26.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:26 smithi155 ceph-mon[131607]: osd.8 [v2:172.21.15.155:6816/3808747539,v1:172.21.15.155:6817/3808747539] boot 2023-11-16T16:45:26.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:26 smithi155 ceph-mon[131607]: osdmap e59: 9 total, 9 up, 9 in 2023-11-16T16:45:26.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:45:26.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:26 smithi155 ceph-mon[131607]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-11-16T16:45:27.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.384+0000 7f8cf1603700 1 -- 172.21.15.2:0/3553306154 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8cec1014f0 msgr2=0x7f8cec101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:27.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.384+0000 7f8cf1603700 1 --2- 172.21.15.2:0/3553306154 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8cec1014f0 0x7f8cec101940 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8ce00098f0 tx=0x7f8ce0009c00 comp rx=0 tx=0).stop 2023-11-16T16:45:27.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.384+0000 7f8cf1603700 1 -- 172.21.15.2:0/3553306154 shutdown_connections 2023-11-16T16:45:27.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.384+0000 7f8cf1603700 1 --2- 172.21.15.2:0/3553306154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cec1002f0 0x7f8cec100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:27.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.384+0000 7f8cf1603700 1 --2- 172.21.15.2:0/3553306154 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8cec101e80 0x7f8cec108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:27.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.384+0000 7f8cf1603700 1 --2- 172.21.15.2:0/3553306154 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8cec1014f0 0x7f8cec101940 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:27.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.384+0000 7f8cf1603700 1 -- 172.21.15.2:0/3553306154 >> 172.21.15.2:0/3553306154 conn(0x7f8cec0fb900 msgr2=0x7f8cec0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:45:27.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.384+0000 7f8cf1603700 1 -- 172.21.15.2:0/3553306154 shutdown_connections 2023-11-16T16:45:27.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.384+0000 7f8cf1603700 1 -- 172.21.15.2:0/3553306154 wait complete. 2023-11-16T16:45:27.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.385+0000 7f8cf1603700 1 Processor -- start 2023-11-16T16:45:27.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.386+0000 7f8cf1603700 1 -- start start 2023-11-16T16:45:27.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.386+0000 7f8cf1603700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cec1002f0 0x7f8cec1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:27.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.386+0000 7f8cf1603700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8cec1014f0 0x7f8cec1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:27.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.386+0000 7f8cf1603700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8cec101e80 0x7f8cec19f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:27.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.386+0000 7f8cf1603700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec19fe40 con 0x7f8cec1014f0 2023-11-16T16:45:27.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.386+0000 7f8cf1603700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec19ff80 con 0x7f8cec101e80 2023-11-16T16:45:27.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.386+0000 7f8cf1603700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec1a0110 con 0x7f8cec1002f0 2023-11-16T16:45:27.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.386+0000 7f8cea7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8cec1014f0 0x7f8cec1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:27.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.386+0000 7f8cea7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8cec1014f0 0x7f8cec1987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60266/0 (socket says 172.21.15.2:60266) 2023-11-16T16:45:27.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.386+0000 7f8ceaffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cec1002f0 0x7f8cec1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:27.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.386+0000 7f8cea7fc700 1 -- 172.21.15.2:0/2279612101 learned_addr learned my addr 172.21.15.2:0/2279612101 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:45:27.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.386+0000 7f8ceb7fe700 1 --2- 172.21.15.2:0/2279612101 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8cec101e80 0x7f8cec19f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:27.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.387+0000 7f8cea7fc700 1 -- 172.21.15.2:0/2279612101 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cec1002f0 msgr2=0x7f8cec1982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:27.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.387+0000 7f8cea7fc700 1 --2- 172.21.15.2:0/2279612101 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cec1002f0 0x7f8cec1982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:27.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.387+0000 7f8cea7fc700 1 -- 172.21.15.2:0/2279612101 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8cec101e80 msgr2=0x7f8cec19f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:27.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.387+0000 7f8cea7fc700 1 --2- 172.21.15.2:0/2279612101 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8cec101e80 0x7f8cec19f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:27.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.387+0000 7f8cea7fc700 1 -- 172.21.15.2:0/2279612101 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce0009520 con 0x7f8cec1014f0 2023-11-16T16:45:27.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.387+0000 7f8cea7fc700 1 --2- 172.21.15.2:0/2279612101 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8cec1014f0 0x7f8cec1987f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0009230 tx=0x7f8ce000b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:45:27.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.387+0000 7f8cd3fff700 1 -- 172.21.15.2:0/2279612101 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce001c070 con 0x7f8cec1014f0 2023-11-16T16:45:27.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.387+0000 7f8cd3fff700 1 -- 172.21.15.2:0/2279612101 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ce000bb90 con 0x7f8cec1014f0 2023-11-16T16:45:27.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.387+0000 7f8cf1603700 1 -- 172.21.15.2:0/2279612101 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cec10bca0 con 0x7f8cec1014f0 2023-11-16T16:45:27.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.387+0000 7f8cd3fff700 1 -- 172.21.15.2:0/2279612101 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0004cf0 con 0x7f8cec1014f0 2023-11-16T16:45:27.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.387+0000 7f8cf1603700 1 -- 172.21.15.2:0/2279612101 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cec10bb10 con 0x7f8cec1014f0 2023-11-16T16:45:27.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.388+0000 7f8cf1603700 1 -- 172.21.15.2:0/2279612101 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cec192a00 con 0x7f8cec1014f0 2023-11-16T16:45:27.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.390+0000 7f8cd3fff700 1 -- 172.21.15.2:0/2279612101 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f8ce0005040 con 0x7f8cec1014f0 2023-11-16T16:45:27.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.392+0000 7f8cd3fff700 1 --2- 172.21.15.2:0/2279612101 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8cd40a1150 0x7f8cd40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:27.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.392+0000 7f8cd3fff700 1 -- 172.21.15.2:0/2279612101 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5858+0+0 (secure 0 0 0) 0x7f8ce00cd390 con 0x7f8cec1014f0 2023-11-16T16:45:27.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.392+0000 7f8ceaffd700 1 --2- 172.21.15.2:0/2279612101 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8cd40a1150 0x7f8cd40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:27.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.393+0000 7f8ceaffd700 1 --2- 172.21.15.2:0/2279612101 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8cd40a1150 0x7f8cd40a3600 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8cec101350 tx=0x7f8cdc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:45:27.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.396+0000 7f8cd3fff700 1 -- 172.21.15.2:0/2279612101 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8ce0097460 con 0x7f8cec1014f0 2023-11-16T16:45:27.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:27 smithi002 ceph-mon[128712]: osdmap e60: 9 total, 9 up, 9 in 2023-11-16T16:45:27.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:27 smithi141 ceph-mon[131184]: osdmap e60: 9 total, 9 up, 9 in 2023-11-16T16:45:27.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:27.612+0000 7f8cf1603700 1 -- 172.21.15.2:0/2279612101 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8cec061a50 con 0x7f8cd40a1150 2023-11-16T16:45:27.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:27 smithi155 ceph-mon[131607]: osdmap e60: 9 total, 9 up, 9 in 2023-11-16T16:45:28.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:28 smithi141 ceph-mon[131184]: osdmap e61: 9 total, 9 up, 9 in 2023-11-16T16:45:28.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:28 smithi141 ceph-mon[131184]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:45:28.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:28 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:45:28.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:28 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:45:28.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:28 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:28.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:28 smithi141 ceph-mon[131184]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:28.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:28 smithi155 ceph-mon[131607]: osdmap e61: 9 total, 9 up, 9 in 2023-11-16T16:45:28.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:28 smithi155 ceph-mon[131607]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:45:28.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:28 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:45:28.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:28 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:45:28.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:28 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:28.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:28 smithi155 ceph-mon[131607]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:28.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:28 smithi002 ceph-mon[128712]: osdmap e61: 9 total, 9 up, 9 in 2023-11-16T16:45:28.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:28 smithi002 ceph-mon[128712]: from='client.14352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:45:28.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:28 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:45:28.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:28 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:45:28.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:28 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:28.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:28 smithi002 ceph-mon[128712]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:29.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:29 smithi002 ceph-mon[128712]: osdmap e62: 9 total, 9 up, 9 in 2023-11-16T16:45:29.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:29 smithi141 ceph-mon[131184]: osdmap e62: 9 total, 9 up, 9 in 2023-11-16T16:45:29.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:29 smithi155 ceph-mon[131607]: osdmap e62: 9 total, 9 up, 9 in 2023-11-16T16:45:30.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:30 smithi141 ceph-mon[131184]: osdmap e63: 9 total, 9 up, 9 in 2023-11-16T16:45:30.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:30 smithi141 ceph-mon[131184]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T16:45:30.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:30 smithi155 ceph-mon[131607]: osdmap e63: 9 total, 9 up, 9 in 2023-11-16T16:45:30.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:30 smithi155 ceph-mon[131607]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T16:45:30.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:30 smithi002 ceph-mon[128712]: osdmap e63: 9 total, 9 up, 9 in 2023-11-16T16:45:30.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:30 smithi002 ceph-mon[128712]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T16:45:31.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:31 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/926573992' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a2692c3-2f49-4706-9a2b-0cb85882387d"}]: dispatch 2023-11-16T16:45:31.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:31 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/926573992' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a2692c3-2f49-4706-9a2b-0cb85882387d"}]': finished 2023-11-16T16:45:31.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:31 smithi141 ceph-mon[131184]: osdmap e64: 10 total, 9 up, 10 in 2023-11-16T16:45:31.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:31 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:31.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:31 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/926573992' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a2692c3-2f49-4706-9a2b-0cb85882387d"}]: dispatch 2023-11-16T16:45:31.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:31 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/926573992' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a2692c3-2f49-4706-9a2b-0cb85882387d"}]': finished 2023-11-16T16:45:31.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:31 smithi155 ceph-mon[131607]: osdmap e64: 10 total, 9 up, 10 in 2023-11-16T16:45:31.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:31 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:31.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:31 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/926573992' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a2692c3-2f49-4706-9a2b-0cb85882387d"}]: dispatch 2023-11-16T16:45:31.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:31 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/926573992' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a2692c3-2f49-4706-9a2b-0cb85882387d"}]': finished 2023-11-16T16:45:31.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:31 smithi002 ceph-mon[128712]: osdmap e64: 10 total, 9 up, 10 in 2023-11-16T16:45:31.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:31 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:32.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:32 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3788150267' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:45:32.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:32 smithi141 ceph-mon[131184]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 126 KiB/s, 0 objects/s recovering 2023-11-16T16:45:32.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:32 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3788150267' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:45:32.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:32 smithi155 ceph-mon[131607]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 126 KiB/s, 0 objects/s recovering 2023-11-16T16:45:32.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:32 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3788150267' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:45:32.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:32 smithi002 ceph-mon[128712]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 126 KiB/s, 0 objects/s recovering 2023-11-16T16:45:35.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:34 smithi002 ceph-mon[128712]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-16T16:45:35.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:34 smithi141 ceph-mon[131184]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-16T16:45:35.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:34 smithi155 ceph-mon[131607]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-16T16:45:37.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:36 smithi002 ceph-mon[128712]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 76 KiB/s, 0 objects/s recovering 2023-11-16T16:45:37.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:36 smithi141 ceph-mon[131184]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 76 KiB/s, 0 objects/s recovering 2023-11-16T16:45:37.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:36 smithi155 ceph-mon[131607]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 76 KiB/s, 0 objects/s recovering 2023-11-16T16:45:39.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:38 smithi141 ceph-mon[131184]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:39.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:38 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-11-16T16:45:39.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:38 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:39.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:38 smithi155 ceph-mon[131607]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:39.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:38 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-11-16T16:45:39.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:38 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:39.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:38 smithi002 ceph-mon[128712]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:39.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:38 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-11-16T16:45:39.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:38 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:40.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:39 smithi141 ceph-mon[131184]: Deploying daemon osd.9 on smithi002 2023-11-16T16:45:40.269 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:39 smithi002 ceph-mon[128712]: Deploying daemon osd.9 on smithi002 2023-11-16T16:45:40.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:39 smithi155 ceph-mon[131607]: Deploying daemon osd.9 on smithi002 2023-11-16T16:45:41.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:40 smithi141 ceph-mon[131184]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:41.265 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:40 smithi002 ceph-mon[128712]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:41.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:40 smithi155 ceph-mon[131607]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:43.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:42 smithi141 ceph-mon[131184]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:43.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:42 smithi155 ceph-mon[131607]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:43.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:42 smithi002 ceph-mon[128712]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:45.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:44 smithi002 ceph-mon[128712]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:45.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:44 smithi141 ceph-mon[131184]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:45.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:44 smithi155 ceph-mon[131607]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:46.565 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:46 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:46.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:46 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:46.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:46 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:46.566 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:46 smithi002 ceph-mon[128712]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:46.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:46 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:46.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:46 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:46.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:46 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:46.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:46 smithi155 ceph-mon[131607]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:46.764 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:46 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:46.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:46 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:46.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:46 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:46.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:46 smithi141 ceph-mon[131184]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:48.884 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 9 on host 'smithi002' 2023-11-16T16:45:48.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:48.881+0000 7f8cd3fff700 1 -- 172.21.15.2:0/2279612101 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8cec061a50 con 0x7f8cd40a1150 2023-11-16T16:45:48.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:48.884+0000 7f8cf1603700 1 -- 172.21.15.2:0/2279612101 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8cd40a1150 msgr2=0x7f8cd40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:48.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:48.884+0000 7f8cf1603700 1 --2- 172.21.15.2:0/2279612101 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8cd40a1150 0x7f8cd40a3600 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8cec101350 tx=0x7f8cdc009260 comp rx=0 tx=0).stop 2023-11-16T16:45:48.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:48.884+0000 7f8cf1603700 1 -- 172.21.15.2:0/2279612101 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8cec1014f0 msgr2=0x7f8cec1987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:48.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:48.884+0000 7f8cf1603700 1 --2- 172.21.15.2:0/2279612101 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8cec1014f0 0x7f8cec1987f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0009230 tx=0x7f8ce000b790 comp rx=0 tx=0).stop 2023-11-16T16:45:48.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:48.884+0000 7f8cf1603700 1 -- 172.21.15.2:0/2279612101 shutdown_connections 2023-11-16T16:45:48.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:48.884+0000 7f8cf1603700 1 --2- 172.21.15.2:0/2279612101 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8cd40a1150 0x7f8cd40a3600 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:48.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:48.884+0000 7f8cf1603700 1 --2- 172.21.15.2:0/2279612101 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cec1002f0 0x7f8cec1982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:48.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:48.884+0000 7f8cf1603700 1 --2- 172.21.15.2:0/2279612101 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8cec101e80 0x7f8cec19f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:48.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:48.884+0000 7f8cf1603700 1 --2- 172.21.15.2:0/2279612101 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8cec1014f0 0x7f8cec1987f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:48.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:48.884+0000 7f8cf1603700 1 -- 172.21.15.2:0/2279612101 >> 172.21.15.2:0/2279612101 conn(0x7f8cec0fb900 msgr2=0x7f8cec106e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:45:48.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:48.884+0000 7f8cf1603700 1 -- 172.21.15.2:0/2279612101 shutdown_connections 2023-11-16T16:45:48.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:45:48.884+0000 7f8cf1603700 1 -- 172.21.15.2:0/2279612101 wait complete. 2023-11-16T16:45:49.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:48 smithi155 ceph-mon[131607]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:49.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:48 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:49.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:48 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:49.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:48 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:49.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:48 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:49.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:48 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:49.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:48 smithi002 ceph-mon[128712]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:49.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:48 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:49.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:48 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:49.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:48 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:49.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:48 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:49.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:48 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:49.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:48 smithi141 ceph-mon[131184]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:49.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:48 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:49.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:48 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:49.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:48 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:49.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:48 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:49.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:48 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:49.417 DEBUG:teuthology.orchestra.run.smithi002:osd.9> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.9.service 2023-11-16T16:45:49.422 INFO:tasks.cephadm:Deploying osd.10 on smithi141 with /dev/vg_nvme/lv_1... 2023-11-16T16:45:49.422 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-11-16T16:45:49.494 INFO:journalctl@ceph.osd.9.smithi002.stdout:-- Logs begin at Thu 2023-11-16 16:19:53 UTC. -- 2023-11-16T16:45:50.073 INFO:teuthology.orchestra.run.smithi141.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T16:45:50.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:49 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:45:50.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:49 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:45:50.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:49 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:45:50.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:49 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:50.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:49 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:50.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:49 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:50.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:49 smithi155 ceph-mon[131607]: from='osd.9 [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-11-16T16:45:50.143 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:49 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:45:50.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:49 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:45:50.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:49 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:45:50.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:49 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:50.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:49 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:50.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:49 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:50.144 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:49 smithi002 ceph-mon[128712]: from='osd.9 [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-11-16T16:45:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:49 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:45:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:49 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:45:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:49 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:45:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:49 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:49 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:49 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:45:50.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:49 smithi141 ceph-mon[131184]: from='osd.9 [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-11-16T16:45:51.121 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:50 smithi002 ceph-mon[128712]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:51.121 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:50 smithi002 ceph-mon[128712]: from='osd.9 [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-11-16T16:45:51.121 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:50 smithi002 ceph-mon[128712]: osdmap e65: 10 total, 9 up, 10 in 2023-11-16T16:45:51.121 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:51.121 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:50 smithi002 ceph-mon[128712]: from='osd.9 [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T16:45:51.122 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 16:45:50 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T16:45:50.978+0000 7f770c66e700 -1 osd.9 0 waiting for initial osdmap 2023-11-16T16:45:51.122 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 16:45:51 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T16:45:50.998+0000 7f7707c62700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:45:51.124 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:50 smithi141 ceph-mon[131184]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:51.124 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:50 smithi141 ceph-mon[131184]: from='osd.9 [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-11-16T16:45:51.124 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:50 smithi141 ceph-mon[131184]: osdmap e65: 10 total, 9 up, 10 in 2023-11-16T16:45:51.124 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:51.124 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:50 smithi141 ceph-mon[131184]: from='osd.9 [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T16:45:51.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:50 smithi155 ceph-mon[131607]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:51.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:50 smithi155 ceph-mon[131607]: from='osd.9 [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-11-16T16:45:51.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:50 smithi155 ceph-mon[131607]: osdmap e65: 10 total, 9 up, 10 in 2023-11-16T16:45:51.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:51.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:50 smithi155 ceph-mon[131607]: from='osd.9 [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2023-11-16T16:45:51.580 INFO:teuthology.orchestra.run.smithi141.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-11-16T16:45:51.581 INFO:teuthology.orchestra.run.smithi141.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-11-16T16:45:51.581 INFO:teuthology.orchestra.run.smithi141.stderr: stderr: 10+0 records in 2023-11-16T16:45:51.581 INFO:teuthology.orchestra.run.smithi141.stderr:10+0 records out 2023-11-16T16:45:51.581 INFO:teuthology.orchestra.run.smithi141.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111334 s, 942 MB/s 2023-11-16T16:45:51.581 INFO:teuthology.orchestra.run.smithi141.stderr:--> Zapping successful for: 2023-11-16T16:45:52.029 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi141:vg_nvme/lv_1 2023-11-16T16:45:52.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:51 smithi002 ceph-mon[128712]: from='osd.9 [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T16:45:52.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:51 smithi002 ceph-mon[128712]: osdmap e66: 10 total, 9 up, 10 in 2023-11-16T16:45:52.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:51 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:52.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:51 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:52.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:51 smithi141 ceph-mon[131184]: from='osd.9 [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T16:45:52.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:51 smithi141 ceph-mon[131184]: osdmap e66: 10 total, 9 up, 10 in 2023-11-16T16:45:52.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:51 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:52.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:51 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:52.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:51 smithi155 ceph-mon[131607]: from='osd.9 [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2023-11-16T16:45:52.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:51 smithi155 ceph-mon[131607]: osdmap e66: 10 total, 9 up, 10 in 2023-11-16T16:45:52.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:51 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:52.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:51 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:52.674 INFO:teuthology.orchestra.run.smithi141.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T16:45:53.241 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:52 smithi141 ceph-mon[131184]: purged_snaps scrub starts 2023-11-16T16:45:53.242 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:52 smithi141 ceph-mon[131184]: purged_snaps scrub ok 2023-11-16T16:45:53.242 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:52 smithi141 ceph-mon[131184]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:53.242 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:53.242 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:52 smithi141 ceph-mon[131184]: osd.9 [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823] boot 2023-11-16T16:45:53.242 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:52 smithi141 ceph-mon[131184]: osdmap e67: 10 total, 10 up, 10 in 2023-11-16T16:45:53.242 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:53.242 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:53.242 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:53.243 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:53.243 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:53.243 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:53.243 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:53.243 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:53.243 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:53.243 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:53.243 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:52 smithi155 ceph-mon[131607]: purged_snaps scrub starts 2023-11-16T16:45:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:52 smithi155 ceph-mon[131607]: purged_snaps scrub ok 2023-11-16T16:45:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:52 smithi155 ceph-mon[131607]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:52 smithi155 ceph-mon[131607]: osd.9 [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823] boot 2023-11-16T16:45:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:52 smithi155 ceph-mon[131607]: osdmap e67: 10 total, 10 up, 10 in 2023-11-16T16:45:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:53.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:53.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:53.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:53.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:53.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:53.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:53.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:53.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:53.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:52 smithi002 ceph-mon[128712]: purged_snaps scrub starts 2023-11-16T16:45:53.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:52 smithi002 ceph-mon[128712]: purged_snaps scrub ok 2023-11-16T16:45:53.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:52 smithi002 ceph-mon[128712]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-11-16T16:45:53.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:53.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:52 smithi002 ceph-mon[128712]: osd.9 [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823] boot 2023-11-16T16:45:53.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:52 smithi002 ceph-mon[128712]: osdmap e67: 10 total, 10 up, 10 in 2023-11-16T16:45:53.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:45:53.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:53.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:53.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:53.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:53.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:53.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:45:53.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:53.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:53.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:45:53.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:45:54.006 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.002+0000 7f0f11756700 1 -- 172.21.15.141:0/156679114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0f0c0ff160 msgr2=0x7f0f0c0ff570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:54.006 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.002+0000 7f0f11756700 1 --2- 172.21.15.141:0/156679114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0f0c0ff160 0x7f0f0c0ff570 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0f00009990 tx=0x7f0f00009ca0 comp rx=0 tx=0).stop 2023-11-16T16:45:54.006 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.002+0000 7f0f11756700 1 -- 172.21.15.141:0/156679114 shutdown_connections 2023-11-16T16:45:54.006 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.002+0000 7f0f11756700 1 --2- 172.21.15.141:0/156679114 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f0c100cf0 0x7f0f0c1075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:54.007 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.002+0000 7f0f11756700 1 --2- 172.21.15.141:0/156679114 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0f0c0ff160 0x7f0f0c0ff570 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:54.007 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.002+0000 7f0f11756700 1 --2- 172.21.15.141:0/156679114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f0c100360 0x7f0f0c1007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:54.007 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.002+0000 7f0f11756700 1 -- 172.21.15.141:0/156679114 >> 172.21.15.141:0/156679114 conn(0x7f0f0c0fa790 msgr2=0x7f0f0c0fcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:45:54.007 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.002+0000 7f0f11756700 1 -- 172.21.15.141:0/156679114 shutdown_connections 2023-11-16T16:45:54.007 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.002+0000 7f0f11756700 1 -- 172.21.15.141:0/156679114 wait complete. 2023-11-16T16:45:54.007 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.002+0000 7f0f11756700 1 Processor -- start 2023-11-16T16:45:54.007 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.003+0000 7f0f11756700 1 -- start start 2023-11-16T16:45:54.007 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.003+0000 7f0f11756700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f0c0ff160 0x7f0f0c195fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:54.007 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.003+0000 7f0f11756700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f0c100360 0x7f0f0c1964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:54.007 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.003+0000 7f0f11756700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0f0c100cf0 0x7f0f0c19d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:54.007 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.003+0000 7f0f11756700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c19db30 con 0x7f0f0c0ff160 2023-11-16T16:45:54.007 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.003+0000 7f0f11756700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c19dc70 con 0x7f0f0c100cf0 2023-11-16T16:45:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.003+0000 7f0f11756700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c19de00 con 0x7f0f0c100360 2023-11-16T16:45:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.003+0000 7f0f0b7fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0f0c100cf0 0x7f0f0c19d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.003+0000 7f0f0b7fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0f0c100cf0 0x7f0f0c19d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.141:60014/0 (socket says 172.21.15.141:60014) 2023-11-16T16:45:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.003+0000 7f0f0b7fe700 1 -- 172.21.15.141:0/3944342988 learned_addr learned my addr 172.21.15.141:0/3944342988 (peer_addr_for_me v2:172.21.15.141:0/0) 2023-11-16T16:45:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.003+0000 7f0f0a7fc700 1 --2- 172.21.15.141:0/3944342988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f0c100360 0x7f0f0c1964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.003+0000 7f0f0b7fe700 1 -- 172.21.15.141:0/3944342988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f0c100360 msgr2=0x7f0f0c1964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.003+0000 7f0f0b7fe700 1 --2- 172.21.15.141:0/3944342988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f0c100360 0x7f0f0c1964e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.003+0000 7f0f0b7fe700 1 -- 172.21.15.141:0/3944342988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f0c0ff160 msgr2=0x7f0f0c195fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:45:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.003+0000 7f0f0b7fe700 1 --2- 172.21.15.141:0/3944342988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f0c0ff160 0x7f0f0c195fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:45:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.003+0000 7f0f0b7fe700 1 -- 172.21.15.141:0/3944342988 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f000095f0 con 0x7f0f0c100cf0 2023-11-16T16:45:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.003+0000 7f0f0b7fe700 1 --2- 172.21.15.141:0/3944342988 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0f0c100cf0 0x7f0f0c19d560 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0f04007ee0 tx=0x7f0f0400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:45:54.008 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.004+0000 7f0ef3fff700 1 -- 172.21.15.141:0/3944342988 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f04017070 con 0x7f0f0c100cf0 2023-11-16T16:45:54.009 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.004+0000 7f0ef3fff700 1 -- 172.21.15.141:0/3944342988 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f0400f040 con 0x7f0f0c100cf0 2023-11-16T16:45:54.009 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.004+0000 7f0f11756700 1 -- 172.21.15.141:0/3944342988 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f0c10aae0 con 0x7f0f0c100cf0 2023-11-16T16:45:54.009 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.004+0000 7f0ef3fff700 1 -- 172.21.15.141:0/3944342988 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f04005740 con 0x7f0f0c100cf0 2023-11-16T16:45:54.009 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.004+0000 7f0f11756700 1 -- 172.21.15.141:0/3944342988 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f0c10a950 con 0x7f0f0c100cf0 2023-11-16T16:45:54.009 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.005+0000 7f0f11756700 1 -- 172.21.15.141:0/3944342988 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f0c04e930 con 0x7f0f0c100cf0 2023-11-16T16:45:54.010 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.005+0000 7f0ef3fff700 1 -- 172.21.15.141:0/3944342988 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f0f0400dc60 con 0x7f0f0c100cf0 2023-11-16T16:45:54.010 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.006+0000 7f0ef3fff700 1 --2- 172.21.15.141:0/3944342988 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0ef40a1150 0x7f0ef40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:45:54.010 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.006+0000 7f0ef3fff700 1 -- 172.21.15.141:0/3944342988 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f0f040cb8b0 con 0x7f0f0c100cf0 2023-11-16T16:45:54.010 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.006+0000 7f0f0affd700 1 --2- 172.21.15.141:0/3944342988 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0ef40a1150 0x7f0ef40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:45:54.010 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.007+0000 7f0f0affd700 1 --2- 172.21.15.141:0/3944342988 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0ef40a1150 0x7f0ef40a3600 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0f000095c0 tx=0x7f0f00009530 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:45:54.012 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.010+0000 7f0ef3fff700 1 -- 172.21.15.141:0/3944342988 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0f04095800 con 0x7f0f0c100cf0 2023-11-16T16:45:54.207 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:45:54.205+0000 7f0f11756700 1 -- 172.21.15.141:0/3944342988 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi141:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f0c0695b0 con 0x7f0ef40a1150 2023-11-16T16:45:54.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:53 smithi141 ceph-mon[131184]: Detected new or changed devices on smithi002 2023-11-16T16:45:54.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:53 smithi141 ceph-mon[131184]: Adjusting osd_memory_target on smithi002 to 4319M 2023-11-16T16:45:54.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:53 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:45:54.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:53 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:45:54.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:53 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:45:54.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:53 smithi141 ceph-mon[131184]: osdmap e68: 10 total, 10 up, 10 in 2023-11-16T16:45:54.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:53 smithi155 ceph-mon[131607]: Detected new or changed devices on smithi002 2023-11-16T16:45:54.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:53 smithi155 ceph-mon[131607]: Adjusting osd_memory_target on smithi002 to 4319M 2023-11-16T16:45:54.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:53 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:45:54.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:53 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:45:54.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:53 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:45:54.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:53 smithi155 ceph-mon[131607]: osdmap e68: 10 total, 10 up, 10 in 2023-11-16T16:45:54.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:53 smithi002 ceph-mon[128712]: Detected new or changed devices on smithi002 2023-11-16T16:45:54.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:53 smithi002 ceph-mon[128712]: Adjusting osd_memory_target on smithi002 to 4319M 2023-11-16T16:45:54.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:53 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:45:54.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:53 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:45:54.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:53 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:45:54.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:53 smithi002 ceph-mon[128712]: osdmap e68: 10 total, 10 up, 10 in 2023-11-16T16:45:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:54 smithi141 ceph-mon[131184]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:45:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:45:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:55.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:54 smithi155 ceph-mon[131607]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:55.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:45:55.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:45:55.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:55.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:54 smithi002 ceph-mon[128712]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:55.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:45:55.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:45:55.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:45:56.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:56 smithi141 ceph-mon[131184]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi141:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:45:56.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:56 smithi155 ceph-mon[131607]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi141:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:45:56.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:55 smithi002 ceph-mon[128712]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi141:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:45:57.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:57 smithi141 ceph-mon[131184]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:57.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:57 smithi141 ceph-mon[131184]: from='client.? 172.21.15.141:0/453135689' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c9f53c3-c1bb-45ce-a05e-94d49b0c654f"}]: dispatch 2023-11-16T16:45:57.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:57 smithi141 ceph-mon[131184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c9f53c3-c1bb-45ce-a05e-94d49b0c654f"}]: dispatch 2023-11-16T16:45:57.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:57 smithi141 ceph-mon[131184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c9f53c3-c1bb-45ce-a05e-94d49b0c654f"}]': finished 2023-11-16T16:45:57.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:57 smithi141 ceph-mon[131184]: osdmap e69: 11 total, 10 up, 11 in 2023-11-16T16:45:57.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:57 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:45:57.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:57 smithi155 ceph-mon[131607]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:57.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:57 smithi155 ceph-mon[131607]: from='client.? 172.21.15.141:0/453135689' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c9f53c3-c1bb-45ce-a05e-94d49b0c654f"}]: dispatch 2023-11-16T16:45:57.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:57 smithi155 ceph-mon[131607]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c9f53c3-c1bb-45ce-a05e-94d49b0c654f"}]: dispatch 2023-11-16T16:45:57.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:57 smithi155 ceph-mon[131607]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c9f53c3-c1bb-45ce-a05e-94d49b0c654f"}]': finished 2023-11-16T16:45:57.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:57 smithi155 ceph-mon[131607]: osdmap e69: 11 total, 10 up, 11 in 2023-11-16T16:45:57.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:57 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:45:57.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:57 smithi002 ceph-mon[128712]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:57.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:57 smithi002 ceph-mon[128712]: from='client.? 172.21.15.141:0/453135689' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c9f53c3-c1bb-45ce-a05e-94d49b0c654f"}]: dispatch 2023-11-16T16:45:57.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:57 smithi002 ceph-mon[128712]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c9f53c3-c1bb-45ce-a05e-94d49b0c654f"}]: dispatch 2023-11-16T16:45:57.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:57 smithi002 ceph-mon[128712]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c9f53c3-c1bb-45ce-a05e-94d49b0c654f"}]': finished 2023-11-16T16:45:57.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:57 smithi002 ceph-mon[128712]: osdmap e69: 11 total, 10 up, 11 in 2023-11-16T16:45:57.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:57 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:45:58.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:58 smithi141 ceph-mon[131184]: from='client.? 172.21.15.141:0/1115921267' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:45:58.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:58 smithi155 ceph-mon[131607]: from='client.? 172.21.15.141:0/1115921267' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:45:58.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:58 smithi002 ceph-mon[128712]: from='client.? 172.21.15.141:0/1115921267' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:45:59.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:45:59 smithi141 ceph-mon[131184]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:59.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:45:59 smithi155 ceph-mon[131607]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:45:59.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:45:59 smithi002 ceph-mon[128712]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:01.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:01 smithi141 ceph-mon[131184]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:01.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:01 smithi155 ceph-mon[131607]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:01.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:01 smithi002 ceph-mon[128712]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:03.246 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:03 smithi141 ceph-mon[131184]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:03.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:03 smithi155 ceph-mon[131607]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:03.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:03 smithi002 ceph-mon[128712]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:04.339 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:04 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-11-16T16:46:04.339 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:04 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:04.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:04 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-11-16T16:46:04.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:04 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:04.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:04 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-11-16T16:46:04.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:04 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:05.237 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:05 smithi141 ceph-mon[131184]: Deploying daemon osd.10 on smithi141 2023-11-16T16:46:05.238 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:05 smithi141 ceph-mon[131184]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:05.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:05 smithi155 ceph-mon[131607]: Deploying daemon osd.10 on smithi141 2023-11-16T16:46:05.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:05 smithi155 ceph-mon[131607]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:05.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:05 smithi002 ceph-mon[128712]: Deploying daemon osd.10 on smithi141 2023-11-16T16:46:05.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:05 smithi002 ceph-mon[128712]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:07.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:07 smithi141 ceph-mon[131184]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:07.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:07 smithi155 ceph-mon[131607]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:07.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:07 smithi002 ceph-mon[128712]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:09.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:08 smithi002 ceph-mon[128712]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:09.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:08 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:09.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:08 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:09.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:08 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:09.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:08 smithi141 ceph-mon[131184]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:09.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:08 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:09.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:08 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:09.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:08 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:09.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:08 smithi155 ceph-mon[131607]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:09.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:08 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:09.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:08 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:09.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:08 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:10.973 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:46:10.972+0000 7f0ef3fff700 1 -- 172.21.15.141:0/3944342988 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f0f0c0695b0 con 0x7f0ef40a1150 2023-11-16T16:46:10.973 INFO:teuthology.orchestra.run.smithi141.stdout:Created osd(s) 10 on host 'smithi141' 2023-11-16T16:46:10.976 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:46:10.974+0000 7f0f11756700 1 -- 172.21.15.141:0/3944342988 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0ef40a1150 msgr2=0x7f0ef40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:10.976 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:46:10.974+0000 7f0f11756700 1 --2- 172.21.15.141:0/3944342988 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0ef40a1150 0x7f0ef40a3600 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0f000095c0 tx=0x7f0f00009530 comp rx=0 tx=0).stop 2023-11-16T16:46:10.976 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:46:10.974+0000 7f0f11756700 1 -- 172.21.15.141:0/3944342988 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0f0c100cf0 msgr2=0x7f0f0c19d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:10.976 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:46:10.974+0000 7f0f11756700 1 --2- 172.21.15.141:0/3944342988 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0f0c100cf0 0x7f0f0c19d560 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0f04007ee0 tx=0x7f0f0400d3b0 comp rx=0 tx=0).stop 2023-11-16T16:46:10.976 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:46:10.975+0000 7f0f11756700 1 -- 172.21.15.141:0/3944342988 shutdown_connections 2023-11-16T16:46:10.976 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:46:10.975+0000 7f0f11756700 1 --2- 172.21.15.141:0/3944342988 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0ef40a1150 0x7f0ef40a3600 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:10.976 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:46:10.975+0000 7f0f11756700 1 --2- 172.21.15.141:0/3944342988 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0f0c100cf0 0x7f0f0c19d560 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:10.976 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:46:10.975+0000 7f0f11756700 1 --2- 172.21.15.141:0/3944342988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f0c0ff160 0x7f0f0c195fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:10.976 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:46:10.975+0000 7f0f11756700 1 --2- 172.21.15.141:0/3944342988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f0c100360 0x7f0f0c1964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:10.976 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:46:10.975+0000 7f0f11756700 1 -- 172.21.15.141:0/3944342988 >> 172.21.15.141:0/3944342988 conn(0x7f0f0c0fa790 msgr2=0x7f0f0c068ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:10.976 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:46:10.975+0000 7f0f11756700 1 -- 172.21.15.141:0/3944342988 shutdown_connections 2023-11-16T16:46:10.976 INFO:teuthology.orchestra.run.smithi141.stderr:2023-11-16T16:46:10.975+0000 7f0f11756700 1 -- 172.21.15.141:0/3944342988 wait complete. 2023-11-16T16:46:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:10 smithi141 ceph-mon[131184]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:10 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:10 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:10 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:10 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:10 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:10 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:10 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:10 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:11.364 DEBUG:teuthology.orchestra.run.smithi141:osd.10> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.10.service 2023-11-16T16:46:11.366 INFO:tasks.cephadm:Deploying osd.11 on smithi155 with /dev/vg_nvme/lv_1... 2023-11-16T16:46:11.366 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-11-16T16:46:11.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:10 smithi155 ceph-mon[131607]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:11.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:10 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:11.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:10 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:11.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:10 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:11.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:10 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:11.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:10 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:11.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:10 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:11.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:10 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:11.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:10 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:11.395 INFO:journalctl@ceph.osd.10.smithi141.stdout:-- Logs begin at Thu 2023-11-16 16:14:35 UTC. -- 2023-11-16T16:46:11.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:10 smithi002 ceph-mon[128712]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:11.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:11.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:11.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:11.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:11.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:11.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:11.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:11.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:10 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:12.017 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:46:12.066 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:11 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:46:12.066 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:11 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:46:12.066 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:11 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:46:12.066 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:11 smithi141 ceph-mon[131184]: from='osd.10 [v2:172.21.15.141:6824/1084001742,v1:172.21.15.141:6825/1084001742]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T16:46:12.066 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:11 smithi141 ceph-mon[131184]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T16:46:12.066 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 16:46:11 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T16:46:11.811+0000 7fa82895b640 -1 osd.10 0 log_to_monitors true 2023-11-16T16:46:12.245 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:11 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:46:12.245 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:11 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:46:12.245 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:11 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:46:12.245 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:11 smithi155 ceph-mon[131607]: from='osd.10 [v2:172.21.15.141:6824/1084001742,v1:172.21.15.141:6825/1084001742]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T16:46:12.245 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:11 smithi155 ceph-mon[131607]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T16:46:12.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:11 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:46:12.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:11 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:46:12.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:11 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:46:12.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:11 smithi002 ceph-mon[128712]: from='osd.10 [v2:172.21.15.141:6824/1084001742,v1:172.21.15.141:6825/1084001742]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T16:46:12.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:11 smithi002 ceph-mon[128712]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-11-16T16:46:12.986 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:12 smithi141 ceph-mon[131184]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:12.986 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:12 smithi141 ceph-mon[131184]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-11-16T16:46:12.986 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:12 smithi141 ceph-mon[131184]: osdmap e70: 11 total, 10 up, 11 in 2023-11-16T16:46:12.986 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:12 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:12.986 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:12 smithi141 ceph-mon[131184]: from='osd.10 [v2:172.21.15.141:6824/1084001742,v1:172.21.15.141:6825/1084001742]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:46:13.249 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:12 smithi141 ceph-mon[131184]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:46:13.249 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 16:46:12 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T16:46:12.988+0000 7fa81dfd2700 -1 osd.10 0 waiting for initial osdmap 2023-11-16T16:46:13.249 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 16:46:13 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T16:46:13.000+0000 7fa815dbf700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:46:13.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:12 smithi155 ceph-mon[131607]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:13.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:12 smithi155 ceph-mon[131607]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-11-16T16:46:13.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:12 smithi155 ceph-mon[131607]: osdmap e70: 11 total, 10 up, 11 in 2023-11-16T16:46:13.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:12 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:13.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:12 smithi155 ceph-mon[131607]: from='osd.10 [v2:172.21.15.141:6824/1084001742,v1:172.21.15.141:6825/1084001742]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:46:13.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:12 smithi155 ceph-mon[131607]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:46:13.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:12 smithi002 ceph-mon[128712]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:13.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:12 smithi002 ceph-mon[128712]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-11-16T16:46:13.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:12 smithi002 ceph-mon[128712]: osdmap e70: 11 total, 10 up, 11 in 2023-11-16T16:46:13.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:12 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:13.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:12 smithi002 ceph-mon[128712]: from='osd.10 [v2:172.21.15.141:6824/1084001742,v1:172.21.15.141:6825/1084001742]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:46:13.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:12 smithi002 ceph-mon[128712]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]: dispatch 2023-11-16T16:46:13.519 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-11-16T16:46:13.519 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-11-16T16:46:13.520 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2023-11-16T16:46:13.520 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2023-11-16T16:46:13.520 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01107 s, 947 MB/s 2023-11-16T16:46:13.520 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2023-11-16T16:46:13.958 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch daemon add osd smithi155:vg_nvme/lv_1 2023-11-16T16:46:14.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:13 smithi141 ceph-mon[131184]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]': finished 2023-11-16T16:46:14.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:13 smithi141 ceph-mon[131184]: osdmap e71: 11 total, 10 up, 11 in 2023-11-16T16:46:14.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:14.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:14.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:14.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:14.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:14.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:14.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:14.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:14.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:14.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:14.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:14.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:14.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:13 smithi141 ceph-mon[131184]: osd.10 [v2:172.21.15.141:6824/1084001742,v1:172.21.15.141:6825/1084001742] boot 2023-11-16T16:46:14.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:13 smithi141 ceph-mon[131184]: osdmap e72: 11 total, 11 up, 11 in 2023-11-16T16:46:14.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:13 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:14.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:13 smithi155 ceph-mon[131607]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]': finished 2023-11-16T16:46:14.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:13 smithi155 ceph-mon[131607]: osdmap e71: 11 total, 10 up, 11 in 2023-11-16T16:46:14.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:14.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:14.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:14.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:14.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:14.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:14.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:14.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:14.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:14.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:14.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:14.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:14.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:13 smithi155 ceph-mon[131607]: osd.10 [v2:172.21.15.141:6824/1084001742,v1:172.21.15.141:6825/1084001742] boot 2023-11-16T16:46:14.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:13 smithi155 ceph-mon[131607]: osdmap e72: 11 total, 11 up, 11 in 2023-11-16T16:46:14.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:13 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:14.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:13 smithi002 ceph-mon[128712]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi141", "root=default"]}]': finished 2023-11-16T16:46:14.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:13 smithi002 ceph-mon[128712]: osdmap e71: 11 total, 10 up, 11 in 2023-11-16T16:46:14.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:13 smithi002 ceph-mon[128712]: osd.10 [v2:172.21.15.141:6824/1084001742,v1:172.21.15.141:6825/1084001742] boot 2023-11-16T16:46:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:13 smithi002 ceph-mon[128712]: osdmap e72: 11 total, 11 up, 11 in 2023-11-16T16:46:14.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:13 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:46:14.601 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:46:15.081 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:14 smithi155 ceph-mon[131607]: purged_snaps scrub starts 2023-11-16T16:46:15.081 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:14 smithi155 ceph-mon[131607]: purged_snaps scrub ok 2023-11-16T16:46:15.081 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:14 smithi155 ceph-mon[131607]: Detected new or changed devices on smithi141 2023-11-16T16:46:15.081 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:14 smithi155 ceph-mon[131607]: Adjusting osd_memory_target on smithi141 to 4292M 2023-11-16T16:46:15.081 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:14 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:46:15.081 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:14 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:46:15.081 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:14 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:46:15.081 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:14 smithi155 ceph-mon[131607]: pgmap v226: 1 pgs: 1 peering; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:15.081 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:14 smithi155 ceph-mon[131607]: osdmap e73: 11 total, 11 up, 11 in 2023-11-16T16:46:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:14 smithi141 ceph-mon[131184]: purged_snaps scrub starts 2023-11-16T16:46:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:14 smithi141 ceph-mon[131184]: purged_snaps scrub ok 2023-11-16T16:46:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:14 smithi141 ceph-mon[131184]: Detected new or changed devices on smithi141 2023-11-16T16:46:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:14 smithi141 ceph-mon[131184]: Adjusting osd_memory_target on smithi141 to 4292M 2023-11-16T16:46:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:14 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:46:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:14 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:46:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:14 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:46:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:14 smithi141 ceph-mon[131184]: pgmap v226: 1 pgs: 1 peering; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:14 smithi141 ceph-mon[131184]: osdmap e73: 11 total, 11 up, 11 in 2023-11-16T16:46:15.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:14 smithi002 ceph-mon[128712]: purged_snaps scrub starts 2023-11-16T16:46:15.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:14 smithi002 ceph-mon[128712]: purged_snaps scrub ok 2023-11-16T16:46:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:14 smithi002 ceph-mon[128712]: Detected new or changed devices on smithi141 2023-11-16T16:46:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:14 smithi002 ceph-mon[128712]: Adjusting osd_memory_target on smithi141 to 4292M 2023-11-16T16:46:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:14 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:46:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:14 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:46:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:14 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:46:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:14 smithi002 ceph-mon[128712]: pgmap v226: 1 pgs: 1 peering; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-11-16T16:46:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:14 smithi002 ceph-mon[128712]: osdmap e73: 11 total, 11 up, 11 in 2023-11-16T16:46:15.805 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.804+0000 7f7194b32700 1 -- 172.21.15.155:0/4258563566 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71900ff420 msgr2=0x7f71900ff830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:15.805 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.804+0000 7f7194b32700 1 --2- 172.21.15.155:0/4258563566 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71900ff420 0x7f71900ff830 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7180009990 tx=0x7f7180009ca0 comp rx=0 tx=0).stop 2023-11-16T16:46:15.806 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.804+0000 7f7194b32700 1 -- 172.21.15.155:0/4258563566 shutdown_connections 2023-11-16T16:46:15.806 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.804+0000 7f7194b32700 1 --2- 172.21.15.155:0/4258563566 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71900ff420 0x7f71900ff830 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:15.806 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.804+0000 7f7194b32700 1 --2- 172.21.15.155:0/4258563566 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7190100fb0 0x7f7190107870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:15.806 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.804+0000 7f7194b32700 1 --2- 172.21.15.155:0/4258563566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7190100620 0x7f7190100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:15.806 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.804+0000 7f7194b32700 1 -- 172.21.15.155:0/4258563566 >> 172.21.15.155:0/4258563566 conn(0x7f71900faa50 msgr2=0x7f71900fcea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:15.806 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.804+0000 7f7194b32700 1 -- 172.21.15.155:0/4258563566 shutdown_connections 2023-11-16T16:46:15.806 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.804+0000 7f7194b32700 1 -- 172.21.15.155:0/4258563566 wait complete. 2023-11-16T16:46:15.808 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f7194b32700 1 Processor -- start 2023-11-16T16:46:15.808 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f7194b32700 1 -- start start 2023-11-16T16:46:15.808 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f7194b32700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71900ff420 0x7f7190069be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:15.808 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f7194b32700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7190100620 0x7f719019b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:15.808 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f7194b32700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7190100fb0 0x7f719019d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:15.808 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f7194b32700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719019de10 con 0x7f71900ff420 2023-11-16T16:46:15.809 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f7194b32700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719019df80 con 0x7f7190100fb0 2023-11-16T16:46:15.809 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f7194b32700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719019e140 con 0x7f7190100620 2023-11-16T16:46:15.809 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f718dd9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7190100620 0x7f719019b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:15.809 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f718dd9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7190100620 0x7f719019b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:39364/0 (socket says 172.21.15.155:39364) 2023-11-16T16:46:15.809 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f718dd9b700 1 -- 172.21.15.155:0/758144115 learned_addr learned my addr 172.21.15.155:0/758144115 (peer_addr_for_me v2:172.21.15.155:0/0) 2023-11-16T16:46:15.810 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f718e59c700 1 --2- 172.21.15.155:0/758144115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71900ff420 0x7f7190069be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:15.810 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f718ed9d700 1 --2- 172.21.15.155:0/758144115 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7190100fb0 0x7f719019d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:15.810 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f718dd9b700 1 -- 172.21.15.155:0/758144115 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7190100fb0 msgr2=0x7f719019d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:15.810 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f718dd9b700 1 --2- 172.21.15.155:0/758144115 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7190100fb0 0x7f719019d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:15.810 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f718dd9b700 1 -- 172.21.15.155:0/758144115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71900ff420 msgr2=0x7f7190069be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:15.810 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f718dd9b700 1 --2- 172.21.15.155:0/758144115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71900ff420 0x7f7190069be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:15.810 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f718dd9b700 1 -- 172.21.15.155:0/758144115 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71800095f0 con 0x7f7190100620 2023-11-16T16:46:15.810 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f718e59c700 1 --2- 172.21.15.155:0/758144115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71900ff420 0x7f7190069be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:46:15.810 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.805+0000 7f718dd9b700 1 --2- 172.21.15.155:0/758144115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7190100620 0x7f719019b030 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f718400b530 tx=0x7f718400b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:15.810 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.806+0000 7f717f7fe700 1 -- 172.21.15.155:0/758144115 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7184011830 con 0x7f7190100620 2023-11-16T16:46:15.810 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.806+0000 7f7194b32700 1 -- 172.21.15.155:0/758144115 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f719010adf0 con 0x7f7190100620 2023-11-16T16:46:15.810 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.806+0000 7f717f7fe700 1 -- 172.21.15.155:0/758144115 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7184011e50 con 0x7f7190100620 2023-11-16T16:46:15.811 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.806+0000 7f717f7fe700 1 -- 172.21.15.155:0/758144115 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718400f2e0 con 0x7f7190100620 2023-11-16T16:46:15.811 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.806+0000 7f7194b32700 1 -- 172.21.15.155:0/758144115 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719010ac60 con 0x7f7190100620 2023-11-16T16:46:15.811 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.807+0000 7f7194b32700 1 -- 172.21.15.155:0/758144115 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7190106a40 con 0x7f7190100620 2023-11-16T16:46:15.811 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.808+0000 7f717f7fe700 1 -- 172.21.15.155:0/758144115 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f7184010580 con 0x7f7190100620 2023-11-16T16:46:15.812 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.808+0000 7f717f7fe700 1 --2- 172.21.15.155:0/758144115 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f71780a1150 0x7f71780a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:15.812 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.808+0000 7f717f7fe700 1 -- 172.21.15.155:0/758144115 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f71840cc3a0 con 0x7f7190100620 2023-11-16T16:46:15.812 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.809+0000 7f718e59c700 1 --2- 172.21.15.155:0/758144115 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f71780a1150 0x7f71780a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:15.812 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.809+0000 7f718e59c700 1 --2- 172.21.15.155:0/758144115 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f71780a1150 0x7f71780a3600 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f718000b5c0 tx=0x7f7180005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:15.813 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:15.812+0000 7f717f7fe700 1 -- 172.21.15.155:0/758144115 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7184096120 con 0x7f7190100620 2023-11-16T16:46:16.010 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:16.009+0000 7f7194b32700 1 -- 172.21.15.155:0/758144115 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7190060f30 con 0x7f71780a1150 2023-11-16T16:46:17.221 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:17 smithi155 ceph-mon[131607]: pgmap v229: 1 pgs: 1 peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:17.221 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:17 smithi155 ceph-mon[131607]: osdmap e74: 11 total, 11 up, 11 in 2023-11-16T16:46:17.221 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:17 smithi155 ceph-mon[131607]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:46:17.222 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:17 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:46:17.222 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:17 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:46:17.222 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:17 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:17.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:17 smithi141 ceph-mon[131184]: pgmap v229: 1 pgs: 1 peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:17 smithi141 ceph-mon[131184]: osdmap e74: 11 total, 11 up, 11 in 2023-11-16T16:46:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:17 smithi141 ceph-mon[131184]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:46:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:17 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:46:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:17 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:46:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:17 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:17.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:17 smithi002 ceph-mon[128712]: pgmap v229: 1 pgs: 1 peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:17.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:17 smithi002 ceph-mon[128712]: osdmap e74: 11 total, 11 up, 11 in 2023-11-16T16:46:17.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:17 smithi002 ceph-mon[128712]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:46:17.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:17 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-16T16:46:17.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:17 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-16T16:46:17.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:17 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:18.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:18 smithi155 ceph-mon[131607]: osdmap e75: 11 total, 11 up, 11 in 2023-11-16T16:46:18.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:18 smithi141 ceph-mon[131184]: osdmap e75: 11 total, 11 up, 11 in 2023-11-16T16:46:18.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:18 smithi002 ceph-mon[128712]: osdmap e75: 11 total, 11 up, 11 in 2023-11-16T16:46:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:19 smithi141 ceph-mon[131184]: pgmap v232: 1 pgs: 1 peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:19 smithi141 ceph-mon[131184]: osdmap e76: 11 total, 11 up, 11 in 2023-11-16T16:46:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:19 smithi141 ceph-mon[131184]: from='client.? 172.21.15.155:0/1331692593' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52f27807-1649-4c8d-9182-d161fa526d54"}]: dispatch 2023-11-16T16:46:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:19 smithi141 ceph-mon[131184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52f27807-1649-4c8d-9182-d161fa526d54"}]: dispatch 2023-11-16T16:46:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:19 smithi141 ceph-mon[131184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52f27807-1649-4c8d-9182-d161fa526d54"}]': finished 2023-11-16T16:46:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:19 smithi141 ceph-mon[131184]: osdmap e77: 12 total, 11 up, 12 in 2023-11-16T16:46:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:19 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:19.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:19 smithi155 ceph-mon[131607]: pgmap v232: 1 pgs: 1 peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:19.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:19 smithi155 ceph-mon[131607]: osdmap e76: 11 total, 11 up, 11 in 2023-11-16T16:46:19.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:19 smithi155 ceph-mon[131607]: from='client.? 172.21.15.155:0/1331692593' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52f27807-1649-4c8d-9182-d161fa526d54"}]: dispatch 2023-11-16T16:46:19.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:19 smithi155 ceph-mon[131607]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52f27807-1649-4c8d-9182-d161fa526d54"}]: dispatch 2023-11-16T16:46:19.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:19 smithi155 ceph-mon[131607]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52f27807-1649-4c8d-9182-d161fa526d54"}]': finished 2023-11-16T16:46:19.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:19 smithi155 ceph-mon[131607]: osdmap e77: 12 total, 11 up, 12 in 2023-11-16T16:46:19.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:19 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:19.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:19 smithi002 ceph-mon[128712]: pgmap v232: 1 pgs: 1 peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:19.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:19 smithi002 ceph-mon[128712]: osdmap e76: 11 total, 11 up, 11 in 2023-11-16T16:46:19.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:19 smithi002 ceph-mon[128712]: from='client.? 172.21.15.155:0/1331692593' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52f27807-1649-4c8d-9182-d161fa526d54"}]: dispatch 2023-11-16T16:46:19.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:19 smithi002 ceph-mon[128712]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52f27807-1649-4c8d-9182-d161fa526d54"}]: dispatch 2023-11-16T16:46:19.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:19 smithi002 ceph-mon[128712]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52f27807-1649-4c8d-9182-d161fa526d54"}]': finished 2023-11-16T16:46:19.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:19 smithi002 ceph-mon[128712]: osdmap e77: 12 total, 11 up, 12 in 2023-11-16T16:46:19.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:19 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:20.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:20 smithi155 ceph-mon[131607]: from='client.? 172.21.15.155:0/2850330269' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:46:20.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:20 smithi002 ceph-mon[128712]: from='client.? 172.21.15.155:0/2850330269' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:46:20.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:20 smithi141 ceph-mon[131184]: from='client.? 172.21.15.155:0/2850330269' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-16T16:46:21.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:21 smithi155 ceph-mon[131607]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T16:46:21.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:21 smithi002 ceph-mon[128712]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T16:46:21.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:21 smithi141 ceph-mon[131184]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-16T16:46:23.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:23 smithi155 ceph-mon[131607]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 100 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.442 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:25 smithi155 ceph-mon[131607]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 85 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.442 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-11-16T16:46:30.442 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:26 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:30.442 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:26 smithi155 ceph-mon[131607]: Deploying daemon osd.11 on smithi155 2023-11-16T16:46:30.442 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:27 smithi155 ceph-mon[131607]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.443 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:29 smithi155 ceph-mon[131607]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.443 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:23 smithi141 ceph-mon[131184]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 100 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.443 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:25 smithi141 ceph-mon[131184]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 85 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.443 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:26 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-11-16T16:46:30.443 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:26 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:30.443 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:26 smithi141 ceph-mon[131184]: Deploying daemon osd.11 on smithi155 2023-11-16T16:46:30.443 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:27 smithi141 ceph-mon[131184]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.443 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:29 smithi141 ceph-mon[131184]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.443 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:23 smithi002 ceph-mon[128712]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 100 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.443 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:25 smithi002 ceph-mon[128712]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 85 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.443 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:26 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-11-16T16:46:30.444 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:26 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:30.444 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:26 smithi002 ceph-mon[128712]: Deploying daemon osd.11 on smithi155 2023-11-16T16:46:30.444 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:27 smithi002 ceph-mon[128712]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.444 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:29 smithi002 ceph-mon[128712]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.764 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:30 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:30.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:30 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:30.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:30 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:30.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:30 smithi141 ceph-mon[131184]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:30 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:30.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:30 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:30.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:30 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:30.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:30 smithi155 ceph-mon[131607]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-11-16T16:46:30.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:30 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:30.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:30 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:30.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:30 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:30.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:30 smithi002 ceph-mon[128712]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-11-16T16:46:32.532 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 11 on host 'smithi155' 2023-11-16T16:46:32.532 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:32.529+0000 7f717f7fe700 1 -- 172.21.15.155:0/758144115 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f7190060f30 con 0x7f71780a1150 2023-11-16T16:46:32.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:32.532+0000 7f7194b32700 1 -- 172.21.15.155:0/758144115 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f71780a1150 msgr2=0x7f71780a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:32.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:32.532+0000 7f7194b32700 1 --2- 172.21.15.155:0/758144115 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f71780a1150 0x7f71780a3600 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f718000b5c0 tx=0x7f7180005fb0 comp rx=0 tx=0).stop 2023-11-16T16:46:32.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:32.532+0000 7f7194b32700 1 -- 172.21.15.155:0/758144115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7190100620 msgr2=0x7f719019b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:32.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:32.532+0000 7f7194b32700 1 --2- 172.21.15.155:0/758144115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7190100620 0x7f719019b030 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f718400b530 tx=0x7f718400b8f0 comp rx=0 tx=0).stop 2023-11-16T16:46:32.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:32.532+0000 7f7194b32700 1 -- 172.21.15.155:0/758144115 shutdown_connections 2023-11-16T16:46:32.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:32.532+0000 7f7194b32700 1 --2- 172.21.15.155:0/758144115 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f71780a1150 0x7f71780a3600 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:32.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:32.532+0000 7f7194b32700 1 --2- 172.21.15.155:0/758144115 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7190100fb0 0x7f719019d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:32.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:32.532+0000 7f7194b32700 1 --2- 172.21.15.155:0/758144115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71900ff420 0x7f7190069be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:32.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:32.532+0000 7f7194b32700 1 --2- 172.21.15.155:0/758144115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7190100620 0x7f719019b030 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:32.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:32.532+0000 7f7194b32700 1 -- 172.21.15.155:0/758144115 >> 172.21.15.155:0/758144115 conn(0x7f71900faa50 msgr2=0x7f7190103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:32.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:32.532+0000 7f7194b32700 1 -- 172.21.15.155:0/758144115 shutdown_connections 2023-11-16T16:46:32.535 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:32.532+0000 7f7194b32700 1 -- 172.21.15.155:0/758144115 wait complete. 2023-11-16T16:46:32.883 DEBUG:teuthology.orchestra.run.smithi155:osd.11> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.11.service 2023-11-16T16:46:32.885 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-11-16T16:46:32.885 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph osd stat -f json 2023-11-16T16:46:32.914 INFO:journalctl@ceph.osd.11.smithi155.stdout:-- Logs begin at Thu 2023-11-16 16:15:48 UTC. -- 2023-11-16T16:46:33.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:33 smithi155 ceph-mon[131607]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:33.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:33 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:33.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:33 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:33.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:33 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:33.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:33 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:33.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:33 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:46:33.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:33 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:33.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:33 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:46:33.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:33 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:46:33.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:33 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:33.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:33 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:33.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:33 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:33.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:33 smithi002 ceph-mon[128712]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:33.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:33 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:33.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:33 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:33.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:33 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:33.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:33 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:33.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:33 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:46:33.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:33 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:33.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:33 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:46:33.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:33 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:46:33.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:33 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:33.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:33 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:33.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:33 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:33.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:33 smithi141 ceph-mon[131184]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:33.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:33 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:33.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:33 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:33.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:33 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:33.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:33 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:33.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:33 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:46:33.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:33 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:33.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:33 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:46:33.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:33 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:46:33.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:33 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:33.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:33 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:33.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:33 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:33.719 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 16:46:33 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T16:46:33.473+0000 7fe4b0c46640 -1 osd.11 0 log_to_monitors true 2023-11-16T16:46:33.760 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:46:34.305 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:34 smithi155 ceph-mon[131607]: from='osd.11 [v2:172.21.15.155:6824/1791652726,v1:172.21.15.155:6825/1791652726]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T16:46:34.305 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:34 smithi155 ceph-mon[131607]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T16:46:34.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:34 smithi002 ceph-mon[128712]: from='osd.11 [v2:172.21.15.155:6824/1791652726,v1:172.21.15.155:6825/1791652726]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T16:46:34.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:34 smithi002 ceph-mon[128712]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T16:46:34.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:34 smithi141 ceph-mon[131184]: from='osd.11 [v2:172.21.15.155:6824/1791652726,v1:172.21.15.155:6825/1791652726]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T16:46:34.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:34 smithi141 ceph-mon[131184]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-11-16T16:46:35.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:35 smithi155 ceph-mon[131607]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:35.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:35 smithi155 ceph-mon[131607]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-11-16T16:46:35.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:35 smithi155 ceph-mon[131607]: osdmap e78: 12 total, 11 up, 12 in 2023-11-16T16:46:35.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:35 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:35.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:35 smithi155 ceph-mon[131607]: from='osd.11 [v2:172.21.15.155:6824/1791652726,v1:172.21.15.155:6825/1791652726]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:46:35.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:35 smithi155 ceph-mon[131607]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:46:35.119 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 16:46:35 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T16:46:35.052+0000 7fe4a62bd700 -1 osd.11 0 waiting for initial osdmap 2023-11-16T16:46:35.119 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 16:46:35 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T16:46:35.069+0000 7fe49f0ac700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-16T16:46:35.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.260+0000 7fbd06227700 1 -- 172.21.15.2:0/3561436776 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd001002f0 msgr2=0x7fbd00100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:35.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.260+0000 7fbd06227700 1 --2- 172.21.15.2:0/3561436776 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd001002f0 0x7fbd00100700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbcf0009990 tx=0x7fbcf0009ca0 comp rx=0 tx=0).stop 2023-11-16T16:46:35.262 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.260+0000 7fbd06227700 1 -- 172.21.15.2:0/3561436776 shutdown_connections 2023-11-16T16:46:35.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.260+0000 7fbd06227700 1 --2- 172.21.15.2:0/3561436776 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd00101e80 0x7fbd00108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:35.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.260+0000 7fbd06227700 1 --2- 172.21.15.2:0/3561436776 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd001002f0 0x7fbd00100700 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:35.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.260+0000 7fbd06227700 1 --2- 172.21.15.2:0/3561436776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd001014f0 0x7fbd00101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:35.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.260+0000 7fbd06227700 1 -- 172.21.15.2:0/3561436776 >> 172.21.15.2:0/3561436776 conn(0x7fbd000fb900 msgr2=0x7fbd000fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:35.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.260+0000 7fbd06227700 1 -- 172.21.15.2:0/3561436776 shutdown_connections 2023-11-16T16:46:35.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.260+0000 7fbd06227700 1 -- 172.21.15.2:0/3561436776 wait complete. 2023-11-16T16:46:35.264 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.261+0000 7fbd06227700 1 Processor -- start 2023-11-16T16:46:35.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.262+0000 7fbd06227700 1 -- start start 2023-11-16T16:46:35.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.262+0000 7fbd06227700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd001002f0 0x7fbd00198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:35.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.262+0000 7fbd06227700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd001014f0 0x7fbd00198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:35.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.262+0000 7fbd06227700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd00101e80 0x7fbd0019f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:35.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.262+0000 7fbd06227700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0019fee0 con 0x7fbd001002f0 2023-11-16T16:46:35.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.262+0000 7fbd06227700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd001a0020 con 0x7fbd00101e80 2023-11-16T16:46:35.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.262+0000 7fbd06227700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd001a01b0 con 0x7fbd001014f0 2023-11-16T16:46:35.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.262+0000 7fbcff7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd001002f0 0x7fbd00198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:35.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.262+0000 7fbcff7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd001002f0 0x7fbd00198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59524/0 (socket says 172.21.15.2:59524) 2023-11-16T16:46:35.265 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.262+0000 7fbcff7fe700 1 -- 172.21.15.2:0/226526538 learned_addr learned my addr 172.21.15.2:0/226526538 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:46:35.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.262+0000 7fbcfeffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd001014f0 0x7fbd00198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:35.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.262+0000 7fbcfffff700 1 --2- 172.21.15.2:0/226526538 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd00101e80 0x7fbd0019f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:35.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.263+0000 7fbcff7fe700 1 -- 172.21.15.2:0/226526538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd001014f0 msgr2=0x7fbd00198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:35.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.263+0000 7fbcff7fe700 1 --2- 172.21.15.2:0/226526538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd001014f0 0x7fbd00198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:35.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.263+0000 7fbcff7fe700 1 -- 172.21.15.2:0/226526538 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd00101e80 msgr2=0x7fbd0019f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:35.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.263+0000 7fbcff7fe700 1 --2- 172.21.15.2:0/226526538 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd00101e80 0x7fbd0019f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:35.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.263+0000 7fbcff7fe700 1 -- 172.21.15.2:0/226526538 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf4009520 con 0x7fbd001002f0 2023-11-16T16:46:35.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.263+0000 7fbcfeffd700 1 --2- 172.21.15.2:0/226526538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd001014f0 0x7fbd00198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:46:35.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.263+0000 7fbcff7fe700 1 --2- 172.21.15.2:0/226526538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd001002f0 0x7fbd00198350 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbcf0005750 tx=0x7fbcf0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:35.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.263+0000 7fbcfcff9700 1 -- 172.21.15.2:0/226526538 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf001d070 con 0x7fbd001002f0 2023-11-16T16:46:35.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.263+0000 7fbd06227700 1 -- 172.21.15.2:0/226526538 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf8009c40 con 0x7fbd001002f0 2023-11-16T16:46:35.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.263+0000 7fbcfcff9700 1 -- 172.21.15.2:0/226526538 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbcf0021410 con 0x7fbd001002f0 2023-11-16T16:46:35.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.264+0000 7fbcfcff9700 1 -- 172.21.15.2:0/226526538 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf0017650 con 0x7fbd001002f0 2023-11-16T16:46:35.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.264+0000 7fbd06227700 1 -- 172.21.15.2:0/226526538 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf00095f0 con 0x7fbd001002f0 2023-11-16T16:46:35.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.265+0000 7fbd06227700 1 -- 172.21.15.2:0/226526538 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd00192a10 con 0x7fbd001002f0 2023-11-16T16:46:35.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.267+0000 7fbcfcff9700 1 -- 172.21.15.2:0/226526538 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fbcf0021580 con 0x7fbd001002f0 2023-11-16T16:46:35.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.269+0000 7fbcfcff9700 1 --2- 172.21.15.2:0/226526538 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbce80a1150 0x7fbce80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:35.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.269+0000 7fbcfcff9700 1 -- 172.21.15.2:0/226526538 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6847+0+0 (secure 0 0 0) 0x7fbcf00cd750 con 0x7fbd001002f0 2023-11-16T16:46:35.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.269+0000 7fbcfeffd700 1 --2- 172.21.15.2:0/226526538 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbce80a1150 0x7fbce80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:35.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.270+0000 7fbcfeffd700 1 --2- 172.21.15.2:0/226526538 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbce80a1150 0x7fbce80a3600 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009fd0 tx=0x7fbcf4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:35.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.274+0000 7fbcfcff9700 1 -- 172.21.15.2:0/226526538 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbcf0097450 con 0x7fbd001002f0 2023-11-16T16:46:35.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:35 smithi002 ceph-mon[128712]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:35.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:35 smithi002 ceph-mon[128712]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-11-16T16:46:35.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:35 smithi002 ceph-mon[128712]: osdmap e78: 12 total, 11 up, 12 in 2023-11-16T16:46:35.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:35 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:35.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:35 smithi002 ceph-mon[128712]: from='osd.11 [v2:172.21.15.155:6824/1791652726,v1:172.21.15.155:6825/1791652726]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:46:35.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:35 smithi002 ceph-mon[128712]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:46:35.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.495+0000 7fbd06227700 1 -- 172.21.15.2:0/226526538 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fbd00061700 con 0x7fbd001002f0 2023-11-16T16:46:35.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.496+0000 7fbcfcff9700 1 -- 172.21.15.2:0/226526538 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) v1 ==== 74+0+133 (secure 0 0 0) 0x7fbcf009aa70 con 0x7fbd001002f0 2023-11-16T16:46:35.498 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:46:35.500 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.498+0000 7fbd06227700 1 -- 172.21.15.2:0/226526538 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbce80a1150 msgr2=0x7fbce80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:35.500 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.498+0000 7fbd06227700 1 --2- 172.21.15.2:0/226526538 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbce80a1150 0x7fbce80a3600 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009fd0 tx=0x7fbcf4009260 comp rx=0 tx=0).stop 2023-11-16T16:46:35.500 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.498+0000 7fbd06227700 1 -- 172.21.15.2:0/226526538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd001002f0 msgr2=0x7fbd00198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:35.500 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.498+0000 7fbd06227700 1 --2- 172.21.15.2:0/226526538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd001002f0 0x7fbd00198350 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbcf0005750 tx=0x7fbcf0009580 comp rx=0 tx=0).stop 2023-11-16T16:46:35.501 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.498+0000 7fbd06227700 1 -- 172.21.15.2:0/226526538 shutdown_connections 2023-11-16T16:46:35.501 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.498+0000 7fbd06227700 1 --2- 172.21.15.2:0/226526538 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbce80a1150 0x7fbce80a3600 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:35.501 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.498+0000 7fbd06227700 1 --2- 172.21.15.2:0/226526538 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbd00101e80 0x7fbd0019f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:35.501 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.498+0000 7fbd06227700 1 --2- 172.21.15.2:0/226526538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd001002f0 0x7fbd00198350 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:35.501 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.498+0000 7fbd06227700 1 --2- 172.21.15.2:0/226526538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd001014f0 0x7fbd00198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:35.501 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.498+0000 7fbd06227700 1 -- 172.21.15.2:0/226526538 >> 172.21.15.2:0/226526538 conn(0x7fbd000fb900 msgr2=0x7fbd00106e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:35.501 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.498+0000 7fbd06227700 1 -- 172.21.15.2:0/226526538 shutdown_connections 2023-11-16T16:46:35.501 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:35.499+0000 7fbd06227700 1 -- 172.21.15.2:0/226526538 wait complete. 2023-11-16T16:46:35.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:35 smithi141 ceph-mon[131184]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:35.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:35 smithi141 ceph-mon[131184]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-11-16T16:46:35.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:35 smithi141 ceph-mon[131184]: osdmap e78: 12 total, 11 up, 12 in 2023-11-16T16:46:35.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:35 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:35.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:35 smithi141 ceph-mon[131184]: from='osd.11 [v2:172.21.15.155:6824/1791652726,v1:172.21.15.155:6825/1791652726]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:46:35.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:35 smithi141 ceph-mon[131184]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2023-11-16T16:46:36.046 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1700153173,"num_in_osds":12,"osd_in_since":1700153178,"num_remapped_pgs":0} 2023-11-16T16:46:36.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:36 smithi155 ceph-mon[131607]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2023-11-16T16:46:36.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:36 smithi155 ceph-mon[131607]: osdmap e79: 12 total, 11 up, 12 in 2023-11-16T16:46:36.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:36 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:36.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:36 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:36.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:36 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:36.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:36 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:36.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:36 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:36.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:36 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:36.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:36 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:36.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:36 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:36.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:36 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:36.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:36 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:36.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:36 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:36.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:36 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:36.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:36 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/226526538' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-16T16:46:36.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:36 smithi002 ceph-mon[128712]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2023-11-16T16:46:36.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:36 smithi002 ceph-mon[128712]: osdmap e79: 12 total, 11 up, 12 in 2023-11-16T16:46:36.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:36 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:36 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:36 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:36 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:36 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:36 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:36 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:36 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:36 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:36 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:36 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:36 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:36.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:36 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/226526538' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-16T16:46:36.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:36 smithi141 ceph-mon[131184]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2023-11-16T16:46:36.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:36 smithi141 ceph-mon[131184]: osdmap e79: 12 total, 11 up, 12 in 2023-11-16T16:46:36.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:36 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:36.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:36 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:36.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:36 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:36.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:36 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:36.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:36 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:36.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:36 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:36.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:36 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:36.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:36 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:46:36.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:36 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:36.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:36 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:36.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:36 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:36.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:36 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:36.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:36 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/226526538' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-16T16:46:37.047 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph osd stat -f json 2023-11-16T16:46:37.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:37 smithi155 ceph-mon[131607]: purged_snaps scrub starts 2023-11-16T16:46:37.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:37 smithi155 ceph-mon[131607]: purged_snaps scrub ok 2023-11-16T16:46:37.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:37 smithi155 ceph-mon[131607]: Detected new or changed devices on smithi155 2023-11-16T16:46:37.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:37 smithi155 ceph-mon[131607]: Adjusting osd_memory_target on smithi155 to 4292M 2023-11-16T16:46:37.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:37 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:46:37.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:37 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:46:37.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:37 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:46:37.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:37 smithi155 ceph-mon[131607]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:37.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:37 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:37.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:37 smithi155 ceph-mon[131607]: osd.11 [v2:172.21.15.155:6824/1791652726,v1:172.21.15.155:6825/1791652726] boot 2023-11-16T16:46:37.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:37 smithi155 ceph-mon[131607]: osdmap e80: 12 total, 12 up, 12 in 2023-11-16T16:46:37.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:37 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:37.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:37 smithi002 ceph-mon[128712]: purged_snaps scrub starts 2023-11-16T16:46:37.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:37 smithi002 ceph-mon[128712]: purged_snaps scrub ok 2023-11-16T16:46:37.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:37 smithi002 ceph-mon[128712]: Detected new or changed devices on smithi155 2023-11-16T16:46:37.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:37 smithi002 ceph-mon[128712]: Adjusting osd_memory_target on smithi155 to 4292M 2023-11-16T16:46:37.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:37 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:46:37.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:37 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:46:37.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:37 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:46:37.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:37 smithi002 ceph-mon[128712]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:37.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:37 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:37.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:37 smithi002 ceph-mon[128712]: osd.11 [v2:172.21.15.155:6824/1791652726,v1:172.21.15.155:6825/1791652726] boot 2023-11-16T16:46:37.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:37 smithi002 ceph-mon[128712]: osdmap e80: 12 total, 12 up, 12 in 2023-11-16T16:46:37.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:37 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:37.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:37 smithi141 ceph-mon[131184]: purged_snaps scrub starts 2023-11-16T16:46:37.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:37 smithi141 ceph-mon[131184]: purged_snaps scrub ok 2023-11-16T16:46:37.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:37 smithi141 ceph-mon[131184]: Detected new or changed devices on smithi155 2023-11-16T16:46:37.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:37 smithi141 ceph-mon[131184]: Adjusting osd_memory_target on smithi155 to 4292M 2023-11-16T16:46:37.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:37 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:46:37.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:37 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:46:37.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:37 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:46:37.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:37 smithi141 ceph-mon[131184]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-11-16T16:46:37.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:37 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:37.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:37 smithi141 ceph-mon[131184]: osd.11 [v2:172.21.15.155:6824/1791652726,v1:172.21.15.155:6825/1791652726] boot 2023-11-16T16:46:37.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:37 smithi141 ceph-mon[131184]: osdmap e80: 12 total, 12 up, 12 in 2023-11-16T16:46:37.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:37 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:46:37.925 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:46:38.465 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:38 smithi002 ceph-mon[128712]: osdmap e81: 12 total, 12 up, 12 in 2023-11-16T16:46:38.465 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:38 smithi002 ceph-mon[128712]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:38.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:38 smithi141 ceph-mon[131184]: osdmap e81: 12 total, 12 up, 12 in 2023-11-16T16:46:38.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:38 smithi141 ceph-mon[131184]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:38.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:38 smithi155 ceph-mon[131607]: osdmap e81: 12 total, 12 up, 12 in 2023-11-16T16:46:38.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:38 smithi155 ceph-mon[131607]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:39.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.344+0000 7f43d3bc7700 1 -- 172.21.15.2:0/3523714637 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43cc1002f0 msgr2=0x7f43cc100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:39.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.344+0000 7f43d3bc7700 1 --2- 172.21.15.2:0/3523714637 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43cc1002f0 0x7f43cc100700 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f43c8009990 tx=0x7f43c8009ca0 comp rx=0 tx=0).stop 2023-11-16T16:46:39.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.344+0000 7f43d3bc7700 1 -- 172.21.15.2:0/3523714637 shutdown_connections 2023-11-16T16:46:39.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.344+0000 7f43d3bc7700 1 --2- 172.21.15.2:0/3523714637 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43cc101e80 0x7f43cc108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:39.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.344+0000 7f43d3bc7700 1 --2- 172.21.15.2:0/3523714637 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43cc1002f0 0x7f43cc100700 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:39.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.344+0000 7f43d3bc7700 1 --2- 172.21.15.2:0/3523714637 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43cc1014f0 0x7f43cc101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:39.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.344+0000 7f43d3bc7700 1 -- 172.21.15.2:0/3523714637 >> 172.21.15.2:0/3523714637 conn(0x7f43cc0fb900 msgr2=0x7f43cc0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:39.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.344+0000 7f43d3bc7700 1 -- 172.21.15.2:0/3523714637 shutdown_connections 2023-11-16T16:46:39.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.344+0000 7f43d3bc7700 1 -- 172.21.15.2:0/3523714637 wait complete. 2023-11-16T16:46:39.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.345+0000 7f43d3bc7700 1 Processor -- start 2023-11-16T16:46:39.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.345+0000 7f43d3bc7700 1 -- start start 2023-11-16T16:46:39.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.345+0000 7f43d3bc7700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43cc1002f0 0x7f43cc196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:39.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.345+0000 7f43d3bc7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43cc1014f0 0x7f43cc1965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:39.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.346+0000 7f43d3bc7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43cc101e80 0x7f43cc19d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:39.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.346+0000 7f43d3bc7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc19dc20 con 0x7f43cc101e80 2023-11-16T16:46:39.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.346+0000 7f43d3bc7700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc19dd60 con 0x7f43cc1002f0 2023-11-16T16:46:39.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.346+0000 7f43d3bc7700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc19def0 con 0x7f43cc1014f0 2023-11-16T16:46:39.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.346+0000 7f43d2164700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43cc101e80 0x7f43cc19d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:39.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.346+0000 7f43d1963700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43cc1002f0 0x7f43cc196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:39.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.346+0000 7f43d1963700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43cc1002f0 0x7f43cc196090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:33760/0 (socket says 172.21.15.2:33760) 2023-11-16T16:46:39.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.346+0000 7f43d1963700 1 -- 172.21.15.2:0/408253471 learned_addr learned my addr 172.21.15.2:0/408253471 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:46:39.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.346+0000 7f43d1162700 1 --2- 172.21.15.2:0/408253471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43cc1014f0 0x7f43cc1965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:39.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.346+0000 7f43d2164700 1 -- 172.21.15.2:0/408253471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43cc1014f0 msgr2=0x7f43cc1965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:39.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.346+0000 7f43d2164700 1 --2- 172.21.15.2:0/408253471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43cc1014f0 0x7f43cc1965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:39.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.346+0000 7f43d2164700 1 -- 172.21.15.2:0/408253471 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43cc1002f0 msgr2=0x7f43cc196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:39.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.346+0000 7f43d2164700 1 --2- 172.21.15.2:0/408253471 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43cc1002f0 0x7f43cc196090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:39.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.346+0000 7f43d2164700 1 -- 172.21.15.2:0/408253471 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c80095f0 con 0x7f43cc101e80 2023-11-16T16:46:39.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.346+0000 7f43d1963700 1 --2- 172.21.15.2:0/408253471 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43cc1002f0 0x7f43cc196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:46:39.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.347+0000 7f43d2164700 1 --2- 172.21.15.2:0/408253471 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43cc101e80 0x7f43cc19d650 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f43c4009f60 tx=0x7f43c4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:39.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.347+0000 7f43c2ffd700 1 -- 172.21.15.2:0/408253471 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c4018070 con 0x7f43cc101e80 2023-11-16T16:46:39.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.347+0000 7f43d3bc7700 1 -- 172.21.15.2:0/408253471 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43cc10bcb0 con 0x7f43cc101e80 2023-11-16T16:46:39.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.347+0000 7f43c2ffd700 1 -- 172.21.15.2:0/408253471 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43c400f040 con 0x7f43cc101e80 2023-11-16T16:46:39.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.347+0000 7f43c2ffd700 1 -- 172.21.15.2:0/408253471 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c400ccc0 con 0x7f43cc101e80 2023-11-16T16:46:39.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.347+0000 7f43d3bc7700 1 -- 172.21.15.2:0/408253471 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43cc10bb20 con 0x7f43cc101e80 2023-11-16T16:46:39.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.348+0000 7f43d3bc7700 1 -- 172.21.15.2:0/408253471 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43cc066ea0 con 0x7f43cc101e80 2023-11-16T16:46:39.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.350+0000 7f43c2ffd700 1 -- 172.21.15.2:0/408253471 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f43c4003a00 con 0x7f43cc101e80 2023-11-16T16:46:39.353 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.351+0000 7f43c2ffd700 1 --2- 172.21.15.2:0/408253471 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f43b80a1150 0x7f43b80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:39.353 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.352+0000 7f43c2ffd700 1 -- 172.21.15.2:0/408253471 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f43c40cbf50 con 0x7f43cc101e80 2023-11-16T16:46:39.353 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.352+0000 7f43d1963700 1 --2- 172.21.15.2:0/408253471 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f43b80a1150 0x7f43b80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:39.354 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.352+0000 7f43d1963700 1 --2- 172.21.15.2:0/408253471 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f43b80a1150 0x7f43b80a3600 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f43c800b2e0 tx=0x7f43c8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:39.357 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.356+0000 7f43c2ffd700 1 -- 172.21.15.2:0/408253471 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f43c4095b30 con 0x7f43cc101e80 2023-11-16T16:46:39.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.564+0000 7f43d3bc7700 1 -- 172.21.15.2:0/408253471 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f43cc107250 con 0x7f43cc101e80 2023-11-16T16:46:39.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.565+0000 7f43c2ffd700 1 -- 172.21.15.2:0/408253471 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) v1 ==== 74+0+133 (secure 0 0 0) 0x7f43c4099150 con 0x7f43cc101e80 2023-11-16T16:46:39.567 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:46:39.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.567+0000 7f43d3bc7700 1 -- 172.21.15.2:0/408253471 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f43b80a1150 msgr2=0x7f43b80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:39.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.567+0000 7f43d3bc7700 1 --2- 172.21.15.2:0/408253471 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f43b80a1150 0x7f43b80a3600 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f43c800b2e0 tx=0x7f43c8005fb0 comp rx=0 tx=0).stop 2023-11-16T16:46:39.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.567+0000 7f43d3bc7700 1 -- 172.21.15.2:0/408253471 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43cc101e80 msgr2=0x7f43cc19d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:39.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.567+0000 7f43d3bc7700 1 --2- 172.21.15.2:0/408253471 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43cc101e80 0x7f43cc19d650 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f43c4009f60 tx=0x7f43c4009f90 comp rx=0 tx=0).stop 2023-11-16T16:46:39.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.567+0000 7f43d3bc7700 1 -- 172.21.15.2:0/408253471 shutdown_connections 2023-11-16T16:46:39.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.567+0000 7f43d3bc7700 1 --2- 172.21.15.2:0/408253471 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f43b80a1150 0x7f43b80a3600 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:39.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.567+0000 7f43d3bc7700 1 --2- 172.21.15.2:0/408253471 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43cc101e80 0x7f43cc19d650 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:39.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.567+0000 7f43d3bc7700 1 --2- 172.21.15.2:0/408253471 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f43cc1002f0 0x7f43cc196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:39.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.567+0000 7f43d3bc7700 1 --2- 172.21.15.2:0/408253471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43cc1014f0 0x7f43cc1965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:39.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.567+0000 7f43d3bc7700 1 -- 172.21.15.2:0/408253471 >> 172.21.15.2:0/408253471 conn(0x7f43cc0fb900 msgr2=0x7f43cc0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:39.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.567+0000 7f43d3bc7700 1 -- 172.21.15.2:0/408253471 shutdown_connections 2023-11-16T16:46:39.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:39.568+0000 7f43d3bc7700 1 -- 172.21.15.2:0/408253471 wait complete. 2023-11-16T16:46:40.020 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1700153196,"num_in_osds":12,"osd_in_since":1700153178,"num_remapped_pgs":0} 2023-11-16T16:46:40.020 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph osd dump --format=json 2023-11-16T16:46:40.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:39 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/408253471' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-16T16:46:40.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:39 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/408253471' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-16T16:46:40.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:39 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/408253471' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-16T16:46:40.880 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:46:41.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:40 smithi141 ceph-mon[131184]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:41.287 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:40 smithi002 ceph-mon[128712]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:41.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:40 smithi155 ceph-mon[131607]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:42.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.371+0000 7f810eaf4700 1 -- 172.21.15.2:0/2357770286 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8108100ee0 msgr2=0x7f8108104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:44.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.371+0000 7f810eaf4700 1 --2- 172.21.15.2:0/2357770286 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8108100ee0 0x7f8108104eb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f80f40099c0 tx=0x7f80f4009cd0 comp rx=0 tx=0).stop 2023-11-16T16:46:44.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.371+0000 7f810eaf4700 1 -- 172.21.15.2:0/2357770286 shutdown_connections 2023-11-16T16:46:44.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.371+0000 7f810eaf4700 1 --2- 172.21.15.2:0/2357770286 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81081055e0 0x7f810810d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:44.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.371+0000 7f810eaf4700 1 --2- 172.21.15.2:0/2357770286 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8108100540 0x7f8108100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:44.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.371+0000 7f810eaf4700 1 --2- 172.21.15.2:0/2357770286 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8108100ee0 0x7f8108104eb0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:44.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.371+0000 7f810eaf4700 1 -- 172.21.15.2:0/2357770286 >> 172.21.15.2:0/2357770286 conn(0x7f81080fbdc0 msgr2=0x7f81080fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:44.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.371+0000 7f810eaf4700 1 -- 172.21.15.2:0/2357770286 shutdown_connections 2023-11-16T16:46:44.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.372+0000 7f810eaf4700 1 -- 172.21.15.2:0/2357770286 wait complete. 2023-11-16T16:46:44.305 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:42 smithi155 ceph-mon[131607]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:44.305 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:42 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3266580214' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:46:44.306 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:42 smithi141 ceph-mon[131184]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:44.306 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:42 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3266580214' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:46:44.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.372+0000 7f810eaf4700 1 Processor -- start 2023-11-16T16:46:44.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.373+0000 7f810eaf4700 1 -- start start 2023-11-16T16:46:44.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.373+0000 7f810eaf4700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8108100540 0x7f81081963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:44.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.373+0000 7f810eaf4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8108100ee0 0x7f8108196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:44.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.373+0000 7f810eaf4700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81081055e0 0x7f810819ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:44.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.373+0000 7f810eaf4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810819b390 con 0x7f8108100ee0 2023-11-16T16:46:44.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.373+0000 7f810eaf4700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810819b500 con 0x7f81081055e0 2023-11-16T16:46:44.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.373+0000 7f810eaf4700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810819b6c0 con 0x7f8108100540 2023-11-16T16:46:44.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.373+0000 7f8107fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8108100ee0 0x7f8108196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:44.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.373+0000 7f810c890700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8108100540 0x7f81081963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:44.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.373+0000 7f810c890700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8108100540 0x7f81081963d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:55156/0 (socket says 172.21.15.2:55156) 2023-11-16T16:46:44.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.373+0000 7f810c890700 1 -- 172.21.15.2:0/3266580214 learned_addr learned my addr 172.21.15.2:0/3266580214 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:46:44.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.373+0000 7f810d091700 1 --2- 172.21.15.2:0/3266580214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81081055e0 0x7f810819ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:44.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.374+0000 7f8107fff700 1 -- 172.21.15.2:0/3266580214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8108100540 msgr2=0x7f81081963d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:44.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.374+0000 7f8107fff700 1 --2- 172.21.15.2:0/3266580214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8108100540 0x7f81081963d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:44.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.374+0000 7f8107fff700 1 -- 172.21.15.2:0/3266580214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81081055e0 msgr2=0x7f810819ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:44.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.374+0000 7f8107fff700 1 --2- 172.21.15.2:0/3266580214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81081055e0 0x7f810819ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:44.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.374+0000 7f8107fff700 1 -- 172.21.15.2:0/3266580214 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f40095f0 con 0x7f8108100ee0 2023-11-16T16:46:44.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.374+0000 7f810c890700 1 --2- 172.21.15.2:0/3266580214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8108100540 0x7f81081963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:46:44.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.374+0000 7f8107fff700 1 --2- 172.21.15.2:0/3266580214 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8108100ee0 0x7f8108196910 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f80f4009990 tx=0x7f80f4005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:44.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.374+0000 7f8105ffb700 1 -- 172.21.15.2:0/3266580214 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f401d070 con 0x7f8108100ee0 2023-11-16T16:46:44.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.374+0000 7f810eaf4700 1 -- 172.21.15.2:0/3266580214 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8108110360 con 0x7f8108100ee0 2023-11-16T16:46:44.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.374+0000 7f8105ffb700 1 -- 172.21.15.2:0/3266580214 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f80f4003f90 con 0x7f8108100ee0 2023-11-16T16:46:44.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.374+0000 7f8105ffb700 1 -- 172.21.15.2:0/3266580214 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f4022b40 con 0x7f8108100ee0 2023-11-16T16:46:44.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.375+0000 7f810eaf4700 1 -- 172.21.15.2:0/3266580214 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81081101d0 con 0x7f8108100ee0 2023-11-16T16:46:44.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.376+0000 7f810eaf4700 1 -- 172.21.15.2:0/3266580214 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81081104f0 con 0x7f8108100ee0 2023-11-16T16:46:44.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.377+0000 7f8105ffb700 1 -- 172.21.15.2:0/3266580214 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f80f4003ad0 con 0x7f8108100ee0 2023-11-16T16:46:44.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.379+0000 7f8105ffb700 1 --2- 172.21.15.2:0/3266580214 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f80f80a1150 0x7f80f80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:44.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.379+0000 7f8105ffb700 1 -- 172.21.15.2:0/3266580214 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f80f40cd900 con 0x7f8108100ee0 2023-11-16T16:46:44.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.379+0000 7f810c890700 1 --2- 172.21.15.2:0/3266580214 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f80f80a1150 0x7f80f80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:44.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.380+0000 7f810c890700 1 --2- 172.21.15.2:0/3266580214 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f80f80a1150 0x7f80f80a3600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f80fc0104e0 tx=0x7f80fc007480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:44.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.383+0000 7f8105ffb700 1 -- 172.21.15.2:0/3266580214 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f80f40974e0 con 0x7f8108100ee0 2023-11-16T16:46:44.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.592+0000 7f810eaf4700 1 -- 172.21.15.2:0/3266580214 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8108066ea0 con 0x7f8108100ee0 2023-11-16T16:46:44.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.593+0000 7f8105ffb700 1 -- 172.21.15.2:0/3266580214 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18374 (secure 0 0 0) 0x7f80f409ab00 con 0x7f8108100ee0 2023-11-16T16:46:44.311 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:46:44.311 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":81,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:46:37.203188+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:46:44.312 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.596+0000 7f810eaf4700 1 -- 172.21.15.2:0/3266580214 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f80f80a1150 msgr2=0x7f80f80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:44.312 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.596+0000 7f810eaf4700 1 --2- 172.21.15.2:0/3266580214 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f80f80a1150 0x7f80f80a3600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f80fc0104e0 tx=0x7f80fc007480 comp rx=0 tx=0).stop 2023-11-16T16:46:44.312 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.596+0000 7f810eaf4700 1 -- 172.21.15.2:0/3266580214 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8108100ee0 msgr2=0x7f8108196910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:44.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.596+0000 7f810eaf4700 1 --2- 172.21.15.2:0/3266580214 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8108100ee0 0x7f8108196910 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f80f4009990 tx=0x7f80f4005740 comp rx=0 tx=0).stop 2023-11-16T16:46:44.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.596+0000 7f810eaf4700 1 -- 172.21.15.2:0/3266580214 shutdown_connections 2023-11-16T16:46:44.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.596+0000 7f810eaf4700 1 --2- 172.21.15.2:0/3266580214 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f80f80a1150 0x7f80f80a3600 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:44.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.596+0000 7f810eaf4700 1 --2- 172.21.15.2:0/3266580214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8108100540 0x7f81081963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:44.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.596+0000 7f810eaf4700 1 --2- 172.21.15.2:0/3266580214 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81081055e0 0x7f810819ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:44.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.596+0000 7f810eaf4700 1 --2- 172.21.15.2:0/3266580214 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8108100ee0 0x7f8108196910 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:44.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.596+0000 7f810eaf4700 1 -- 172.21.15.2:0/3266580214 >> 172.21.15.2:0/3266580214 conn(0x7f81080fbdc0 msgr2=0x7f81080fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:44.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.596+0000 7f810eaf4700 1 -- 172.21.15.2:0/3266580214 shutdown_connections 2023-11-16T16:46:44.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:42.596+0000 7f810eaf4700 1 -- 172.21.15.2:0/3266580214 wait complete. 2023-11-16T16:46:44.314 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:04.306584+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-16T16:46:44.315 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph osd pool get .mgr pg_num 2023-11-16T16:46:44.315 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:42 smithi002 ceph-mon[128712]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:44.315 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:42 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3266580214' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:46:45.187 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:46:45.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:44 smithi141 ceph-mon[131184]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:45.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:44 smithi155 ceph-mon[131607]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:45.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:44 smithi002 ceph-mon[128712]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:46.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.715+0000 7f31e5975700 1 -- 172.21.15.2:0/1240287105 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31e0100540 msgr2=0x7f31e0100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:46.718 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.715+0000 7f31e5975700 1 --2- 172.21.15.2:0/1240287105 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31e0100540 0x7f31e0100910 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f31d00099c0 tx=0x7f31d0009cd0 comp rx=0 tx=0).stop 2023-11-16T16:46:46.718 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.715+0000 7f31e5975700 1 -- 172.21.15.2:0/1240287105 shutdown_connections 2023-11-16T16:46:46.718 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.715+0000 7f31e5975700 1 --2- 172.21.15.2:0/1240287105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31e01055e0 0x7f31e010d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:46.718 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.715+0000 7f31e5975700 1 --2- 172.21.15.2:0/1240287105 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31e0100540 0x7f31e0100910 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:46.718 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.715+0000 7f31e5975700 1 --2- 172.21.15.2:0/1240287105 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31e0100ee0 0x7f31e0104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:46.718 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.715+0000 7f31e5975700 1 -- 172.21.15.2:0/1240287105 >> 172.21.15.2:0/1240287105 conn(0x7f31e00fbdc0 msgr2=0x7f31e00fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:46.718 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.715+0000 7f31e5975700 1 -- 172.21.15.2:0/1240287105 shutdown_connections 2023-11-16T16:46:46.718 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.715+0000 7f31e5975700 1 -- 172.21.15.2:0/1240287105 wait complete. 2023-11-16T16:46:46.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.716+0000 7f31e5975700 1 Processor -- start 2023-11-16T16:46:46.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.717+0000 7f31e5975700 1 -- start start 2023-11-16T16:46:46.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.717+0000 7f31e5975700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31e0100540 0x7f31e0198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:46.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.717+0000 7f31e5975700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31e0100ee0 0x7f31e0198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:46.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.717+0000 7f31e5975700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31e01055e0 0x7f31e019cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:46.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.717+0000 7f31e5975700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e019d580 con 0x7f31e0100ee0 2023-11-16T16:46:46.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.717+0000 7f31e5975700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e019d6f0 con 0x7f31e0100540 2023-11-16T16:46:46.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.717+0000 7f31e5975700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e019d8b0 con 0x7f31e01055e0 2023-11-16T16:46:46.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.717+0000 7f31de7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31e0100ee0 0x7f31e0198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:46.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.717+0000 7f31de7fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31e0100ee0 0x7f31e0198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36022/0 (socket says 172.21.15.2:36022) 2023-11-16T16:46:46.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.717+0000 7f31de7fc700 1 -- 172.21.15.2:0/1690976105 learned_addr learned my addr 172.21.15.2:0/1690976105 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:46:46.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.717+0000 7f31df7fe700 1 --2- 172.21.15.2:0/1690976105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31e01055e0 0x7f31e019cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:46.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.717+0000 7f31deffd700 1 --2- 172.21.15.2:0/1690976105 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31e0100540 0x7f31e0198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:46.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.718+0000 7f31de7fc700 1 -- 172.21.15.2:0/1690976105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31e01055e0 msgr2=0x7f31e019cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:46.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.718+0000 7f31de7fc700 1 --2- 172.21.15.2:0/1690976105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31e01055e0 0x7f31e019cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:46.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.718+0000 7f31de7fc700 1 -- 172.21.15.2:0/1690976105 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31e0100540 msgr2=0x7f31e0198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:46.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.718+0000 7f31de7fc700 1 --2- 172.21.15.2:0/1690976105 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31e0100540 0x7f31e0198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:46.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.718+0000 7f31de7fc700 1 -- 172.21.15.2:0/1690976105 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d00095f0 con 0x7f31e0100ee0 2023-11-16T16:46:46.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.718+0000 7f31de7fc700 1 --2- 172.21.15.2:0/1690976105 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31e0100ee0 0x7f31e0198ba0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f31d400e970 tx=0x7f31d400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:46.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.718+0000 7f31e4973700 1 -- 172.21.15.2:0/1690976105 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d4009790 con 0x7f31e0100ee0 2023-11-16T16:46:46.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.718+0000 7f31e5975700 1 -- 172.21.15.2:0/1690976105 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e0110320 con 0x7f31e0100ee0 2023-11-16T16:46:46.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.718+0000 7f31e4973700 1 -- 172.21.15.2:0/1690976105 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f31d400c7e0 con 0x7f31e0100ee0 2023-11-16T16:46:46.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.718+0000 7f31e4973700 1 -- 172.21.15.2:0/1690976105 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d40054e0 con 0x7f31e0100ee0 2023-11-16T16:46:46.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.718+0000 7f31e5975700 1 -- 172.21.15.2:0/1690976105 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e0110190 con 0x7f31e0100ee0 2023-11-16T16:46:46.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.719+0000 7f31e5975700 1 -- 172.21.15.2:0/1690976105 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e01104b0 con 0x7f31e0100ee0 2023-11-16T16:46:46.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.721+0000 7f31e4973700 1 -- 172.21.15.2:0/1690976105 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f31d400c950 con 0x7f31e0100ee0 2023-11-16T16:46:46.724 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.723+0000 7f31e4973700 1 --2- 172.21.15.2:0/1690976105 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31c80a11d0 0x7f31c80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:46.725 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.723+0000 7f31e4973700 1 -- 172.21.15.2:0/1690976105 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f31d4014070 con 0x7f31e0100ee0 2023-11-16T16:46:46.725 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.723+0000 7f31deffd700 1 --2- 172.21.15.2:0/1690976105 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31c80a11d0 0x7f31c80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:46.725 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.724+0000 7f31deffd700 1 --2- 172.21.15.2:0/1690976105 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31c80a11d0 0x7f31c80a3680 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f31d0009fd0 tx=0x7f31d0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:46.729 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.727+0000 7f31e4973700 1 -- 172.21.15.2:0/1690976105 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f31d4096260 con 0x7f31e0100ee0 2023-11-16T16:46:46.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.933+0000 7f31e5975700 1 -- 172.21.15.2:0/1690976105 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f31e01a5700 con 0x7f31e0100ee0 2023-11-16T16:46:46.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.934+0000 7f31e4973700 1 -- 172.21.15.2:0/1690976105 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f31d4099880 con 0x7f31e0100ee0 2023-11-16T16:46:46.935 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T16:46:46.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.936+0000 7f31e5975700 1 -- 172.21.15.2:0/1690976105 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31c80a11d0 msgr2=0x7f31c80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:46.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.936+0000 7f31e5975700 1 --2- 172.21.15.2:0/1690976105 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31c80a11d0 0x7f31c80a3680 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f31d0009fd0 tx=0x7f31d0005fb0 comp rx=0 tx=0).stop 2023-11-16T16:46:46.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.936+0000 7f31e5975700 1 -- 172.21.15.2:0/1690976105 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31e0100ee0 msgr2=0x7f31e0198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:46.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.936+0000 7f31e5975700 1 --2- 172.21.15.2:0/1690976105 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31e0100ee0 0x7f31e0198ba0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f31d400e970 tx=0x7f31d400ec80 comp rx=0 tx=0).stop 2023-11-16T16:46:46.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.936+0000 7f31e5975700 1 -- 172.21.15.2:0/1690976105 shutdown_connections 2023-11-16T16:46:46.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.936+0000 7f31e5975700 1 --2- 172.21.15.2:0/1690976105 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31c80a11d0 0x7f31c80a3680 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:46.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.936+0000 7f31e5975700 1 --2- 172.21.15.2:0/1690976105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31e01055e0 0x7f31e019cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:46.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.936+0000 7f31e5975700 1 --2- 172.21.15.2:0/1690976105 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31e0100540 0x7f31e0198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:46.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.936+0000 7f31e5975700 1 --2- 172.21.15.2:0/1690976105 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31e0100ee0 0x7f31e0198ba0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:46.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.936+0000 7f31e5975700 1 -- 172.21.15.2:0/1690976105 >> 172.21.15.2:0/1690976105 conn(0x7f31e00fbdc0 msgr2=0x7f31e00fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:46.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.936+0000 7f31e5975700 1 -- 172.21.15.2:0/1690976105 shutdown_connections 2023-11-16T16:46:46.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:46.936+0000 7f31e5975700 1 -- 172.21.15.2:0/1690976105 wait complete. 2023-11-16T16:46:47.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:46 smithi141 ceph-mon[131184]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:47.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:46 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1690976105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:46:47.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:46 smithi155 ceph-mon[131607]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:47.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:46 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1690976105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:46:47.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:46 smithi002 ceph-mon[128712]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:47.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:46 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1690976105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:46:47.471 INFO:tasks.cephadm:Adding mds.a on smithi002 2023-11-16T16:46:47.471 INFO:tasks.cephadm:Adding mds.d on smithi002 2023-11-16T16:46:47.471 INFO:tasks.cephadm:Adding mds.g on smithi002 2023-11-16T16:46:47.471 INFO:tasks.cephadm:Adding mds.j on smithi002 2023-11-16T16:46:47.471 INFO:tasks.cephadm:Adding mds.b on smithi141 2023-11-16T16:46:47.472 INFO:tasks.cephadm:Adding mds.e on smithi141 2023-11-16T16:46:47.472 INFO:tasks.cephadm:Adding mds.h on smithi141 2023-11-16T16:46:47.472 INFO:tasks.cephadm:Adding mds.k on smithi141 2023-11-16T16:46:47.472 INFO:tasks.cephadm:Adding mds.c on smithi155 2023-11-16T16:46:47.472 INFO:tasks.cephadm:Adding mds.f on smithi155 2023-11-16T16:46:47.472 INFO:tasks.cephadm:Adding mds.i on smithi155 2023-11-16T16:46:47.472 INFO:tasks.cephadm:Adding mds.l on smithi155 2023-11-16T16:46:47.472 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph orch apply mds all '12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi141=b;smithi141=e;smithi141=h;smithi141=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l' 2023-11-16T16:46:48.130 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T16:46:49.151 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:48 smithi155 ceph-mon[131607]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:49.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:48 smithi141 ceph-mon[131184]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:49.343 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.340+0000 7f11497dc700 1 -- 172.21.15.155:0/3508220966 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11440fe960 msgr2=0x7f11440fed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:49.343 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.340+0000 7f11497dc700 1 --2- 172.21.15.155:0/3508220966 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11440fe960 0x7f11440fed70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1138009990 tx=0x7f1138009ca0 comp rx=0 tx=0).stop 2023-11-16T16:46:49.343 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.340+0000 7f11497dc700 1 -- 172.21.15.155:0/3508220966 shutdown_connections 2023-11-16T16:46:49.343 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.340+0000 7f11497dc700 1 --2- 172.21.15.155:0/3508220966 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11440fe960 0x7f11440fed70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:49.343 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.340+0000 7f11497dc700 1 --2- 172.21.15.155:0/3508220966 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11441004f0 0x7f1144106db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:49.344 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.340+0000 7f11497dc700 1 --2- 172.21.15.155:0/3508220966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11440ffb60 0x7f11440fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:49.344 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.340+0000 7f11497dc700 1 -- 172.21.15.155:0/3508220966 >> 172.21.15.155:0/3508220966 conn(0x7f11440f9f70 msgr2=0x7f11440fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:49.344 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.340+0000 7f11497dc700 1 -- 172.21.15.155:0/3508220966 shutdown_connections 2023-11-16T16:46:49.344 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.340+0000 7f11497dc700 1 -- 172.21.15.155:0/3508220966 wait complete. 2023-11-16T16:46:49.344 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.340+0000 7f11497dc700 1 Processor -- start 2023-11-16T16:46:49.344 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.341+0000 7f11497dc700 1 -- start start 2023-11-16T16:46:49.344 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.341+0000 7f11497dc700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11440fe960 0x7f1144108220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:49.344 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.341+0000 7f11497dc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11440ffb60 0x7f114419adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:49.344 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.341+0000 7f11497dc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11441004f0 0x7f114419d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:49.344 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.341+0000 7f11497dc700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114419dba0 con 0x7f11440ffb60 2023-11-16T16:46:49.344 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.341+0000 7f11497dc700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114419dd10 con 0x7f11440fe960 2023-11-16T16:46:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.341+0000 7f11497dc700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114419ded0 con 0x7f11441004f0 2023-11-16T16:46:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.341+0000 7f11437fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11441004f0 0x7f114419d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.341+0000 7f11437fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11441004f0 0x7f114419d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:39496/0 (socket says 172.21.15.155:39496) 2023-11-16T16:46:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.341+0000 7f11437fe700 1 -- 172.21.15.155:0/73319750 learned_addr learned my addr 172.21.15.155:0/73319750 (peer_addr_for_me v2:172.21.15.155:0/0) 2023-11-16T16:46:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.341+0000 7f1142ffd700 1 --2- 172.21.15.155:0/73319750 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11440fe960 0x7f1144108220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.341+0000 7f11427fc700 1 --2- 172.21.15.155:0/73319750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11440ffb60 0x7f114419adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.341+0000 7f11437fe700 1 -- 172.21.15.155:0/73319750 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11440fe960 msgr2=0x7f1144108220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.341+0000 7f11437fe700 1 --2- 172.21.15.155:0/73319750 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11440fe960 0x7f1144108220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.341+0000 7f11437fe700 1 -- 172.21.15.155:0/73319750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11440ffb60 msgr2=0x7f114419adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.341+0000 7f11437fe700 1 --2- 172.21.15.155:0/73319750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11440ffb60 0x7f114419adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.341+0000 7f11437fe700 1 -- 172.21.15.155:0/73319750 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11380095f0 con 0x7f11441004f0 2023-11-16T16:46:49.346 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.341+0000 7f1142ffd700 1 --2- 172.21.15.155:0/73319750 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11440fe960 0x7f1144108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:46:49.346 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.341+0000 7f11437fe700 1 --2- 172.21.15.155:0/73319750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11441004f0 0x7f114419d400 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f113c007ee0 tx=0x7f113c00d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:49.346 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.342+0000 7f112bfff700 1 -- 172.21.15.155:0/73319750 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113c00ddd0 con 0x7f11441004f0 2023-11-16T16:46:49.346 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.342+0000 7f11497dc700 1 -- 172.21.15.155:0/73319750 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f114410ab40 con 0x7f11441004f0 2023-11-16T16:46:49.346 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.342+0000 7f112bfff700 1 -- 172.21.15.155:0/73319750 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f113c00f040 con 0x7f11441004f0 2023-11-16T16:46:49.346 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.342+0000 7f112bfff700 1 -- 172.21.15.155:0/73319750 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113c015400 con 0x7f11441004f0 2023-11-16T16:46:49.346 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.342+0000 7f11497dc700 1 -- 172.21.15.155:0/73319750 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f114410a9b0 con 0x7f11441004f0 2023-11-16T16:46:49.347 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.343+0000 7f11497dc700 1 -- 172.21.15.155:0/73319750 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1144105fb0 con 0x7f11441004f0 2023-11-16T16:46:49.347 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.344+0000 7f112bfff700 1 -- 172.21.15.155:0/73319750 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f113c004000 con 0x7f11441004f0 2023-11-16T16:46:49.347 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.344+0000 7f112bfff700 1 --2- 172.21.15.155:0/73319750 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f112c0a1150 0x7f112c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:49.347 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.344+0000 7f112bfff700 1 -- 172.21.15.155:0/73319750 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f113c0cc650 con 0x7f11441004f0 2023-11-16T16:46:49.347 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.345+0000 7f1142ffd700 1 --2- 172.21.15.155:0/73319750 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f112c0a1150 0x7f112c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:49.347 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.345+0000 7f1142ffd700 1 --2- 172.21.15.155:0/73319750 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f112c0a1150 0x7f112c0a3600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1138006010 tx=0x7f113800b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:49.349 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.348+0000 7f112bfff700 1 -- 172.21.15.155:0/73319750 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f113c096240 con 0x7f11441004f0 2023-11-16T16:46:49.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:48 smithi002 ceph-mon[128712]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:49.544 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.543+0000 7f11497dc700 1 -- 172.21.15.155:0/73319750 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi141=b;smithi141=e;smithi141=h;smithi141=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f1144061cc0 con 0x7f112c0a1150 2023-11-16T16:46:49.553 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.552+0000 7f112bfff700 1 -- 172.21.15.155:0/73319750 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f1144061cc0 con 0x7f112c0a1150 2023-11-16T16:46:49.553 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled mds.all update... 2023-11-16T16:46:49.555 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.554+0000 7f11497dc700 1 -- 172.21.15.155:0/73319750 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f112c0a1150 msgr2=0x7f112c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:49.555 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.554+0000 7f11497dc700 1 --2- 172.21.15.155:0/73319750 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f112c0a1150 0x7f112c0a3600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1138006010 tx=0x7f113800b540 comp rx=0 tx=0).stop 2023-11-16T16:46:49.555 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.554+0000 7f11497dc700 1 -- 172.21.15.155:0/73319750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11441004f0 msgr2=0x7f114419d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:49.556 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.554+0000 7f11497dc700 1 --2- 172.21.15.155:0/73319750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11441004f0 0x7f114419d400 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f113c007ee0 tx=0x7f113c00d3b0 comp rx=0 tx=0).stop 2023-11-16T16:46:49.556 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.554+0000 7f11497dc700 1 -- 172.21.15.155:0/73319750 shutdown_connections 2023-11-16T16:46:49.556 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.554+0000 7f11497dc700 1 --2- 172.21.15.155:0/73319750 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f112c0a1150 0x7f112c0a3600 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:49.556 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.554+0000 7f11497dc700 1 --2- 172.21.15.155:0/73319750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11441004f0 0x7f114419d400 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:49.556 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.554+0000 7f11497dc700 1 --2- 172.21.15.155:0/73319750 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11440fe960 0x7f1144108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:49.556 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.554+0000 7f11497dc700 1 --2- 172.21.15.155:0/73319750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11440ffb60 0x7f114419adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:49.556 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.554+0000 7f11497dc700 1 -- 172.21.15.155:0/73319750 >> 172.21.15.155:0/73319750 conn(0x7f11440f9f70 msgr2=0x7f11441030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:49.556 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.554+0000 7f11497dc700 1 -- 172.21.15.155:0/73319750 shutdown_connections 2023-11-16T16:46:49.556 INFO:teuthology.orchestra.run.smithi155.stderr:2023-11-16T16:46:49.554+0000 7f11497dc700 1 -- 172.21.15.155:0/73319750 wait complete. 2023-11-16T16:46:49.878 DEBUG:teuthology.orchestra.run.smithi002:mds.a> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.a.service 2023-11-16T16:46:49.883 DEBUG:teuthology.orchestra.run.smithi002:mds.d> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.d.service 2023-11-16T16:46:49.929 DEBUG:teuthology.orchestra.run.smithi002:mds.g> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.g.service 2023-11-16T16:46:49.966 INFO:journalctl@ceph.mds.a.smithi002.stdout:-- Logs begin at Thu 2023-11-16 16:19:53 UTC. -- 2023-11-16T16:46:49.970 DEBUG:teuthology.orchestra.run.smithi002:mds.j> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.j.service 2023-11-16T16:46:50.003 INFO:journalctl@ceph.mds.d.smithi002.stdout:-- Logs begin at Thu 2023-11-16 16:19:53 UTC. -- 2023-11-16T16:46:50.007 DEBUG:teuthology.orchestra.run.smithi141:mds.b> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.b.service 2023-11-16T16:46:50.009 DEBUG:teuthology.orchestra.run.smithi141:mds.e> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.e.service 2023-11-16T16:46:50.039 INFO:journalctl@ceph.mds.b.smithi141.stdout:-- Logs begin at Thu 2023-11-16 16:14:35 UTC. -- 2023-11-16T16:46:50.040 DEBUG:teuthology.orchestra.run.smithi141:mds.h> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.h.service 2023-11-16T16:46:50.043 INFO:journalctl@ceph.mds.g.smithi002.stdout:-- Logs begin at Thu 2023-11-16 16:19:53 UTC. -- 2023-11-16T16:46:50.066 INFO:journalctl@ceph.mds.e.smithi141.stdout:-- Logs begin at Thu 2023-11-16 16:14:35 UTC. -- 2023-11-16T16:46:50.067 DEBUG:teuthology.orchestra.run.smithi141:mds.k> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.k.service 2023-11-16T16:46:50.077 INFO:journalctl@ceph.mds.j.smithi002.stdout:-- Logs begin at Thu 2023-11-16 16:19:53 UTC. -- 2023-11-16T16:46:50.094 INFO:journalctl@ceph.mds.h.smithi141.stdout:-- Logs begin at Thu 2023-11-16 16:14:35 UTC. -- 2023-11-16T16:46:50.095 DEBUG:teuthology.orchestra.run.smithi155:mds.c> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.c.service 2023-11-16T16:46:50.098 DEBUG:teuthology.orchestra.run.smithi155:mds.f> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.f.service 2023-11-16T16:46:50.120 INFO:journalctl@ceph.mds.k.smithi141.stdout:-- Logs begin at Thu 2023-11-16 16:14:35 UTC. -- 2023-11-16T16:46:50.127 INFO:journalctl@ceph.mds.c.smithi155.stdout:-- Logs begin at Thu 2023-11-16 16:15:48 UTC. -- 2023-11-16T16:46:50.129 DEBUG:teuthology.orchestra.run.smithi155:mds.i> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.i.service 2023-11-16T16:46:50.155 INFO:journalctl@ceph.mds.f.smithi155.stdout:-- Logs begin at Thu 2023-11-16 16:15:48 UTC. -- 2023-11-16T16:46:50.156 DEBUG:teuthology.orchestra.run.smithi155:mds.l> sudo journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.l.service 2023-11-16T16:46:50.181 INFO:journalctl@ceph.mds.i.smithi155.stdout:-- Logs begin at Thu 2023-11-16 16:15:48 UTC. -- 2023-11-16T16:46:50.181 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-11-16T16:46:50.182 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:46:50.206 INFO:journalctl@ceph.mds.l.smithi155.stdout:-- Logs begin at Thu 2023-11-16 16:15:48 UTC. -- 2023-11-16T16:46:50.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.405+0000 7fbce33ff700 1 -- 172.21.15.2:0/3600073883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcdc1002b0 msgr2=0x7fbcdc1006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:50.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.405+0000 7fbce33ff700 1 --2- 172.21.15.2:0/3600073883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcdc1002b0 0x7fbcdc1006c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80098f0 tx=0x7fbcd8009c00 comp rx=0 tx=0).stop 2023-11-16T16:46:50.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.405+0000 7fbce33ff700 1 -- 172.21.15.2:0/3600073883 shutdown_connections 2023-11-16T16:46:50.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.405+0000 7fbce33ff700 1 --2- 172.21.15.2:0/3600073883 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbcdc101e30 0x7fbcdc1086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.405+0000 7fbce33ff700 1 --2- 172.21.15.2:0/3600073883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcdc1002b0 0x7fbcdc1006c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.405+0000 7fbce33ff700 1 --2- 172.21.15.2:0/3600073883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcdc1014a0 0x7fbcdc1018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.405+0000 7fbce33ff700 1 -- 172.21.15.2:0/3600073883 >> 172.21.15.2:0/3600073883 conn(0x7fbcdc0fb890 msgr2=0x7fbcdc0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:50.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.405+0000 7fbce33ff700 1 -- 172.21.15.2:0/3600073883 shutdown_connections 2023-11-16T16:46:50.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.405+0000 7fbce33ff700 1 -- 172.21.15.2:0/3600073883 wait complete. 2023-11-16T16:46:50.410 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.406+0000 7fbce33ff700 1 Processor -- start 2023-11-16T16:46:50.410 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.406+0000 7fbce33ff700 1 -- start start 2023-11-16T16:46:50.410 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.407+0000 7fbce33ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbcdc1002b0 0x7fbcdc198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:50.410 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.407+0000 7fbce33ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcdc1014a0 0x7fbcdc198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:50.410 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.407+0000 7fbce33ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcdc101e30 0x7fbcdc19f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:50.410 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.407+0000 7fbce33ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc19fdd0 con 0x7fbcdc101e30 2023-11-16T16:46:50.410 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.407+0000 7fbce33ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc19ff10 con 0x7fbcdc1002b0 2023-11-16T16:46:50.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.407+0000 7fbce33ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcdc1a00a0 con 0x7fbcdc1014a0 2023-11-16T16:46:50.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.407+0000 7fbce119b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbcdc1002b0 0x7fbcdc198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:50.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.407+0000 7fbce099a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcdc1014a0 0x7fbcdc198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:50.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.407+0000 7fbce199c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcdc101e30 0x7fbcdc19f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:50.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.407+0000 7fbce119b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbcdc1002b0 0x7fbcdc198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:60092/0 (socket says 172.21.15.2:60092) 2023-11-16T16:46:50.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.407+0000 7fbce099a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcdc1014a0 0x7fbcdc198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:47892/0 (socket says 172.21.15.2:47892) 2023-11-16T16:46:50.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.407+0000 7fbce119b700 1 -- 172.21.15.2:0/3951837027 learned_addr learned my addr 172.21.15.2:0/3951837027 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:46:50.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.407+0000 7fbce119b700 1 -- 172.21.15.2:0/3951837027 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcdc1014a0 msgr2=0x7fbcdc198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:50.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.407+0000 7fbce119b700 1 --2- 172.21.15.2:0/3951837027 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcdc1014a0 0x7fbcdc198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.407+0000 7fbce119b700 1 -- 172.21.15.2:0/3951837027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcdc101e30 msgr2=0x7fbcdc19f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:50.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.407+0000 7fbce119b700 1 --2- 172.21.15.2:0/3951837027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcdc101e30 0x7fbcdc19f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.407+0000 7fbce119b700 1 -- 172.21.15.2:0/3951837027 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc009520 con 0x7fbcdc1002b0 2023-11-16T16:46:50.412 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.408+0000 7fbce119b700 1 --2- 172.21.15.2:0/3951837027 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbcdc1002b0 0x7fbcdc198250 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8005e90 tx=0x7fbcd800b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:50.412 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.408+0000 7fbcd27fc700 1 -- 172.21.15.2:0/3951837027 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd801c070 con 0x7fbcdc1002b0 2023-11-16T16:46:50.412 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.408+0000 7fbcd27fc700 1 -- 172.21.15.2:0/3951837027 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbcd800f7f0 con 0x7fbcdc1002b0 2023-11-16T16:46:50.412 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.408+0000 7fbcd27fc700 1 -- 172.21.15.2:0/3951837027 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8003990 con 0x7fbcdc1002b0 2023-11-16T16:46:50.412 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.408+0000 7fbce33ff700 1 -- 172.21.15.2:0/3951837027 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd4009d00 con 0x7fbcdc1002b0 2023-11-16T16:46:50.412 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.408+0000 7fbce33ff700 1 -- 172.21.15.2:0/3951837027 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd8009520 con 0x7fbcdc1002b0 2023-11-16T16:46:50.412 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.409+0000 7fbce33ff700 1 -- 172.21.15.2:0/3951837027 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcdc192910 con 0x7fbcdc1002b0 2023-11-16T16:46:50.413 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.411+0000 7fbcd27fc700 1 -- 172.21.15.2:0/3951837027 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fbcd8020030 con 0x7fbcdc1002b0 2023-11-16T16:46:50.413 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.412+0000 7fbcd27fc700 1 --2- 172.21.15.2:0/3951837027 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbcc80a11d0 0x7fbcc80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:50.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.412+0000 7fbcd27fc700 1 -- 172.21.15.2:0/3951837027 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbcd80cd3c0 con 0x7fbcdc1002b0 2023-11-16T16:46:50.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.412+0000 7fbce099a700 1 --2- 172.21.15.2:0/3951837027 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbcc80a11d0 0x7fbcc80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:50.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.413+0000 7fbce099a700 1 --2- 172.21.15.2:0/3951837027 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbcc80a11d0 0x7fbcc80a3680 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbccc009df0 tx=0x7fbccc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:50.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.416+0000 7fbcd27fc700 1 -- 172.21.15.2:0/3951837027 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbcd8096fa0 con 0x7fbcdc1002b0 2023-11-16T16:46:50.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.627+0000 7fbce33ff700 1 -- 172.21.15.2:0/3951837027 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbcdc061700 con 0x7fbcdc1002b0 2023-11-16T16:46:50.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.629+0000 7fbcd27fc700 1 -- 172.21.15.2:0/3951837027 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18374 (secure 0 0 0) 0x7fbcd809a5c0 con 0x7fbcdc1002b0 2023-11-16T16:46:50.631 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:46:50.631 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":81,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:46:37.203188+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:46:50.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.631+0000 7fbce33ff700 1 -- 172.21.15.2:0/3951837027 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbcc80a11d0 msgr2=0x7fbcc80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:50.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.631+0000 7fbce33ff700 1 --2- 172.21.15.2:0/3951837027 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbcc80a11d0 0x7fbcc80a3680 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbccc009df0 tx=0x7fbccc009260 comp rx=0 tx=0).stop 2023-11-16T16:46:50.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.631+0000 7fbce33ff700 1 -- 172.21.15.2:0/3951837027 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbcdc1002b0 msgr2=0x7fbcdc198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:50.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.631+0000 7fbce33ff700 1 --2- 172.21.15.2:0/3951837027 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbcdc1002b0 0x7fbcdc198250 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8005e90 tx=0x7fbcd800b730 comp rx=0 tx=0).stop 2023-11-16T16:46:50.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.631+0000 7fbce33ff700 1 -- 172.21.15.2:0/3951837027 shutdown_connections 2023-11-16T16:46:50.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.631+0000 7fbce33ff700 1 --2- 172.21.15.2:0/3951837027 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbcc80a11d0 0x7fbcc80a3680 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.631+0000 7fbce33ff700 1 --2- 172.21.15.2:0/3951837027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbcdc101e30 0x7fbcdc19f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.631+0000 7fbce33ff700 1 --2- 172.21.15.2:0/3951837027 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbcdc1002b0 0x7fbcdc198250 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.631+0000 7fbce33ff700 1 --2- 172.21.15.2:0/3951837027 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcdc1014a0 0x7fbcdc198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.631+0000 7fbce33ff700 1 -- 172.21.15.2:0/3951837027 >> 172.21.15.2:0/3951837027 conn(0x7fbcdc0fb890 msgr2=0x7fbcdc106e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:50.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.631+0000 7fbce33ff700 1 -- 172.21.15.2:0/3951837027 shutdown_connections 2023-11-16T16:46:50.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.631+0000 7fbce33ff700 1 -- 172.21.15.2:0/3951837027 wait complete. 2023-11-16T16:46:50.653 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:04.306584+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-16T16:46:50.653 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:46:50.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.860+0000 7f8dfe7ff700 1 -- 172.21.15.2:0/3143195455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8df81002c0 msgr2=0x7f8df81006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:50.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.860+0000 7f8dfe7ff700 1 --2- 172.21.15.2:0/3143195455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8df81002c0 0x7f8df81006d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009990 tx=0x7f8de8009ca0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.860+0000 7f8dfe7ff700 1 -- 172.21.15.2:0/3143195455 shutdown_connections 2023-11-16T16:46:50.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.860+0000 7f8dfe7ff700 1 --2- 172.21.15.2:0/3143195455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8df81002c0 0x7f8df81006d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.860+0000 7f8dfe7ff700 1 --2- 172.21.15.2:0/3143195455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8df8101e40 0x7f8df8108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.860+0000 7f8dfe7ff700 1 --2- 172.21.15.2:0/3143195455 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8df81014b0 0x7f8df8101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.860+0000 7f8dfe7ff700 1 -- 172.21.15.2:0/3143195455 >> 172.21.15.2:0/3143195455 conn(0x7f8df80fb8a0 msgr2=0x7f8df80fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:50.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.860+0000 7f8dfe7ff700 1 -- 172.21.15.2:0/3143195455 shutdown_connections 2023-11-16T16:46:50.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.860+0000 7f8dfe7ff700 1 -- 172.21.15.2:0/3143195455 wait complete. 2023-11-16T16:46:50.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.861+0000 7f8dfe7ff700 1 Processor -- start 2023-11-16T16:46:50.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.861+0000 7f8dfe7ff700 1 -- start start 2023-11-16T16:46:50.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.861+0000 7f8dfe7ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8df81002c0 0x7f8df80748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:50.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.861+0000 7f8dfe7ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8df81014b0 0x7f8df8072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:50.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.861+0000 7f8dfe7ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8df8101e40 0x7f8df8073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:50.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.861+0000 7f8dfe7ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df8073bc0 con 0x7f8df8101e40 2023-11-16T16:46:50.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.861+0000 7f8dfe7ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df8073d30 con 0x7f8df81002c0 2023-11-16T16:46:50.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.861+0000 7f8dfe7ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df81a0070 con 0x7f8df81014b0 2023-11-16T16:46:50.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.862+0000 7f8df7fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8df81002c0 0x7f8df80748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:50.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.862+0000 7f8df7fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8df81002c0 0x7f8df80748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:60114/0 (socket says 172.21.15.2:60114) 2023-11-16T16:46:50.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.862+0000 7f8df7fff700 1 -- 172.21.15.2:0/2069330556 learned_addr learned my addr 172.21.15.2:0/2069330556 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:46:50.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.862+0000 7f8dfcd9c700 1 --2- 172.21.15.2:0/2069330556 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8df8101e40 0x7f8df8073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:50.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.862+0000 7f8df7fff700 1 -- 172.21.15.2:0/2069330556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8df81014b0 msgr2=0x7f8df8072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:50.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.862+0000 7f8df7fff700 1 --2- 172.21.15.2:0/2069330556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8df81014b0 0x7f8df8072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.862+0000 7f8df7fff700 1 -- 172.21.15.2:0/2069330556 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8df8101e40 msgr2=0x7f8df8073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:50.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.862+0000 7f8df7fff700 1 --2- 172.21.15.2:0/2069330556 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8df8101e40 0x7f8df8073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:50.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.862+0000 7f8df7fff700 1 -- 172.21.15.2:0/2069330556 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de80095f0 con 0x7f8df81002c0 2023-11-16T16:46:50.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.862+0000 7f8dfcd9c700 1 --2- 172.21.15.2:0/2069330556 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8df8101e40 0x7f8df8073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:46:50.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.862+0000 7f8df7fff700 1 --2- 172.21.15.2:0/2069330556 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8df81002c0 0x7f8df80748a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8de800b680 tx=0x7f8de8005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:50.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.863+0000 7f8df57fa700 1 -- 172.21.15.2:0/2069330556 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de801d070 con 0x7f8df81002c0 2023-11-16T16:46:50.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.863+0000 7f8dfe7ff700 1 -- 172.21.15.2:0/2069330556 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df810bc10 con 0x7f8df81002c0 2023-11-16T16:46:50.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.863+0000 7f8df57fa700 1 -- 172.21.15.2:0/2069330556 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8de8021410 con 0x7f8df81002c0 2023-11-16T16:46:50.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.863+0000 7f8df57fa700 1 -- 172.21.15.2:0/2069330556 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de8017650 con 0x7f8df81002c0 2023-11-16T16:46:50.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.863+0000 7f8dfe7ff700 1 -- 172.21.15.2:0/2069330556 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df810ba80 con 0x7f8df81002c0 2023-11-16T16:46:50.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.864+0000 7f8dfe7ff700 1 -- 172.21.15.2:0/2069330556 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df81a0670 con 0x7f8df81002c0 2023-11-16T16:46:50.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.866+0000 7f8df57fa700 1 -- 172.21.15.2:0/2069330556 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f8de8021580 con 0x7f8df81002c0 2023-11-16T16:46:50.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:50 smithi155 ceph-mon[131607]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi141=b;smithi141=e;smithi141=h;smithi141=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:46:50.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:50 smithi155 ceph-mon[131607]: Saving service mds.all spec with placement smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi141=b;smithi141=e;smithi141=h;smithi141=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2023-11-16T16:46:50.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:50.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:50.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:50 smithi155 ceph-mon[131607]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:50.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:50.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:50.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:50.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:50.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:50.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:50 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:50.869 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.867+0000 7f8df57fa700 1 --2- 172.21.15.2:0/2069330556 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8de00a1150 0x7f8de00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:50.869 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.867+0000 7f8df57fa700 1 -- 172.21.15.2:0/2069330556 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8de80cda00 con 0x7f8df81002c0 2023-11-16T16:46:50.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.867+0000 7f8df77fe700 1 --2- 172.21.15.2:0/2069330556 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8de00a1150 0x7f8de00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:50.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.868+0000 7f8df77fe700 1 --2- 172.21.15.2:0/2069330556 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8de00a1150 0x7f8de00a3600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8dec005d90 tx=0x7f8dec005d00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:50.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:50.871+0000 7f8df57fa700 1 -- 172.21.15.2:0/2069330556 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8de80975e0 con 0x7f8df81002c0 2023-11-16T16:46:50.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:50 smithi002 ceph-mon[128712]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi141=b;smithi141=e;smithi141=h;smithi141=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:46:50.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:50 smithi002 ceph-mon[128712]: Saving service mds.all spec with placement smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi141=b;smithi141=e;smithi141=h;smithi141=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2023-11-16T16:46:50.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:50.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:50.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:50 smithi002 ceph-mon[128712]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:50.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:50.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:50.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:50.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:50.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:50.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:50 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:51.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:50 smithi141 ceph-mon[131184]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi141=b;smithi141=e;smithi141=h;smithi141=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:46:51.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:50 smithi141 ceph-mon[131184]: Saving service mds.all spec with placement smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi141=b;smithi141=e;smithi141=h;smithi141=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2023-11-16T16:46:51.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:51.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:46:51.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:50 smithi141 ceph-mon[131184]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:51.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:51.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:46:51.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:51.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:51.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:51.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:50 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:51.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.082+0000 7f8dfe7ff700 1 -- 172.21.15.2:0/2069330556 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8df81071f0 con 0x7f8df81002c0 2023-11-16T16:46:51.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.083+0000 7f8df57fa700 1 -- 172.21.15.2:0/2069330556 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8de809ac00 con 0x7f8df81002c0 2023-11-16T16:46:51.085 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T16:46:51.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.085+0000 7f8dfe7ff700 1 -- 172.21.15.2:0/2069330556 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8de00a1150 msgr2=0x7f8de00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:51.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.085+0000 7f8dfe7ff700 1 --2- 172.21.15.2:0/2069330556 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8de00a1150 0x7f8de00a3600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8dec005d90 tx=0x7f8dec005d00 comp rx=0 tx=0).stop 2023-11-16T16:46:51.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.085+0000 7f8dfe7ff700 1 -- 172.21.15.2:0/2069330556 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8df81002c0 msgr2=0x7f8df80748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:51.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.085+0000 7f8dfe7ff700 1 --2- 172.21.15.2:0/2069330556 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8df81002c0 0x7f8df80748a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8de800b680 tx=0x7f8de8005740 comp rx=0 tx=0).stop 2023-11-16T16:46:51.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.085+0000 7f8dfe7ff700 1 -- 172.21.15.2:0/2069330556 shutdown_connections 2023-11-16T16:46:51.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.085+0000 7f8dfe7ff700 1 --2- 172.21.15.2:0/2069330556 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8de00a1150 0x7f8de00a3600 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:51.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.085+0000 7f8dfe7ff700 1 --2- 172.21.15.2:0/2069330556 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8df8101e40 0x7f8df8073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:51.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.085+0000 7f8dfe7ff700 1 --2- 172.21.15.2:0/2069330556 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8df81002c0 0x7f8df80748a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:51.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.085+0000 7f8dfe7ff700 1 --2- 172.21.15.2:0/2069330556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8df81014b0 0x7f8df8072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:51.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.085+0000 7f8dfe7ff700 1 -- 172.21.15.2:0/2069330556 >> 172.21.15.2:0/2069330556 conn(0x7f8df80fb8a0 msgr2=0x7f8df80fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:51.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.085+0000 7f8dfe7ff700 1 -- 172.21.15.2:0/2069330556 shutdown_connections 2023-11-16T16:46:51.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.085+0000 7f8dfe7ff700 1 -- 172.21.15.2:0/2069330556 wait complete. 2023-11-16T16:46:51.106 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:46:51.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.308+0000 7f2a9b1ff700 1 -- 172.21.15.2:0/1577761663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a94101d30 msgr2=0x7f2a941085f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:51.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.308+0000 7f2a9b1ff700 1 --2- 172.21.15.2:0/1577761663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a94101d30 0x7f2a941085f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00b0a0 tx=0x7f2a8c00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:46:51.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.308+0000 7f2a9b1ff700 1 -- 172.21.15.2:0/1577761663 shutdown_connections 2023-11-16T16:46:51.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.308+0000 7f2a9b1ff700 1 --2- 172.21.15.2:0/1577761663 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a941001b0 0x7f2a941005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:51.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.308+0000 7f2a9b1ff700 1 --2- 172.21.15.2:0/1577761663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a94101d30 0x7f2a941085f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:51.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.308+0000 7f2a9b1ff700 1 --2- 172.21.15.2:0/1577761663 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a941013a0 0x7f2a941017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:51.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.308+0000 7f2a9b1ff700 1 -- 172.21.15.2:0/1577761663 >> 172.21.15.2:0/1577761663 conn(0x7f2a940fb790 msgr2=0x7f2a940fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:51.311 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.308+0000 7f2a9b1ff700 1 -- 172.21.15.2:0/1577761663 shutdown_connections 2023-11-16T16:46:51.311 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.308+0000 7f2a9b1ff700 1 -- 172.21.15.2:0/1577761663 wait complete. 2023-11-16T16:46:51.312 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.309+0000 7f2a9b1ff700 1 Processor -- start 2023-11-16T16:46:51.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.309+0000 7f2a9b1ff700 1 -- start start 2023-11-16T16:46:51.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.309+0000 7f2a9b1ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a941001b0 0x7f2a941982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:51.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.309+0000 7f2a9b1ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a941013a0 0x7f2a94198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:51.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.310+0000 7f2a9b1ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a94101d30 0x7f2a9419f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:51.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.310+0000 7f2a9b1ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9419fe60 con 0x7f2a94101d30 2023-11-16T16:46:51.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.310+0000 7f2a9b1ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9419ffa0 con 0x7f2a941013a0 2023-11-16T16:46:51.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.310+0000 7f2a9b1ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a941a0130 con 0x7f2a941001b0 2023-11-16T16:46:51.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.310+0000 7f2a9979c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a94101d30 0x7f2a9419f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:51.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.310+0000 7f2a98f9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a941001b0 0x7f2a941982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:51.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.310+0000 7f2a9979c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a94101d30 0x7f2a9419f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36086/0 (socket says 172.21.15.2:36086) 2023-11-16T16:46:51.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.310+0000 7f2a9979c700 1 -- 172.21.15.2:0/4206002828 learned_addr learned my addr 172.21.15.2:0/4206002828 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:46:51.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.310+0000 7f2a93fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a941013a0 0x7f2a94198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:51.314 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.310+0000 7f2a9979c700 1 -- 172.21.15.2:0/4206002828 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a941001b0 msgr2=0x7f2a941982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:51.314 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.310+0000 7f2a9979c700 1 --2- 172.21.15.2:0/4206002828 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a941001b0 0x7f2a941982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:51.314 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.310+0000 7f2a9979c700 1 -- 172.21.15.2:0/4206002828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a941013a0 msgr2=0x7f2a94198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:51.314 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.310+0000 7f2a9979c700 1 --2- 172.21.15.2:0/4206002828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a941013a0 0x7f2a94198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:51.314 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.310+0000 7f2a9979c700 1 -- 172.21.15.2:0/4206002828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c009d00 con 0x7f2a94101d30 2023-11-16T16:46:51.314 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.310+0000 7f2a9979c700 1 --2- 172.21.15.2:0/4206002828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a94101d30 0x7f2a9419f890 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00d010 tx=0x7f2a8c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:51.314 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.311+0000 7f2a91ffb700 1 -- 172.21.15.2:0/4206002828 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c00e040 con 0x7f2a94101d30 2023-11-16T16:46:51.314 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.311+0000 7f2a91ffb700 1 -- 172.21.15.2:0/4206002828 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a8c00bdb0 con 0x7f2a94101d30 2023-11-16T16:46:51.314 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.311+0000 7f2a91ffb700 1 -- 172.21.15.2:0/4206002828 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c005260 con 0x7f2a94101d30 2023-11-16T16:46:51.314 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.311+0000 7f2a9b1ff700 1 -- 172.21.15.2:0/4206002828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a9410bb70 con 0x7f2a94101d30 2023-11-16T16:46:51.314 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.311+0000 7f2a9b1ff700 1 -- 172.21.15.2:0/4206002828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a9410b9e0 con 0x7f2a94101d30 2023-11-16T16:46:51.315 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.312+0000 7f2a9b1ff700 1 -- 172.21.15.2:0/4206002828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a941a0680 con 0x7f2a94101d30 2023-11-16T16:46:51.315 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.314+0000 7f2a91ffb700 1 -- 172.21.15.2:0/4206002828 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f2a8c01d480 con 0x7f2a94101d30 2023-11-16T16:46:51.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.315+0000 7f2a91ffb700 1 --2- 172.21.15.2:0/4206002828 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2a840a11d0 0x7f2a840a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:51.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.315+0000 7f2a91ffb700 1 -- 172.21.15.2:0/4206002828 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2a8c0cd820 con 0x7f2a94101d30 2023-11-16T16:46:51.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.315+0000 7f2a98f9b700 1 --2- 172.21.15.2:0/4206002828 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2a840a11d0 0x7f2a840a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:51.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.316+0000 7f2a98f9b700 1 --2- 172.21.15.2:0/4206002828 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2a840a11d0 0x7f2a840a3680 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2a94101200 tx=0x7f2a88009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:51.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.319+0000 7f2a91ffb700 1 -- 172.21.15.2:0/4206002828 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2a8c097400 con 0x7f2a94101d30 2023-11-16T16:46:51.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.576+0000 7f2a9b1ff700 1 -- 172.21.15.2:0/4206002828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a941a0c30 con 0x7f2a94101d30 2023-11-16T16:46:51.578 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.576+0000 7f2a91ffb700 1 -- 172.21.15.2:0/4206002828 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f2a8c09aa20 con 0x7f2a94101d30 2023-11-16T16:46:51.578 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:46:51.578 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-11-16T16:46:51.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.578+0000 7f2a9b1ff700 1 -- 172.21.15.2:0/4206002828 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2a840a11d0 msgr2=0x7f2a840a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:51.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.578+0000 7f2a9b1ff700 1 --2- 172.21.15.2:0/4206002828 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2a840a11d0 0x7f2a840a3680 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2a94101200 tx=0x7f2a88009260 comp rx=0 tx=0).stop 2023-11-16T16:46:51.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.578+0000 7f2a9b1ff700 1 -- 172.21.15.2:0/4206002828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a94101d30 msgr2=0x7f2a9419f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:51.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.578+0000 7f2a9b1ff700 1 --2- 172.21.15.2:0/4206002828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a94101d30 0x7f2a9419f890 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00d010 tx=0x7f2a8c00bc80 comp rx=0 tx=0).stop 2023-11-16T16:46:51.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.579+0000 7f2a9b1ff700 1 -- 172.21.15.2:0/4206002828 shutdown_connections 2023-11-16T16:46:51.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.579+0000 7f2a9b1ff700 1 --2- 172.21.15.2:0/4206002828 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2a840a11d0 0x7f2a840a3680 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:51.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.579+0000 7f2a9b1ff700 1 --2- 172.21.15.2:0/4206002828 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a941001b0 0x7f2a941982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:51.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.579+0000 7f2a9b1ff700 1 --2- 172.21.15.2:0/4206002828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a94101d30 0x7f2a9419f890 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:51.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.579+0000 7f2a9b1ff700 1 --2- 172.21.15.2:0/4206002828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a941013a0 0x7f2a94198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:51.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.579+0000 7f2a9b1ff700 1 -- 172.21.15.2:0/4206002828 >> 172.21.15.2:0/4206002828 conn(0x7f2a940fb790 msgr2=0x7f2a94106c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:51.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.579+0000 7f2a9b1ff700 1 -- 172.21.15.2:0/4206002828 shutdown_connections 2023-11-16T16:46:51.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:51.579+0000 7f2a9b1ff700 1 -- 172.21.15.2:0/4206002828 wait complete. 2023-11-16T16:46:51.582 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 1 2023-11-16T16:46:51.870 INFO:journalctl@ceph.mds.l.smithi155.stdout:Nov 16 16:46:51 smithi155 systemd[1]: Starting Ceph mds.l for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:46:51.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:51 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:46:51.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:51 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:46:51.871 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:51 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:46:51.871 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:51 smithi155 ceph-mon[131607]: Service mds.all has no related daemon already placed. 2023-11-16T16:46:51.871 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:51 smithi155 ceph-mon[131607]: Deploying daemon mds.l on smithi155 2023-11-16T16:46:51.871 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:51 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3951837027' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:46:51.871 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:51 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2069330556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:46:51.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:51 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:46:51.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:51 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:46:51.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:51 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:46:51.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:51 smithi002 ceph-mon[128712]: Service mds.all has no related daemon already placed. 2023-11-16T16:46:51.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:51 smithi002 ceph-mon[128712]: Deploying daemon mds.l on smithi155 2023-11-16T16:46:51.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:51 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3951837027' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:46:51.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:51 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2069330556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:46:52.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:51 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:46:52.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:51 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:46:52.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:51 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:46:52.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:51 smithi141 ceph-mon[131184]: Service mds.all has no related daemon already placed. 2023-11-16T16:46:52.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:51 smithi141 ceph-mon[131184]: Deploying daemon mds.l on smithi155 2023-11-16T16:46:52.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:51 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3951837027' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:46:52.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:51 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2069330556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:46:52.206 INFO:journalctl@ceph.mds.l.smithi155.stdout:Nov 16 16:46:51 smithi155 podman[156616]: 2023-11-16T16:46:52.206 INFO:journalctl@ceph.mds.l.smithi155.stdout:Nov 16 16:46:52 smithi155 ceph-mds[156656]: -- 172.21.15.155:0/388581914 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fc80ff9520 con 0x55fc80fce800 2023-11-16T16:46:52.206 INFO:journalctl@ceph.mds.l.smithi155.stdout:Nov 16 16:46:52 smithi155 bash[156616]: 9aaccb5bb9bfec8a90c39e0e9d2584f8f9a750acdc9fe98dd1f9951e0991c258 2023-11-16T16:46:52.206 INFO:journalctl@ceph.mds.l.smithi155.stdout:Nov 16 16:46:52 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-l[156632]: starting mds.l at 2023-11-16T16:46:52.562 INFO:journalctl@ceph.mds.l.smithi155.stdout:Nov 16 16:46:52 smithi155 systemd[1]: Started Ceph mds.l for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:46:52.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:52 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/4206002828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:52.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:52 smithi155 ceph-mon[131607]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:52.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:52 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.155:6832/4280048431,v1:172.21.15.155:6833/4280048431] up:boot 2023-11-16T16:46:52.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:52 smithi155 ceph-mon[131607]: fsmap 1 up:standby 2023-11-16T16:46:52.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T16:46:52.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:52.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:52.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:52.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:52.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:52.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:52 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:52.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:52 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/4206002828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:52.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:52 smithi002 ceph-mon[128712]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:52.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:52 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.155:6832/4280048431,v1:172.21.15.155:6833/4280048431] up:boot 2023-11-16T16:46:52.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:52 smithi002 ceph-mon[128712]: fsmap 1 up:standby 2023-11-16T16:46:52.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T16:46:52.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:52.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:52.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:52.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:52.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:52.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:52 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:53.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:52 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/4206002828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:53.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:52 smithi141 ceph-mon[131184]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:53.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:52 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.155:6832/4280048431,v1:172.21.15.155:6833/4280048431] up:boot 2023-11-16T16:46:53.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:52 smithi141 ceph-mon[131184]: fsmap 1 up:standby 2023-11-16T16:46:53.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T16:46:53.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:53.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:53.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:53.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:53.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:53.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:52 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:53.600 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:46:53.665 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:53 smithi141 ceph-mon[131184]: Deploying daemon mds.k on smithi141 2023-11-16T16:46:53.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.814+0000 7fb8e75ff700 1 -- 172.21.15.2:0/2256570950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8e010c860 msgr2=0x7fb8e010ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:53.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.814+0000 7fb8e75ff700 1 --2- 172.21.15.2:0/2256570950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8e010c860 0x7fb8e010ec40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800b0a0 tx=0x7fb8d800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:46:53.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.814+0000 7fb8e75ff700 1 -- 172.21.15.2:0/2256570950 shutdown_connections 2023-11-16T16:46:53.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.814+0000 7fb8e75ff700 1 --2- 172.21.15.2:0/2256570950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8e010c860 0x7fb8e010ec40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:53.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.814+0000 7fb8e75ff700 1 --2- 172.21.15.2:0/2256570950 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb8e0074b30 0x7fb8e0072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:53.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.814+0000 7fb8e75ff700 1 --2- 172.21.15.2:0/2256570950 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8e0073550 0x7fb8e00739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:53.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.814+0000 7fb8e75ff700 1 -- 172.21.15.2:0/2256570950 >> 172.21.15.2:0/2256570950 conn(0x7fb8e00fb870 msgr2=0x7fb8e00fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:53.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.814+0000 7fb8e75ff700 1 -- 172.21.15.2:0/2256570950 shutdown_connections 2023-11-16T16:46:53.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.814+0000 7fb8e75ff700 1 -- 172.21.15.2:0/2256570950 wait complete. 2023-11-16T16:46:53.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.815+0000 7fb8e75ff700 1 Processor -- start 2023-11-16T16:46:53.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.815+0000 7fb8e75ff700 1 -- start start 2023-11-16T16:46:53.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.816+0000 7fb8e75ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8e0073550 0x7fb8e019c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:53.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.816+0000 7fb8e75ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8e0074b30 0x7fb8e019cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:53.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.816+0000 7fb8e75ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb8e010c860 0x7fb8e01a3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:53.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.816+0000 7fb8e75ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e01a42b0 con 0x7fb8e0073550 2023-11-16T16:46:53.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.816+0000 7fb8e75ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e01a43f0 con 0x7fb8e010c860 2023-11-16T16:46:53.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.816+0000 7fb8e75ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e01a4580 con 0x7fb8e0074b30 2023-11-16T16:46:53.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.816+0000 7fb8e539b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8e0073550 0x7fb8e019c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:53.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.816+0000 7fb8e539b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8e0073550 0x7fb8e019c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36104/0 (socket says 172.21.15.2:36104) 2023-11-16T16:46:53.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.816+0000 7fb8e539b700 1 -- 172.21.15.2:0/2849031302 learned_addr learned my addr 172.21.15.2:0/2849031302 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:46:53.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.816+0000 7fb8e5b9c700 1 --2- 172.21.15.2:0/2849031302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb8e010c860 0x7fb8e01a3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:53.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.816+0000 7fb8e4b9a700 1 --2- 172.21.15.2:0/2849031302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8e0074b30 0x7fb8e019cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:53.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.816+0000 7fb8e539b700 1 -- 172.21.15.2:0/2849031302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8e0074b30 msgr2=0x7fb8e019cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:53.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.816+0000 7fb8e539b700 1 --2- 172.21.15.2:0/2849031302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8e0074b30 0x7fb8e019cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:53.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.816+0000 7fb8e539b700 1 -- 172.21.15.2:0/2849031302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb8e010c860 msgr2=0x7fb8e01a3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:53.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.816+0000 7fb8e539b700 1 --2- 172.21.15.2:0/2849031302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb8e010c860 0x7fb8e01a3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:53.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.816+0000 7fb8e539b700 1 -- 172.21.15.2:0/2849031302 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d8009d00 con 0x7fb8e0073550 2023-11-16T16:46:53.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.816+0000 7fb8e4b9a700 1 --2- 172.21.15.2:0/2849031302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8e0074b30 0x7fb8e019cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:46:53.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.817+0000 7fb8e539b700 1 --2- 172.21.15.2:0/2849031302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8e0073550 0x7fb8e019c730 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc00b500 tx=0x7fb8dc00b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:53.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.817+0000 7fb8d67fc700 1 -- 172.21.15.2:0/2849031302 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc011830 con 0x7fb8e0073550 2023-11-16T16:46:53.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.817+0000 7fb8d67fc700 1 -- 172.21.15.2:0/2849031302 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb8dc011e50 con 0x7fb8e0073550 2023-11-16T16:46:53.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.817+0000 7fb8e75ff700 1 -- 172.21.15.2:0/2849031302 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e01122f0 con 0x7fb8e0073550 2023-11-16T16:46:53.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.817+0000 7fb8d67fc700 1 -- 172.21.15.2:0/2849031302 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc00f2e0 con 0x7fb8e0073550 2023-11-16T16:46:53.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.817+0000 7fb8e75ff700 1 -- 172.21.15.2:0/2849031302 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e0112160 con 0x7fb8e0073550 2023-11-16T16:46:53.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.818+0000 7fb8e75ff700 1 -- 172.21.15.2:0/2849031302 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e01023b0 con 0x7fb8e0073550 2023-11-16T16:46:53.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.820+0000 7fb8d67fc700 1 -- 172.21.15.2:0/2849031302 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fb8dc0105a0 con 0x7fb8e0073550 2023-11-16T16:46:53.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.821+0000 7fb8d67fc700 1 --2- 172.21.15.2:0/2849031302 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb8cc0a1150 0x7fb8cc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:53.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.821+0000 7fb8d67fc700 1 -- 172.21.15.2:0/2849031302 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb8dc0cbf20 con 0x7fb8e0073550 2023-11-16T16:46:53.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.822+0000 7fb8e4b9a700 1 --2- 172.21.15.2:0/2849031302 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb8cc0a1150 0x7fb8cc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:53.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.822+0000 7fb8e4b9a700 1 --2- 172.21.15.2:0/2849031302 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb8cc0a1150 0x7fb8cc0a3600 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0005950 tx=0x7fb8d00058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:53.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:53.825+0000 7fb8d67fc700 1 -- 172.21.15.2:0/2849031302 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb8dc095a70 con 0x7fb8e0073550 2023-11-16T16:46:53.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:53 smithi155 ceph-mon[131607]: Deploying daemon mds.k on smithi141 2023-11-16T16:46:53.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:53 smithi002 ceph-mon[128712]: Deploying daemon mds.k on smithi141 2023-11-16T16:46:54.015 INFO:journalctl@ceph.mds.k.smithi141.stdout:Nov 16 16:46:53 smithi141 systemd[1]: Starting Ceph mds.k for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:46:54.083 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:54.082+0000 7fb8e75ff700 1 -- 172.21.15.2:0/2849031302 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8e002cfe0 con 0x7fb8e0073550 2023-11-16T16:46:54.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:54.082+0000 7fb8d67fc700 1 -- 172.21.15.2:0/2849031302 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7fb8dc099090 con 0x7fb8e0073550 2023-11-16T16:46:54.084 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:46:54.084 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-11-16T16:46:54.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:54.085+0000 7fb8e75ff700 1 -- 172.21.15.2:0/2849031302 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb8cc0a1150 msgr2=0x7fb8cc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:54.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:54.085+0000 7fb8e75ff700 1 --2- 172.21.15.2:0/2849031302 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb8cc0a1150 0x7fb8cc0a3600 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0005950 tx=0x7fb8d00058e0 comp rx=0 tx=0).stop 2023-11-16T16:46:54.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:54.085+0000 7fb8e75ff700 1 -- 172.21.15.2:0/2849031302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8e0073550 msgr2=0x7fb8e019c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:54.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:54.085+0000 7fb8e75ff700 1 --2- 172.21.15.2:0/2849031302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8e0073550 0x7fb8e019c730 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc00b500 tx=0x7fb8dc00b810 comp rx=0 tx=0).stop 2023-11-16T16:46:54.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:54.085+0000 7fb8e75ff700 1 -- 172.21.15.2:0/2849031302 shutdown_connections 2023-11-16T16:46:54.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:54.085+0000 7fb8e75ff700 1 --2- 172.21.15.2:0/2849031302 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb8cc0a1150 0x7fb8cc0a3600 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:54.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:54.085+0000 7fb8e75ff700 1 --2- 172.21.15.2:0/2849031302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb8e010c860 0x7fb8e01a3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:54.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:54.085+0000 7fb8e75ff700 1 --2- 172.21.15.2:0/2849031302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8e0073550 0x7fb8e019c730 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:54.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:54.085+0000 7fb8e75ff700 1 --2- 172.21.15.2:0/2849031302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8e0074b30 0x7fb8e019cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:54.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:54.085+0000 7fb8e75ff700 1 -- 172.21.15.2:0/2849031302 >> 172.21.15.2:0/2849031302 conn(0x7fb8e00fb870 msgr2=0x7fb8e00fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:54.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:54.085+0000 7fb8e75ff700 1 -- 172.21.15.2:0/2849031302 shutdown_connections 2023-11-16T16:46:54.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:54.085+0000 7fb8e75ff700 1 -- 172.21.15.2:0/2849031302 wait complete. 2023-11-16T16:46:54.088 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 2 2023-11-16T16:46:54.485 INFO:journalctl@ceph.mds.k.smithi141.stdout:Nov 16 16:46:54 smithi141 podman[155379]: 2023-11-16T16:46:54.486 INFO:journalctl@ceph.mds.k.smithi141.stdout:Nov 16 16:46:54 smithi141 ceph-mds[155419]: -- 172.21.15.141:0/1637029851 <== mon.2 v2:172.21.15.141:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5580aaf03520 con 0x5580aaed9000 2023-11-16T16:46:54.486 INFO:journalctl@ceph.mds.k.smithi141.stdout:Nov 16 16:46:54 smithi141 bash[155379]: 4b16fdad147226f583253907dface7623f6bda13920d26f5c2af3d2a1ca084f7 2023-11-16T16:46:54.486 INFO:journalctl@ceph.mds.k.smithi141.stdout:Nov 16 16:46:54 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-k[155395]: starting mds.k at 2023-11-16T16:46:54.764 INFO:journalctl@ceph.mds.k.smithi141.stdout:Nov 16 16:46:54 smithi141 systemd[1]: Started Ceph mds.k for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:46:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:54 smithi141 ceph-mon[131184]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:54 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2849031302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:54 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:55.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:54 smithi155 ceph-mon[131607]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:55.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:54 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2849031302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:55.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:55.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:55.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:55.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:55.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:55.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:54 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:55.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:54 smithi002 ceph-mon[128712]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:55.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:54 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2849031302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:55.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:55.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:55.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:55.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:55.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:55.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:54 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:56.106 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:46:56.170 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:55 smithi002 ceph-mon[128712]: Deploying daemon mds.j on smithi002 2023-11-16T16:46:56.171 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:55 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.141:6832/3738610702,v1:172.21.15.141:6833/3738610702] up:boot 2023-11-16T16:46:56.171 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:55 smithi002 ceph-mon[128712]: fsmap 2 up:standby 2023-11-16T16:46:56.171 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:55 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T16:46:56.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:55 smithi141 ceph-mon[131184]: Deploying daemon mds.j on smithi002 2023-11-16T16:46:56.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:55 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.141:6832/3738610702,v1:172.21.15.141:6833/3738610702] up:boot 2023-11-16T16:46:56.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:55 smithi141 ceph-mon[131184]: fsmap 2 up:standby 2023-11-16T16:46:56.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:55 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T16:46:56.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.340+0000 7f08d7dff700 1 -- 172.21.15.2:0/2315439545 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d0073550 msgr2=0x7f08d00739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:56.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.340+0000 7f08d7dff700 1 --2- 172.21.15.2:0/2315439545 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d0073550 0x7f08d00739c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f08c0009990 tx=0x7f08c0009ca0 comp rx=0 tx=0).stop 2023-11-16T16:46:56.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.340+0000 7f08d7dff700 1 -- 172.21.15.2:0/2315439545 shutdown_connections 2023-11-16T16:46:56.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.340+0000 7f08d7dff700 1 --2- 172.21.15.2:0/2315439545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08d010c860 0x7f08d010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:56.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.340+0000 7f08d7dff700 1 --2- 172.21.15.2:0/2315439545 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08d0074b30 0x7f08d0072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:56.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.340+0000 7f08d7dff700 1 --2- 172.21.15.2:0/2315439545 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d0073550 0x7f08d00739c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:56.342 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.340+0000 7f08d7dff700 1 -- 172.21.15.2:0/2315439545 >> 172.21.15.2:0/2315439545 conn(0x7f08d00fb870 msgr2=0x7f08d00fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:56.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.340+0000 7f08d7dff700 1 -- 172.21.15.2:0/2315439545 shutdown_connections 2023-11-16T16:46:56.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.340+0000 7f08d7dff700 1 -- 172.21.15.2:0/2315439545 wait complete. 2023-11-16T16:46:56.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.341+0000 7f08d7dff700 1 Processor -- start 2023-11-16T16:46:56.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.342+0000 7f08d7dff700 1 -- start start 2023-11-16T16:46:56.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.342+0000 7f08d7dff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08d0073550 0x7f08d019c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:56.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.342+0000 7f08d7dff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08d0074b30 0x7f08d019cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:56.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.342+0000 7f08d7dff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d010c860 0x7f08d01a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:56.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.342+0000 7f08d7dff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d01a4170 con 0x7f08d0074b30 2023-11-16T16:46:56.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.342+0000 7f08d7dff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d01a42b0 con 0x7f08d0073550 2023-11-16T16:46:56.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.342+0000 7f08d7dff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d01a4440 con 0x7f08d010c860 2023-11-16T16:46:56.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.342+0000 7f08d539a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08d0074b30 0x7f08d019cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:56.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.342+0000 7f08d539a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08d0074b30 0x7f08d019cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35730/0 (socket says 172.21.15.2:35730) 2023-11-16T16:46:56.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.342+0000 7f08d539a700 1 -- 172.21.15.2:0/699647501 learned_addr learned my addr 172.21.15.2:0/699647501 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:46:56.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.342+0000 7f08d5b9b700 1 --2- 172.21.15.2:0/699647501 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08d0073550 0x7f08d019c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:56.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.342+0000 7f08d639c700 1 --2- 172.21.15.2:0/699647501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d010c860 0x7f08d01a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:56.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.343+0000 7f08d539a700 1 -- 172.21.15.2:0/699647501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d010c860 msgr2=0x7f08d01a3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:56.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.343+0000 7f08d539a700 1 --2- 172.21.15.2:0/699647501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d010c860 0x7f08d01a3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:56.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.343+0000 7f08d539a700 1 -- 172.21.15.2:0/699647501 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08d0073550 msgr2=0x7f08d019c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:56.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.343+0000 7f08d539a700 1 --2- 172.21.15.2:0/699647501 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08d0073550 0x7f08d019c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:56.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.343+0000 7f08d539a700 1 -- 172.21.15.2:0/699647501 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c8009c40 con 0x7f08d0074b30 2023-11-16T16:46:56.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.343+0000 7f08d5b9b700 1 --2- 172.21.15.2:0/699647501 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08d0073550 0x7f08d019c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:46:56.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.343+0000 7f08d539a700 1 --2- 172.21.15.2:0/699647501 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08d0074b30 0x7f08d019cb30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f08c0004710 tx=0x7f08c0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:56.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.343+0000 7f08c6ffd700 1 -- 172.21.15.2:0/699647501 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c001d070 con 0x7f08d0074b30 2023-11-16T16:46:56.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.343+0000 7f08d7dff700 1 -- 172.21.15.2:0/699647501 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08c00095f0 con 0x7f08d0074b30 2023-11-16T16:46:56.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.343+0000 7f08c6ffd700 1 -- 172.21.15.2:0/699647501 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08c0021410 con 0x7f08d0074b30 2023-11-16T16:46:56.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.343+0000 7f08c6ffd700 1 -- 172.21.15.2:0/699647501 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c0017650 con 0x7f08d0074b30 2023-11-16T16:46:56.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.343+0000 7f08d7dff700 1 -- 172.21.15.2:0/699647501 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d0112270 con 0x7f08d0074b30 2023-11-16T16:46:56.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.344+0000 7f08d7dff700 1 -- 172.21.15.2:0/699647501 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d01023b0 con 0x7f08d0074b30 2023-11-16T16:46:56.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.347+0000 7f08c6ffd700 1 -- 172.21.15.2:0/699647501 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f08c0021580 con 0x7f08d0074b30 2023-11-16T16:46:56.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.348+0000 7f08c6ffd700 1 --2- 172.21.15.2:0/699647501 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08bc0a1150 0x7f08bc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:56.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.348+0000 7f08c6ffd700 1 -- 172.21.15.2:0/699647501 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f08c00cdfc0 con 0x7f08d0074b30 2023-11-16T16:46:56.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.348+0000 7f08d5b9b700 1 --2- 172.21.15.2:0/699647501 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08bc0a1150 0x7f08bc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:56.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.349+0000 7f08d5b9b700 1 --2- 172.21.15.2:0/699647501 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08bc0a1150 0x7f08bc0a3600 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f08d0074f00 tx=0x7f08cc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:56.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.351+0000 7f08c6ffd700 1 -- 172.21.15.2:0/699647501 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f08c0097ba0 con 0x7f08d0074b30 2023-11-16T16:46:56.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:55 smithi155 ceph-mon[131607]: Deploying daemon mds.j on smithi002 2023-11-16T16:46:56.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:55 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.141:6832/3738610702,v1:172.21.15.141:6833/3738610702] up:boot 2023-11-16T16:46:56.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:55 smithi155 ceph-mon[131607]: fsmap 2 up:standby 2023-11-16T16:46:56.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:55 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T16:46:56.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.636+0000 7f08d7dff700 1 -- 172.21.15.2:0/699647501 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08d0067a60 con 0x7f08d0074b30 2023-11-16T16:46:56.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.637+0000 7f08c6ffd700 1 -- 172.21.15.2:0/699647501 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2020 (secure 0 0 0) 0x7f08c009b1c0 con 0x7f08d0074b30 2023-11-16T16:46:56.638 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:46:56.639 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-11-16T16:46:56.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.639+0000 7f08d7dff700 1 -- 172.21.15.2:0/699647501 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08bc0a1150 msgr2=0x7f08bc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:56.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.639+0000 7f08d7dff700 1 --2- 172.21.15.2:0/699647501 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08bc0a1150 0x7f08bc0a3600 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f08d0074f00 tx=0x7f08cc009260 comp rx=0 tx=0).stop 2023-11-16T16:46:56.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.639+0000 7f08d7dff700 1 -- 172.21.15.2:0/699647501 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08d0074b30 msgr2=0x7f08d019cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:56.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.639+0000 7f08d7dff700 1 --2- 172.21.15.2:0/699647501 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08d0074b30 0x7f08d019cb30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f08c0004710 tx=0x7f08c0009580 comp rx=0 tx=0).stop 2023-11-16T16:46:56.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.639+0000 7f08d7dff700 1 -- 172.21.15.2:0/699647501 shutdown_connections 2023-11-16T16:46:56.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.639+0000 7f08d7dff700 1 --2- 172.21.15.2:0/699647501 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08bc0a1150 0x7f08bc0a3600 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:56.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.639+0000 7f08d7dff700 1 --2- 172.21.15.2:0/699647501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d010c860 0x7f08d01a3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:56.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.639+0000 7f08d7dff700 1 --2- 172.21.15.2:0/699647501 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08d0073550 0x7f08d019c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:56.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.639+0000 7f08d7dff700 1 --2- 172.21.15.2:0/699647501 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08d0074b30 0x7f08d019cb30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:56.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.639+0000 7f08d7dff700 1 -- 172.21.15.2:0/699647501 >> 172.21.15.2:0/699647501 conn(0x7f08d00fb870 msgr2=0x7f08d00fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:56.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.639+0000 7f08d7dff700 1 -- 172.21.15.2:0/699647501 shutdown_connections 2023-11-16T16:46:56.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:56.639+0000 7f08d7dff700 1 -- 172.21.15.2:0/699647501 wait complete. 2023-11-16T16:46:56.642 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 3 2023-11-16T16:46:57.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:56 smithi002 ceph-mon[128712]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:57.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:56 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/699647501' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:57.156 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 16:46:56 smithi002 systemd[1]: Starting Ceph mds.j for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:46:57.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:56 smithi141 ceph-mon[131184]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:57.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:56 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/699647501' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:57.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:56 smithi155 ceph-mon[131607]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:57.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:56 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/699647501' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:46:57.577 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 16:46:57 smithi002 podman[161549]: 2023-11-16T16:46:57.577 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 16:46:57 smithi002 bash[161549]: cf1bb173fe07dfe34bcd297184b112f844d7c27be7d1aacb6585384f05739c44 2023-11-16T16:46:57.577 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 16:46:57 smithi002 ceph-mds[161590]: -- 172.21.15.2:0/3456425923 <== mon.2 v2:172.21.15.141:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bfe3077520 con 0x55bfe304ec00 2023-11-16T16:46:57.578 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 16:46:57 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-j[161565]: starting mds.j at 2023-11-16T16:46:57.905 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 16:46:57 smithi002 systemd[1]: Started Ceph mds.j for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:46:58.664 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:46:58.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.885+0000 7f3e40dff700 1 -- 172.21.15.2:0/2803096606 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e3c073550 msgr2=0x7f3e3c0739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:58.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.885+0000 7f3e40dff700 1 --2- 172.21.15.2:0/2803096606 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e3c073550 0x7f3e3c0739c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3e30009990 tx=0x7f3e30009ca0 comp rx=0 tx=0).stop 2023-11-16T16:46:58.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.885+0000 7f3e40dff700 1 -- 172.21.15.2:0/2803096606 shutdown_connections 2023-11-16T16:46:58.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.885+0000 7f3e40dff700 1 --2- 172.21.15.2:0/2803096606 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e3c074b30 0x7f3e3c072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:58.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.885+0000 7f3e40dff700 1 --2- 172.21.15.2:0/2803096606 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e3c10c860 0x7f3e3c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:58.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.885+0000 7f3e40dff700 1 --2- 172.21.15.2:0/2803096606 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e3c073550 0x7f3e3c0739c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:58.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.885+0000 7f3e40dff700 1 -- 172.21.15.2:0/2803096606 >> 172.21.15.2:0/2803096606 conn(0x7f3e3c0fb870 msgr2=0x7f3e3c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:58.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.885+0000 7f3e40dff700 1 -- 172.21.15.2:0/2803096606 shutdown_connections 2023-11-16T16:46:58.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.885+0000 7f3e40dff700 1 -- 172.21.15.2:0/2803096606 wait complete. 2023-11-16T16:46:58.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.886+0000 7f3e40dff700 1 Processor -- start 2023-11-16T16:46:58.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.887+0000 7f3e40dff700 1 -- start start 2023-11-16T16:46:58.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.887+0000 7f3e40dff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e3c073550 0x7f3e3c19c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:58.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.887+0000 7f3e40dff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e3c074b30 0x7f3e3c19cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:58.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.887+0000 7f3e40dff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e3c10c860 0x7f3e3c1a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:58.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.887+0000 7f3e40dff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c1a4170 con 0x7f3e3c074b30 2023-11-16T16:46:58.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.887+0000 7f3e40dff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c1a42b0 con 0x7f3e3c073550 2023-11-16T16:46:58.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.887+0000 7f3e40dff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c1a4440 con 0x7f3e3c10c860 2023-11-16T16:46:58.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.887+0000 7f3e39d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e3c074b30 0x7f3e3c19cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:58.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.887+0000 7f3e39d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e3c074b30 0x7f3e3c19cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35776/0 (socket says 172.21.15.2:35776) 2023-11-16T16:46:58.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.887+0000 7f3e39d9b700 1 -- 172.21.15.2:0/3980614660 learned_addr learned my addr 172.21.15.2:0/3980614660 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:46:58.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.887+0000 7f3e3a59c700 1 --2- 172.21.15.2:0/3980614660 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e3c073550 0x7f3e3c19c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:58.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.887+0000 7f3e3ad9d700 1 --2- 172.21.15.2:0/3980614660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e3c10c860 0x7f3e3c1a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:58.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.888+0000 7f3e39d9b700 1 -- 172.21.15.2:0/3980614660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e3c10c860 msgr2=0x7f3e3c1a3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:58.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.888+0000 7f3e39d9b700 1 --2- 172.21.15.2:0/3980614660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e3c10c860 0x7f3e3c1a3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:58.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.888+0000 7f3e39d9b700 1 -- 172.21.15.2:0/3980614660 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e3c073550 msgr2=0x7f3e3c19c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:58.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.888+0000 7f3e39d9b700 1 --2- 172.21.15.2:0/3980614660 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e3c073550 0x7f3e3c19c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:58.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.888+0000 7f3e39d9b700 1 -- 172.21.15.2:0/3980614660 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e300095f0 con 0x7f3e3c074b30 2023-11-16T16:46:58.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.888+0000 7f3e39d9b700 1 --2- 172.21.15.2:0/3980614660 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e3c074b30 0x7f3e3c19cb30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3e30004710 tx=0x7f3e30009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:58.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.888+0000 7f3e2b7fe700 1 -- 172.21.15.2:0/3980614660 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3001d070 con 0x7f3e3c074b30 2023-11-16T16:46:58.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.888+0000 7f3e2b7fe700 1 -- 172.21.15.2:0/3980614660 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e30021410 con 0x7f3e3c074b30 2023-11-16T16:46:58.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.888+0000 7f3e2b7fe700 1 -- 172.21.15.2:0/3980614660 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e30017650 con 0x7f3e3c074b30 2023-11-16T16:46:58.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.888+0000 7f3e40dff700 1 -- 172.21.15.2:0/3980614660 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3c112270 con 0x7f3e3c074b30 2023-11-16T16:46:58.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.888+0000 7f3e40dff700 1 -- 172.21.15.2:0/3980614660 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e3c1120e0 con 0x7f3e3c074b30 2023-11-16T16:46:58.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.890+0000 7f3e40dff700 1 -- 172.21.15.2:0/3980614660 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3c1023b0 con 0x7f3e3c074b30 2023-11-16T16:46:58.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.892+0000 7f3e2b7fe700 1 -- 172.21.15.2:0/3980614660 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f3e30021580 con 0x7f3e3c074b30 2023-11-16T16:46:58.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.894+0000 7f3e2b7fe700 1 --2- 172.21.15.2:0/3980614660 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3e240a11d0 0x7f3e240a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:46:58.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.895+0000 7f3e2b7fe700 1 -- 172.21.15.2:0/3980614660 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3e300cdf70 con 0x7f3e3c074b30 2023-11-16T16:46:58.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.895+0000 7f3e3a59c700 1 --2- 172.21.15.2:0/3980614660 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3e240a11d0 0x7f3e240a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:46:58.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.895+0000 7f3e3a59c700 1 --2- 172.21.15.2:0/3980614660 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3e240a11d0 0x7f3e240a3680 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c074f00 tx=0x7f3e2c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:46:58.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:58.897+0000 7f3e2b7fe700 1 -- 172.21.15.2:0/3980614660 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3e30097b50 con 0x7f3e3c074b30 2023-11-16T16:46:58.898 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:58 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:58.899 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:58 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:58.899 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:58 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:58.899 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:58 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:58.899 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:58 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:58.899 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:58 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:58.899 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:58 smithi002 ceph-mon[128712]: Deploying daemon mds.d on smithi002 2023-11-16T16:46:58.899 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:58 smithi002 ceph-mon[128712]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:58.900 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:58 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.2:6834/4133614271,v1:172.21.15.2:6835/4133614271] up:boot 2023-11-16T16:46:58.900 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:58 smithi002 ceph-mon[128712]: fsmap 3 up:standby 2023-11-16T16:46:58.900 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:58 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T16:46:59.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:58 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:59.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:58 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:59.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:58 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:59.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:58 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:59.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:58 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:59.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:58 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:59.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:58 smithi141 ceph-mon[131184]: Deploying daemon mds.d on smithi002 2023-11-16T16:46:59.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:58 smithi141 ceph-mon[131184]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:59.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:58 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.2:6834/4133614271,v1:172.21.15.2:6835/4133614271] up:boot 2023-11-16T16:46:59.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:58 smithi141 ceph-mon[131184]: fsmap 3 up:standby 2023-11-16T16:46:59.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:58 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T16:46:59.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:58 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:59.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:58 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:59.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:58 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:46:59.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:58 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:46:59.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:58 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:46:59.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:58 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:46:59.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:58 smithi155 ceph-mon[131607]: Deploying daemon mds.d on smithi002 2023-11-16T16:46:59.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:58 smithi155 ceph-mon[131607]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:46:59.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:58 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.2:6834/4133614271,v1:172.21.15.2:6835/4133614271] up:boot 2023-11-16T16:46:59.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:58 smithi155 ceph-mon[131607]: fsmap 3 up:standby 2023-11-16T16:46:59.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:58 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T16:46:59.172 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:59.171+0000 7f3e40dff700 1 -- 172.21.15.2:0/3980614660 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e3c02cfe0 con 0x7f3e3c074b30 2023-11-16T16:46:59.172 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:59.172+0000 7f3e2b7fe700 1 -- 172.21.15.2:0/3980614660 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2774 (secure 0 0 0) 0x7f3e3009b170 con 0x7f3e3c074b30 2023-11-16T16:46:59.173 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:46:59.173 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-11-16T16:46:59.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:59.174+0000 7f3e40dff700 1 -- 172.21.15.2:0/3980614660 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3e240a11d0 msgr2=0x7f3e240a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:59.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:59.174+0000 7f3e40dff700 1 --2- 172.21.15.2:0/3980614660 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3e240a11d0 0x7f3e240a3680 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c074f00 tx=0x7f3e2c009260 comp rx=0 tx=0).stop 2023-11-16T16:46:59.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:59.174+0000 7f3e40dff700 1 -- 172.21.15.2:0/3980614660 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e3c074b30 msgr2=0x7f3e3c19cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:46:59.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:59.174+0000 7f3e40dff700 1 --2- 172.21.15.2:0/3980614660 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e3c074b30 0x7f3e3c19cb30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3e30004710 tx=0x7f3e30009580 comp rx=0 tx=0).stop 2023-11-16T16:46:59.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:59.174+0000 7f3e40dff700 1 -- 172.21.15.2:0/3980614660 shutdown_connections 2023-11-16T16:46:59.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:59.174+0000 7f3e40dff700 1 --2- 172.21.15.2:0/3980614660 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3e240a11d0 0x7f3e240a3680 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:59.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:59.174+0000 7f3e40dff700 1 --2- 172.21.15.2:0/3980614660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3e3c10c860 0x7f3e3c1a3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:59.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:59.174+0000 7f3e40dff700 1 --2- 172.21.15.2:0/3980614660 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3e3c073550 0x7f3e3c19c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:59.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:59.174+0000 7f3e40dff700 1 --2- 172.21.15.2:0/3980614660 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e3c074b30 0x7f3e3c19cb30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:46:59.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:59.174+0000 7f3e40dff700 1 -- 172.21.15.2:0/3980614660 >> 172.21.15.2:0/3980614660 conn(0x7f3e3c0fb870 msgr2=0x7f3e3c0fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:46:59.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:59.175+0000 7f3e40dff700 1 -- 172.21.15.2:0/3980614660 shutdown_connections 2023-11-16T16:46:59.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:46:59.175+0000 7f3e40dff700 1 -- 172.21.15.2:0/3980614660 wait complete. 2023-11-16T16:46:59.177 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 4 2023-11-16T16:47:00.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:46:59 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3980614660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:00.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:46:59 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3980614660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:00.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:46:59 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3980614660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:00.406 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 16:46:59 smithi002 systemd[1]: Starting Ceph mds.d for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:47:00.709 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 16:47:00 smithi002 podman[161940]: 2023-11-16T16:47:00.996 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 16:47:00 smithi002 bash[161940]: 915282f55dd1d0298fd2f194553661b37ea22fe32585d1f4d0114872929cf8b8 2023-11-16T16:47:00.996 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 16:47:00 smithi002 ceph-mds[161980]: -- 172.21.15.2:0/2405670783 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5643ede27520 con 0x5643eddfe800 2023-11-16T16:47:00.996 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 16:47:00 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-d[161956]: starting mds.d at 2023-11-16T16:47:00.996 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 16:47:00 smithi002 systemd[1]: Started Ceph mds.d for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:47:01.196 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:01.262 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:00 smithi002 ceph-mon[128712]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:01.302 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:00 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:01.302 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:00 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:01.302 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:00 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:01.302 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:00 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:01.302 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:00 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:01.303 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:00 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:01.303 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:00 smithi141 ceph-mon[131184]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:01.303 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:00 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:01.303 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:00 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:01.303 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:00 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:01.303 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:00 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:01.303 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:00 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:01.304 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:00 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:01.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:00 smithi155 ceph-mon[131607]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:01.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:00 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:01.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:00 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:01.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:00 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:01.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:00 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:01.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:00 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:01.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:00 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:01.409 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.408+0000 7f151abff700 1 -- 172.21.15.2:0/508411921 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1514101e10 msgr2=0x7f15141086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:01.409 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.408+0000 7f151abff700 1 --2- 172.21.15.2:0/508411921 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1514101e10 0x7f15141086d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f150c00b0a0 tx=0x7f150c00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:01.409 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.408+0000 7f151abff700 1 -- 172.21.15.2:0/508411921 shutdown_connections 2023-11-16T16:47:01.410 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.408+0000 7f151abff700 1 --2- 172.21.15.2:0/508411921 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1514101e10 0x7f15141086d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:01.410 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.408+0000 7f151abff700 1 --2- 172.21.15.2:0/508411921 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1514100290 0x7f15141006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:01.410 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.408+0000 7f151abff700 1 --2- 172.21.15.2:0/508411921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1514101480 0x7f15141018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:01.410 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.408+0000 7f151abff700 1 -- 172.21.15.2:0/508411921 >> 172.21.15.2:0/508411921 conn(0x7f15140fb870 msgr2=0x7f15140fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:01.410 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.408+0000 7f151abff700 1 -- 172.21.15.2:0/508411921 shutdown_connections 2023-11-16T16:47:01.410 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.408+0000 7f151abff700 1 -- 172.21.15.2:0/508411921 wait complete. 2023-11-16T16:47:01.412 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.409+0000 7f151abff700 1 Processor -- start 2023-11-16T16:47:01.412 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.409+0000 7f151abff700 1 -- start start 2023-11-16T16:47:01.413 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.410+0000 7f151abff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1514100290 0x7f1514198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:01.413 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.410+0000 7f151abff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1514101480 0x7f15141987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:01.413 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.410+0000 7f151abff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1514101e10 0x7f151419f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:01.413 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.410+0000 7f151abff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151419fdf0 con 0x7f1514101e10 2023-11-16T16:47:01.413 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.410+0000 7f151abff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151419ff30 con 0x7f1514100290 2023-11-16T16:47:01.413 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.410+0000 7f151abff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15141a00c0 con 0x7f1514101480 2023-11-16T16:47:01.413 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.410+0000 7f151919c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1514101e10 0x7f151419f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:01.413 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.410+0000 7f151899b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1514100290 0x7f1514198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:01.413 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.410+0000 7f151919c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1514101e10 0x7f151419f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35822/0 (socket says 172.21.15.2:35822) 2023-11-16T16:47:01.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.410+0000 7f151919c700 1 -- 172.21.15.2:0/1806825842 learned_addr learned my addr 172.21.15.2:0/1806825842 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:01.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.410+0000 7f1513fff700 1 --2- 172.21.15.2:0/1806825842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1514101480 0x7f15141987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:01.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.410+0000 7f151919c700 1 -- 172.21.15.2:0/1806825842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1514101480 msgr2=0x7f15141987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:01.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.410+0000 7f151919c700 1 --2- 172.21.15.2:0/1806825842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1514101480 0x7f15141987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:01.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.410+0000 7f151919c700 1 -- 172.21.15.2:0/1806825842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1514100290 msgr2=0x7f1514198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:01.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.410+0000 7f151919c700 1 --2- 172.21.15.2:0/1806825842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1514100290 0x7f1514198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:01.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.410+0000 7f151919c700 1 -- 172.21.15.2:0/1806825842 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1508009520 con 0x7f1514101e10 2023-11-16T16:47:01.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.410+0000 7f151899b700 1 --2- 172.21.15.2:0/1806825842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1514100290 0x7f1514198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:01.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.410+0000 7f1513fff700 1 --2- 172.21.15.2:0/1806825842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1514101480 0x7f15141987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:01.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.411+0000 7f151919c700 1 --2- 172.21.15.2:0/1806825842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1514101e10 0x7f151419f820 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f150c00d010 tx=0x7f150c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:01.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.411+0000 7f1511ffb700 1 -- 172.21.15.2:0/1806825842 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c00e040 con 0x7f1514101e10 2023-11-16T16:47:01.415 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.411+0000 7f1511ffb700 1 -- 172.21.15.2:0/1806825842 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f150c003e10 con 0x7f1514101e10 2023-11-16T16:47:01.415 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.411+0000 7f151abff700 1 -- 172.21.15.2:0/1806825842 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f150c009d00 con 0x7f1514101e10 2023-11-16T16:47:01.415 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.411+0000 7f1511ffb700 1 -- 172.21.15.2:0/1806825842 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c0052a0 con 0x7f1514101e10 2023-11-16T16:47:01.415 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.411+0000 7f151abff700 1 -- 172.21.15.2:0/1806825842 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f151410bc70 con 0x7f1514101e10 2023-11-16T16:47:01.415 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.413+0000 7f151abff700 1 -- 172.21.15.2:0/1806825842 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1514067a60 con 0x7f1514101e10 2023-11-16T16:47:01.415 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.414+0000 7f1511ffb700 1 -- 172.21.15.2:0/1806825842 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f150c005400 con 0x7f1514101e10 2023-11-16T16:47:01.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.416+0000 7f1511ffb700 1 --2- 172.21.15.2:0/1806825842 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f15040a11d0 0x7f15040a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:01.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.416+0000 7f1511ffb700 1 -- 172.21.15.2:0/1806825842 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f150c0cd8c0 con 0x7f1514101e10 2023-11-16T16:47:01.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.416+0000 7f151899b700 1 --2- 172.21.15.2:0/1806825842 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f15040a11d0 0x7f15040a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:01.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.417+0000 7f151899b700 1 --2- 172.21.15.2:0/1806825842 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f15040a11d0 0x7f15040a3680 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1508009fd0 tx=0x7f1508009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:01.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.420+0000 7f1511ffb700 1 -- 172.21.15.2:0/1806825842 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f150c0974a0 con 0x7f1514101e10 2023-11-16T16:47:01.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.678+0000 7f151abff700 1 -- 172.21.15.2:0/1806825842 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15141a0a80 con 0x7f1514101e10 2023-11-16T16:47:01.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.679+0000 7f1511ffb700 1 -- 172.21.15.2:0/1806825842 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3528 (secure 0 0 0) 0x7f150c09aac0 con 0x7f1514101e10 2023-11-16T16:47:01.680 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:01.680 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/1610211225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":1610211225},{"type":"v1","addr":"172.21.15.2:6837","nonce":1610211225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-11-16T16:47:01.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.681+0000 7f151abff700 1 -- 172.21.15.2:0/1806825842 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f15040a11d0 msgr2=0x7f15040a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:01.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.681+0000 7f151abff700 1 --2- 172.21.15.2:0/1806825842 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f15040a11d0 0x7f15040a3680 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1508009fd0 tx=0x7f1508009260 comp rx=0 tx=0).stop 2023-11-16T16:47:01.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.681+0000 7f151abff700 1 -- 172.21.15.2:0/1806825842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1514101e10 msgr2=0x7f151419f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:01.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.681+0000 7f151abff700 1 --2- 172.21.15.2:0/1806825842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1514101e10 0x7f151419f820 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f150c00d010 tx=0x7f150c00bc80 comp rx=0 tx=0).stop 2023-11-16T16:47:01.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.682+0000 7f151abff700 1 -- 172.21.15.2:0/1806825842 shutdown_connections 2023-11-16T16:47:01.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.682+0000 7f151abff700 1 --2- 172.21.15.2:0/1806825842 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f15040a11d0 0x7f15040a3680 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:01.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.682+0000 7f151abff700 1 --2- 172.21.15.2:0/1806825842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1514101e10 0x7f151419f820 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:01.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.682+0000 7f151abff700 1 --2- 172.21.15.2:0/1806825842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1514100290 0x7f1514198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:01.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.682+0000 7f151abff700 1 --2- 172.21.15.2:0/1806825842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1514101480 0x7f15141987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:01.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.682+0000 7f151abff700 1 -- 172.21.15.2:0/1806825842 >> 172.21.15.2:0/1806825842 conn(0x7f15140fb870 msgr2=0x7f1514106cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:01.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.682+0000 7f151abff700 1 -- 172.21.15.2:0/1806825842 shutdown_connections 2023-11-16T16:47:01.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:01.682+0000 7f151abff700 1 -- 172.21.15.2:0/1806825842 wait complete. 2023-11-16T16:47:01.684 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 5 2023-11-16T16:47:02.199 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:02 smithi141 ceph-mon[131184]: Deploying daemon mds.e on smithi141 2023-11-16T16:47:02.199 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:02 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.2:6836/1610211225,v1:172.21.15.2:6837/1610211225] up:boot 2023-11-16T16:47:02.199 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:02 smithi141 ceph-mon[131184]: fsmap 4 up:standby 2023-11-16T16:47:02.199 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:02 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T16:47:02.199 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:02 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1806825842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:02.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:02 smithi155 ceph-mon[131607]: Deploying daemon mds.e on smithi141 2023-11-16T16:47:02.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:02 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.2:6836/1610211225,v1:172.21.15.2:6837/1610211225] up:boot 2023-11-16T16:47:02.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:02 smithi155 ceph-mon[131607]: fsmap 4 up:standby 2023-11-16T16:47:02.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:02 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T16:47:02.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:02 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1806825842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:02.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:02 smithi002 ceph-mon[128712]: Deploying daemon mds.e on smithi141 2023-11-16T16:47:02.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:02 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.2:6836/1610211225,v1:172.21.15.2:6837/1610211225] up:boot 2023-11-16T16:47:02.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:02 smithi002 ceph-mon[128712]: fsmap 4 up:standby 2023-11-16T16:47:02.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:02 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T16:47:02.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:02 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1806825842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:02.514 INFO:journalctl@ceph.mds.e.smithi141.stdout:Nov 16 16:47:02 smithi141 systemd[1]: Starting Ceph mds.e for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:47:02.885 INFO:journalctl@ceph.mds.e.smithi141.stdout:Nov 16 16:47:02 smithi141 podman[155717]: 2023-11-16T16:47:03.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:03 smithi141 ceph-mon[131184]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:03.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:03 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:03.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:03 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:03.265 INFO:journalctl@ceph.mds.e.smithi141.stdout:Nov 16 16:47:02 smithi141 ceph-mds[155757]: -- 172.21.15.141:0/3893370604 <== mon.2 v2:172.21.15.141:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5593fbab1520 con 0x5593fba87000 2023-11-16T16:47:03.265 INFO:journalctl@ceph.mds.e.smithi141.stdout:Nov 16 16:47:02 smithi141 bash[155717]: 475a2474adfc9fdbe95690d9f80c53729488e52702b8d6f8a7f829a7179c1020 2023-11-16T16:47:03.265 INFO:journalctl@ceph.mds.e.smithi141.stdout:Nov 16 16:47:02 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-e[155733]: starting mds.e at 2023-11-16T16:47:03.265 INFO:journalctl@ceph.mds.e.smithi141.stdout:Nov 16 16:47:02 smithi141 systemd[1]: Started Ceph mds.e for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:47:03.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:03 smithi155 ceph-mon[131607]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:03.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:03 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:03.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:03 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:03.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:03 smithi002 ceph-mon[128712]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:03.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:03 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:03.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:03 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:03.702 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:03.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.918+0000 7f2c47bff700 1 -- 172.21.15.2:0/23289434 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c40100290 msgr2=0x7f2c401006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:03.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.918+0000 7f2c47bff700 1 --2- 172.21.15.2:0/23289434 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c40100290 0x7f2c401006a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c009990 tx=0x7f2c3c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:03.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.918+0000 7f2c47bff700 1 -- 172.21.15.2:0/23289434 shutdown_connections 2023-11-16T16:47:03.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.918+0000 7f2c47bff700 1 --2- 172.21.15.2:0/23289434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c40101e10 0x7f2c401086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:03.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.918+0000 7f2c47bff700 1 --2- 172.21.15.2:0/23289434 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c40100290 0x7f2c401006a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:03.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.918+0000 7f2c47bff700 1 --2- 172.21.15.2:0/23289434 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2c40101480 0x7f2c401018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:03.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.918+0000 7f2c47bff700 1 -- 172.21.15.2:0/23289434 >> 172.21.15.2:0/23289434 conn(0x7f2c400fb870 msgr2=0x7f2c400fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:03.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.918+0000 7f2c47bff700 1 -- 172.21.15.2:0/23289434 shutdown_connections 2023-11-16T16:47:03.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.918+0000 7f2c47bff700 1 -- 172.21.15.2:0/23289434 wait complete. 2023-11-16T16:47:03.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.919+0000 7f2c47bff700 1 Processor -- start 2023-11-16T16:47:03.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.919+0000 7f2c47bff700 1 -- start start 2023-11-16T16:47:03.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.920+0000 7f2c47bff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c40100290 0x7f2c401981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:03.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.920+0000 7f2c47bff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2c40101480 0x7f2c40198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:03.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.920+0000 7f2c47bff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c40101e10 0x7f2c4019f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:03.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.920+0000 7f2c47bff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4019fd40 con 0x7f2c40101e10 2023-11-16T16:47:03.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.920+0000 7f2c47bff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4019fe80 con 0x7f2c40101480 2023-11-16T16:47:03.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.920+0000 7f2c47bff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c401a0010 con 0x7f2c40100290 2023-11-16T16:47:03.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.920+0000 7f2c4619c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c40101e10 0x7f2c4019f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:03.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.920+0000 7f2c4599b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c40100290 0x7f2c401981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:03.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.920+0000 7f2c4619c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c40101e10 0x7f2c4019f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35840/0 (socket says 172.21.15.2:35840) 2023-11-16T16:47:03.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.920+0000 7f2c4619c700 1 -- 172.21.15.2:0/3228276415 learned_addr learned my addr 172.21.15.2:0/3228276415 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:03.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.920+0000 7f2c4519a700 1 --2- 172.21.15.2:0/3228276415 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2c40101480 0x7f2c40198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:03.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.920+0000 7f2c4599b700 1 -- 172.21.15.2:0/3228276415 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2c40101480 msgr2=0x7f2c40198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:03.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.920+0000 7f2c4599b700 1 --2- 172.21.15.2:0/3228276415 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2c40101480 0x7f2c40198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:03.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.920+0000 7f2c4599b700 1 -- 172.21.15.2:0/3228276415 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c40101e10 msgr2=0x7f2c4019f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:03.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.920+0000 7f2c4599b700 1 --2- 172.21.15.2:0/3228276415 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c40101e10 0x7f2c4019f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:03.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.920+0000 7f2c4599b700 1 -- 172.21.15.2:0/3228276415 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3c0095f0 con 0x7f2c40100290 2023-11-16T16:47:03.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.921+0000 7f2c4619c700 1 --2- 172.21.15.2:0/3228276415 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c40101e10 0x7f2c4019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:03.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.921+0000 7f2c4599b700 1 --2- 172.21.15.2:0/3228276415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c40100290 0x7f2c401981c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c0098e0 tx=0x7f2c3c005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:03.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.921+0000 7f2c36ffd700 1 -- 172.21.15.2:0/3228276415 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c01d070 con 0x7f2c40100290 2023-11-16T16:47:03.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.921+0000 7f2c47bff700 1 -- 172.21.15.2:0/3228276415 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c4010bc40 con 0x7f2c40100290 2023-11-16T16:47:03.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.921+0000 7f2c36ffd700 1 -- 172.21.15.2:0/3228276415 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c3c021410 con 0x7f2c40100290 2023-11-16T16:47:03.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.921+0000 7f2c36ffd700 1 -- 172.21.15.2:0/3228276415 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c017650 con 0x7f2c40100290 2023-11-16T16:47:03.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.921+0000 7f2c47bff700 1 -- 172.21.15.2:0/3228276415 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4010bab0 con 0x7f2c40100290 2023-11-16T16:47:03.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.922+0000 7f2c47bff700 1 -- 172.21.15.2:0/3228276415 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c401928d0 con 0x7f2c40100290 2023-11-16T16:47:03.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.924+0000 7f2c36ffd700 1 -- 172.21.15.2:0/3228276415 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f2c3c021580 con 0x7f2c40100290 2023-11-16T16:47:03.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.926+0000 7f2c36ffd700 1 --2- 172.21.15.2:0/3228276415 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2c2c0a1150 0x7f2c2c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:03.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.926+0000 7f2c36ffd700 1 -- 172.21.15.2:0/3228276415 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2c3c0cdea0 con 0x7f2c40100290 2023-11-16T16:47:03.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.926+0000 7f2c4519a700 1 --2- 172.21.15.2:0/3228276415 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2c2c0a1150 0x7f2c2c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:03.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.926+0000 7f2c4519a700 1 --2- 172.21.15.2:0/3228276415 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2c2c0a1150 0x7f2c2c0a3600 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2c30009c60 tx=0x7f2c30009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:03.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:03.928+0000 7f2c36ffd700 1 -- 172.21.15.2:0/3228276415 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2c3c097a80 con 0x7f2c40100290 2023-11-16T16:47:04.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:04 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.141:6834/3516869188,v1:172.21.15.141:6835/3516869188] up:boot 2023-11-16T16:47:04.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:04 smithi002 ceph-mon[128712]: fsmap 5 up:standby 2023-11-16T16:47:04.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:04 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T16:47:04.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:04 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:04.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:04 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:04.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:04 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:04.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:04 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:04.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:04 smithi002 ceph-mon[128712]: Deploying daemon mds.f on smithi155 2023-11-16T16:47:04.185 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:04.184+0000 7f2c47bff700 1 -- 172.21.15.2:0/3228276415 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c4002cfe0 con 0x7f2c40100290 2023-11-16T16:47:04.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:04.185+0000 7f2c36ffd700 1 -- 172.21.15.2:0/3228276415 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4288 (secure 0 0 0) 0x7f2c3c09b0a0 con 0x7f2c40100290 2023-11-16T16:47:04.186 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:04.186 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/1610211225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":1610211225},{"type":"v1","addr":"172.21.15.2:6837","nonce":1610211225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3516869188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3516869188},{"type":"v1","addr":"172.21.15.141:6835","nonce":3516869188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-11-16T16:47:04.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:04.188+0000 7f2c47bff700 1 -- 172.21.15.2:0/3228276415 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2c2c0a1150 msgr2=0x7f2c2c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:04.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:04.188+0000 7f2c47bff700 1 --2- 172.21.15.2:0/3228276415 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2c2c0a1150 0x7f2c2c0a3600 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2c30009c60 tx=0x7f2c30009260 comp rx=0 tx=0).stop 2023-11-16T16:47:04.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:04.188+0000 7f2c47bff700 1 -- 172.21.15.2:0/3228276415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c40100290 msgr2=0x7f2c401981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:04.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:04.188+0000 7f2c47bff700 1 --2- 172.21.15.2:0/3228276415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c40100290 0x7f2c401981c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c0098e0 tx=0x7f2c3c005740 comp rx=0 tx=0).stop 2023-11-16T16:47:04.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:04.188+0000 7f2c47bff700 1 -- 172.21.15.2:0/3228276415 shutdown_connections 2023-11-16T16:47:04.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:04.188+0000 7f2c47bff700 1 --2- 172.21.15.2:0/3228276415 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2c2c0a1150 0x7f2c2c0a3600 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:04.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:04.188+0000 7f2c47bff700 1 --2- 172.21.15.2:0/3228276415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c40100290 0x7f2c401981c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:04.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:04.188+0000 7f2c47bff700 1 --2- 172.21.15.2:0/3228276415 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c40101e10 0x7f2c4019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:04.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:04.188+0000 7f2c47bff700 1 --2- 172.21.15.2:0/3228276415 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2c40101480 0x7f2c40198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:04.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:04.188+0000 7f2c47bff700 1 -- 172.21.15.2:0/3228276415 >> 172.21.15.2:0/3228276415 conn(0x7f2c400fb870 msgr2=0x7f2c40106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:04.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:04.188+0000 7f2c47bff700 1 -- 172.21.15.2:0/3228276415 shutdown_connections 2023-11-16T16:47:04.190 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:04.188+0000 7f2c47bff700 1 -- 172.21.15.2:0/3228276415 wait complete. 2023-11-16T16:47:04.190 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 6 2023-11-16T16:47:04.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:04 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.141:6834/3516869188,v1:172.21.15.141:6835/3516869188] up:boot 2023-11-16T16:47:04.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:04 smithi141 ceph-mon[131184]: fsmap 5 up:standby 2023-11-16T16:47:04.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:04 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T16:47:04.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:04 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:04.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:04 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:04.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:04 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:04.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:04 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:04.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:04 smithi141 ceph-mon[131184]: Deploying daemon mds.f on smithi155 2023-11-16T16:47:04.287 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:04 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.141:6834/3516869188,v1:172.21.15.141:6835/3516869188] up:boot 2023-11-16T16:47:04.287 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:04 smithi155 ceph-mon[131607]: fsmap 5 up:standby 2023-11-16T16:47:04.287 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:04 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T16:47:04.287 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:04 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:04.288 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:04 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:04.288 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:04 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:04.288 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:04 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:04.288 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:04 smithi155 ceph-mon[131607]: Deploying daemon mds.f on smithi155 2023-11-16T16:47:04.618 INFO:journalctl@ceph.mds.f.smithi155.stdout:Nov 16 16:47:04 smithi155 systemd[1]: Starting Ceph mds.f for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:47:05.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:05 smithi155 ceph-mon[131607]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:05.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:05 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3228276415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:05.118 INFO:journalctl@ceph.mds.f.smithi155.stdout:Nov 16 16:47:04 smithi155 podman[156964]: 2023-11-16T16:47:05.118 INFO:journalctl@ceph.mds.f.smithi155.stdout:Nov 16 16:47:05 smithi155 ceph-mds[157004]: -- 172.21.15.155:0/1623820035 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5560721ad520 con 0x556072183000 2023-11-16T16:47:05.118 INFO:journalctl@ceph.mds.f.smithi155.stdout:Nov 16 16:47:05 smithi155 bash[156964]: eba0770ed5742196b6052ea9b3b0129eb19c30234fea40dc554c3988cdbae117 2023-11-16T16:47:05.118 INFO:journalctl@ceph.mds.f.smithi155.stdout:Nov 16 16:47:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-f[156980]: starting mds.f at 2023-11-16T16:47:05.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:05 smithi002 ceph-mon[128712]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:05.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:05 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3228276415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:05.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:05 smithi141 ceph-mon[131184]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:05.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:05 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3228276415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:05.617 INFO:journalctl@ceph.mds.f.smithi155.stdout:Nov 16 16:47:05 smithi155 systemd[1]: Started Ceph mds.f for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:47:06.209 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:06.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.427+0000 7fc94b5ff700 1 -- 172.21.15.2:0/4006092327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9441002b0 msgr2=0x7fc9441006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:06.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.427+0000 7fc94b5ff700 1 --2- 172.21.15.2:0/4006092327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9441002b0 0x7fc9441006c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc940009990 tx=0x7fc940009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:06.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.427+0000 7fc94b5ff700 1 -- 172.21.15.2:0/4006092327 shutdown_connections 2023-11-16T16:47:06.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.427+0000 7fc94b5ff700 1 --2- 172.21.15.2:0/4006092327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9441002b0 0x7fc9441006c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:06.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.427+0000 7fc94b5ff700 1 --2- 172.21.15.2:0/4006092327 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc944101e30 0x7fc9441086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:06.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.427+0000 7fc94b5ff700 1 --2- 172.21.15.2:0/4006092327 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9441014a0 0x7fc9441018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:06.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.427+0000 7fc94b5ff700 1 -- 172.21.15.2:0/4006092327 >> 172.21.15.2:0/4006092327 conn(0x7fc9440fb890 msgr2=0x7fc9440fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:06.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.427+0000 7fc94b5ff700 1 -- 172.21.15.2:0/4006092327 shutdown_connections 2023-11-16T16:47:06.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.427+0000 7fc94b5ff700 1 -- 172.21.15.2:0/4006092327 wait complete. 2023-11-16T16:47:06.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.428+0000 7fc94b5ff700 1 Processor -- start 2023-11-16T16:47:06.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.428+0000 7fc94b5ff700 1 -- start start 2023-11-16T16:47:06.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.428+0000 7fc94b5ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9441002b0 0x7fc9440748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:06.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.428+0000 7fc94b5ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9441014a0 0x7fc944072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:06.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.428+0000 7fc94b5ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc944101e30 0x7fc944073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:06.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.428+0000 7fc94b5ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc944073bc0 con 0x7fc944101e30 2023-11-16T16:47:06.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.428+0000 7fc94b5ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc944073d30 con 0x7fc9441002b0 2023-11-16T16:47:06.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.428+0000 7fc94b5ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94419ffb0 con 0x7fc9441014a0 2023-11-16T16:47:06.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.429+0000 7fc948b9a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9441014a0 0x7fc944072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:06.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.429+0000 7fc949b9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc944101e30 0x7fc944073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:06.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.429+0000 7fc948b9a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9441014a0 0x7fc944072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:49672/0 (socket says 172.21.15.2:49672) 2023-11-16T16:47:06.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.429+0000 7fc948b9a700 1 -- 172.21.15.2:0/947132274 learned_addr learned my addr 172.21.15.2:0/947132274 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:06.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.429+0000 7fc94939b700 1 --2- 172.21.15.2:0/947132274 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9441002b0 0x7fc9440748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:06.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.429+0000 7fc948b9a700 1 -- 172.21.15.2:0/947132274 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9441002b0 msgr2=0x7fc9440748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:06.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.429+0000 7fc948b9a700 1 --2- 172.21.15.2:0/947132274 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9441002b0 0x7fc9440748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:06.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.429+0000 7fc948b9a700 1 -- 172.21.15.2:0/947132274 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc944101e30 msgr2=0x7fc944073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:06.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.429+0000 7fc948b9a700 1 --2- 172.21.15.2:0/947132274 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc944101e30 0x7fc944073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:06.435 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.429+0000 7fc948b9a700 1 -- 172.21.15.2:0/947132274 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc934009520 con 0x7fc9441014a0 2023-11-16T16:47:06.435 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.429+0000 7fc949b9c700 1 --2- 172.21.15.2:0/947132274 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc944101e30 0x7fc944073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:06.435 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.429+0000 7fc948b9a700 1 --2- 172.21.15.2:0/947132274 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9441014a0 0x7fc944072ee0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc93400eb30 tx=0x7fc93400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:06.435 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.429+0000 7fc93a7fc700 1 -- 172.21.15.2:0/947132274 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc934009880 con 0x7fc9441014a0 2023-11-16T16:47:06.435 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.430+0000 7fc93a7fc700 1 -- 172.21.15.2:0/947132274 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9340044e0 con 0x7fc9441014a0 2023-11-16T16:47:06.435 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.430+0000 7fc93a7fc700 1 -- 172.21.15.2:0/947132274 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc934003d80 con 0x7fc9441014a0 2023-11-16T16:47:06.435 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.430+0000 7fc94b5ff700 1 -- 172.21.15.2:0/947132274 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9400095f0 con 0x7fc9441014a0 2023-11-16T16:47:06.436 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.430+0000 7fc94b5ff700 1 -- 172.21.15.2:0/947132274 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94410bc00 con 0x7fc9441014a0 2023-11-16T16:47:06.436 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.431+0000 7fc94b5ff700 1 -- 172.21.15.2:0/947132274 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9441071e0 con 0x7fc9441014a0 2023-11-16T16:47:06.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.433+0000 7fc93a7fc700 1 -- 172.21.15.2:0/947132274 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fc934003f20 con 0x7fc9441014a0 2023-11-16T16:47:06.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.434+0000 7fc93a7fc700 1 --2- 172.21.15.2:0/947132274 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc9300a11d0 0x7fc9300a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:06.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.434+0000 7fc93a7fc700 1 -- 172.21.15.2:0/947132274 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc934014070 con 0x7fc9441014a0 2023-11-16T16:47:06.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.434+0000 7fc94939b700 1 --2- 172.21.15.2:0/947132274 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc9300a11d0 0x7fc9300a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:06.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.435+0000 7fc94939b700 1 --2- 172.21.15.2:0/947132274 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc9300a11d0 0x7fc9300a3680 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc94000b2e0 tx=0x7fc940005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:06.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.436+0000 7fc93a7fc700 1 -- 172.21.15.2:0/947132274 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc934096b10 con 0x7fc9441014a0 2023-11-16T16:47:06.457 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:06 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:06.457 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:06 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:06.457 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:06 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:06.457 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:06 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:06.457 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:06 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:06.457 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:06 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:06.458 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:06 smithi155 ceph-mon[131607]: Deploying daemon mds.c on smithi155 2023-11-16T16:47:06.458 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:06 smithi155 ceph-mon[131607]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:06.458 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:06 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.155:6834/2527313191,v1:172.21.15.155:6835/2527313191] up:boot 2023-11-16T16:47:06.458 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:06 smithi155 ceph-mon[131607]: fsmap 6 up:standby 2023-11-16T16:47:06.458 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:06 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T16:47:06.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:06 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:06.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:06 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:06.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:06 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:06.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:06 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:06.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:06 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:06.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:06 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:06.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:06 smithi141 ceph-mon[131184]: Deploying daemon mds.c on smithi155 2023-11-16T16:47:06.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:06 smithi141 ceph-mon[131184]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:06.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:06 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.155:6834/2527313191,v1:172.21.15.155:6835/2527313191] up:boot 2023-11-16T16:47:06.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:06 smithi141 ceph-mon[131184]: fsmap 6 up:standby 2023-11-16T16:47:06.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:06 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T16:47:06.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:06 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:06.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:06 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:06.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:06 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:06.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:06 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:06.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:06 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:06.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:06 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:06.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:06 smithi002 ceph-mon[128712]: Deploying daemon mds.c on smithi155 2023-11-16T16:47:06.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:06 smithi002 ceph-mon[128712]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:06.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:06 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.155:6834/2527313191,v1:172.21.15.155:6835/2527313191] up:boot 2023-11-16T16:47:06.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:06 smithi002 ceph-mon[128712]: fsmap 6 up:standby 2023-11-16T16:47:06.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:06 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T16:47:06.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.694+0000 7fc94b5ff700 1 -- 172.21.15.2:0/947132274 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc94402cfe0 con 0x7fc9441014a0 2023-11-16T16:47:06.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.695+0000 7fc93a7fc700 1 -- 172.21.15.2:0/947132274 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5048 (secure 0 0 0) 0x7fc93409a130 con 0x7fc9441014a0 2023-11-16T16:47:06.696 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:06.696 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/1610211225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":1610211225},{"type":"v1","addr":"172.21.15.2:6837","nonce":1610211225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3516869188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3516869188},{"type":"v1","addr":"172.21.15.141:6835","nonce":3516869188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2527313191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":2527313191},{"type":"v1","addr":"172.21.15.155:6835","nonce":2527313191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2023-11-16T16:47:06.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.697+0000 7fc94b5ff700 1 -- 172.21.15.2:0/947132274 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc9300a11d0 msgr2=0x7fc9300a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:06.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.697+0000 7fc94b5ff700 1 --2- 172.21.15.2:0/947132274 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc9300a11d0 0x7fc9300a3680 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc94000b2e0 tx=0x7fc940005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:06.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.697+0000 7fc94b5ff700 1 -- 172.21.15.2:0/947132274 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9441014a0 msgr2=0x7fc944072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:06.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.697+0000 7fc94b5ff700 1 --2- 172.21.15.2:0/947132274 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9441014a0 0x7fc944072ee0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc93400eb30 tx=0x7fc93400ee40 comp rx=0 tx=0).stop 2023-11-16T16:47:06.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.697+0000 7fc94b5ff700 1 -- 172.21.15.2:0/947132274 shutdown_connections 2023-11-16T16:47:06.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.697+0000 7fc94b5ff700 1 --2- 172.21.15.2:0/947132274 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc9300a11d0 0x7fc9300a3680 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:06.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.697+0000 7fc94b5ff700 1 --2- 172.21.15.2:0/947132274 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc944101e30 0x7fc944073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:06.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.697+0000 7fc94b5ff700 1 --2- 172.21.15.2:0/947132274 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9441002b0 0x7fc9440748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:06.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.697+0000 7fc94b5ff700 1 --2- 172.21.15.2:0/947132274 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9441014a0 0x7fc944072ee0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:06.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.697+0000 7fc94b5ff700 1 -- 172.21.15.2:0/947132274 >> 172.21.15.2:0/947132274 conn(0x7fc9440fb890 msgr2=0x7fc9440fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:06.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.697+0000 7fc94b5ff700 1 -- 172.21.15.2:0/947132274 shutdown_connections 2023-11-16T16:47:06.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:06.697+0000 7fc94b5ff700 1 -- 172.21.15.2:0/947132274 wait complete. 2023-11-16T16:47:06.700 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 7 2023-11-16T16:47:06.868 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:47:06 smithi155 systemd[1]: Starting Ceph mds.c for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:47:07.192 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:47:06 smithi155 podman[157304]: 2023-11-16T16:47:07.192 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:47:07 smithi155 ceph-mds[157344]: -- 172.21.15.155:0/696174398 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559f7a313520 con 0x559f7a2e8c00 2023-11-16T16:47:07.192 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:47:07 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: starting mds.c at 2023-11-16T16:47:07.192 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:47:07 smithi155 bash[157304]: 78b25505c4e9cd8865823cc5b2ee12cade25f87a0a8fe9dbaf54f7dea84da93f 2023-11-16T16:47:07.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:07 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/947132274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:07.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:07 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/947132274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:07.618 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:47:07 smithi155 systemd[1]: Started Ceph mds.c for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:47:07.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:07 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/947132274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:08.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:08 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.155:6836/1607756751,v1:172.21.15.155:6837/1607756751] up:boot 2023-11-16T16:47:08.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:08 smithi141 ceph-mon[131184]: fsmap 7 up:standby 2023-11-16T16:47:08.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:08 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T16:47:08.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:08 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:08.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:08 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:08.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:08 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:08.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:08 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:08.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:08 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:08.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:08 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:08.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:08 smithi141 ceph-mon[131184]: Deploying daemon mds.h on smithi141 2023-11-16T16:47:08.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:08 smithi141 ceph-mon[131184]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:08.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:08 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.155:6836/1607756751,v1:172.21.15.155:6837/1607756751] up:boot 2023-11-16T16:47:08.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:08 smithi155 ceph-mon[131607]: fsmap 7 up:standby 2023-11-16T16:47:08.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:08 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T16:47:08.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:08 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:08.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:08 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:08.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:08 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:08.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:08 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:08.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:08 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:08.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:08 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:08.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:08 smithi155 ceph-mon[131607]: Deploying daemon mds.h on smithi141 2023-11-16T16:47:08.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:08 smithi155 ceph-mon[131607]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:08.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:08 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.155:6836/1607756751,v1:172.21.15.155:6837/1607756751] up:boot 2023-11-16T16:47:08.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:08 smithi002 ceph-mon[128712]: fsmap 7 up:standby 2023-11-16T16:47:08.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:08 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T16:47:08.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:08 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:08.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:08 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:08.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:08 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:08.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:08 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:08.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:08 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:08.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:08 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:08.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:08 smithi002 ceph-mon[128712]: Deploying daemon mds.h on smithi141 2023-11-16T16:47:08.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:08 smithi002 ceph-mon[128712]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:08.719 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:08.933 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.931+0000 7f64aa3ff700 1 -- 172.21.15.2:0/2636371362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64a4100290 msgr2=0x7f64a41006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:08.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.931+0000 7f64aa3ff700 1 --2- 172.21.15.2:0/2636371362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64a4100290 0x7f64a41006a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6494009990 tx=0x7f6494009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:08.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.931+0000 7f64aa3ff700 1 -- 172.21.15.2:0/2636371362 shutdown_connections 2023-11-16T16:47:08.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.931+0000 7f64aa3ff700 1 --2- 172.21.15.2:0/2636371362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64a4100290 0x7f64a41006a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:08.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.931+0000 7f64aa3ff700 1 --2- 172.21.15.2:0/2636371362 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64a4101e10 0x7f64a41086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:08.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.931+0000 7f64aa3ff700 1 --2- 172.21.15.2:0/2636371362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64a4101480 0x7f64a41018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.931+0000 7f64aa3ff700 1 -- 172.21.15.2:0/2636371362 >> 172.21.15.2:0/2636371362 conn(0x7f64a40fb870 msgr2=0x7f64a40fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.932+0000 7f64aa3ff700 1 -- 172.21.15.2:0/2636371362 shutdown_connections 2023-11-16T16:47:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.932+0000 7f64aa3ff700 1 -- 172.21.15.2:0/2636371362 wait complete. 2023-11-16T16:47:08.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.933+0000 7f64aa3ff700 1 Processor -- start 2023-11-16T16:47:08.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.933+0000 7f64aa3ff700 1 -- start start 2023-11-16T16:47:08.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.933+0000 7f64aa3ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64a4100290 0x7f64a4198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:08.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.933+0000 7f64aa3ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64a4101480 0x7f64a41986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:08.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.933+0000 7f64aa3ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64a4101e10 0x7f64a419f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:08.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.933+0000 7f64aa3ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a419fce0 con 0x7f64a4101e10 2023-11-16T16:47:08.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.933+0000 7f64aa3ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a419fe20 con 0x7f64a4101480 2023-11-16T16:47:08.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.933+0000 7f64aa3ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a419ffb0 con 0x7f64a4100290 2023-11-16T16:47:08.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.933+0000 7f64a3fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64a4100290 0x7f64a4198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:08.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.933+0000 7f64a37fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64a4101480 0x7f64a41986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:08.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.933+0000 7f64a3fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64a4100290 0x7f64a4198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:49692/0 (socket says 172.21.15.2:49692) 2023-11-16T16:47:08.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.933+0000 7f64a899c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64a4101e10 0x7f64a419f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:08.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.933+0000 7f64a3fff700 1 -- 172.21.15.2:0/2358173349 learned_addr learned my addr 172.21.15.2:0/2358173349 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:08.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.934+0000 7f64a3fff700 1 -- 172.21.15.2:0/2358173349 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64a4101480 msgr2=0x7f64a41986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:08.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.934+0000 7f64a3fff700 1 --2- 172.21.15.2:0/2358173349 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64a4101480 0x7f64a41986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:08.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.934+0000 7f64a3fff700 1 -- 172.21.15.2:0/2358173349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64a4101e10 msgr2=0x7f64a419f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.934+0000 7f64a3fff700 1 --2- 172.21.15.2:0/2358173349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64a4101e10 0x7f64a419f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.934+0000 7f64a3fff700 1 -- 172.21.15.2:0/2358173349 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6498009520 con 0x7f64a4100290 2023-11-16T16:47:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.934+0000 7f64a37fe700 1 --2- 172.21.15.2:0/2358173349 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64a4101480 0x7f64a41986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.934+0000 7f64a899c700 1 --2- 172.21.15.2:0/2358173349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64a4101e10 0x7f64a419f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.934+0000 7f64a3fff700 1 --2- 172.21.15.2:0/2358173349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64a4100290 0x7f64a4198160 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f64940095c0 tx=0x7f6494005720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.934+0000 7f64a17fa700 1 -- 172.21.15.2:0/2358173349 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649401d070 con 0x7f64a4100290 2023-11-16T16:47:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.934+0000 7f64a17fa700 1 -- 172.21.15.2:0/2358173349 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6494022470 con 0x7f64a4100290 2023-11-16T16:47:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.934+0000 7f64aa3ff700 1 -- 172.21.15.2:0/2358173349 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f649c009c40 con 0x7f64a4100290 2023-11-16T16:47:08.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.935+0000 7f64a17fa700 1 -- 172.21.15.2:0/2358173349 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6494017650 con 0x7f64a4100290 2023-11-16T16:47:08.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.935+0000 7f64aa3ff700 1 -- 172.21.15.2:0/2358173349 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64940095f0 con 0x7f64a4100290 2023-11-16T16:47:08.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.936+0000 7f64aa3ff700 1 -- 172.21.15.2:0/2358173349 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a4192870 con 0x7f64a4100290 2023-11-16T16:47:08.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.938+0000 7f64a17fa700 1 -- 172.21.15.2:0/2358173349 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f64940225e0 con 0x7f64a4100290 2023-11-16T16:47:08.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.939+0000 7f64a17fa700 1 --2- 172.21.15.2:0/2358173349 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f648c0a1150 0x7f648c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:08.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.940+0000 7f64a17fa700 1 -- 172.21.15.2:0/2358173349 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f64940cdfe0 con 0x7f64a4100290 2023-11-16T16:47:08.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.940+0000 7f64a37fe700 1 --2- 172.21.15.2:0/2358173349 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f648c0a1150 0x7f648c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:08.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.940+0000 7f64a37fe700 1 --2- 172.21.15.2:0/2358173349 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f648c0a1150 0x7f648c0a3600 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6498009fd0 tx=0x7f6498009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:08.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:08.941+0000 7f64a17fa700 1 -- 172.21.15.2:0/2358173349 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6494097c20 con 0x7f64a4100290 2023-11-16T16:47:09.015 INFO:journalctl@ceph.mds.h.smithi141.stdout:Nov 16 16:47:08 smithi141 systemd[1]: Starting Ceph mds.h for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:47:09.199 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:09.198+0000 7f64aa3ff700 1 -- 172.21.15.2:0/2358173349 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64a4067a60 con 0x7f64a4100290 2023-11-16T16:47:09.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:09.198+0000 7f64a17fa700 1 -- 172.21.15.2:0/2358173349 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5808 (secure 0 0 0) 0x7f649409b240 con 0x7f64a4100290 2023-11-16T16:47:09.200 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:09.201 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/1610211225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":1610211225},{"type":"v1","addr":"172.21.15.2:6837","nonce":1610211225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3516869188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3516869188},{"type":"v1","addr":"172.21.15.141:6835","nonce":3516869188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2527313191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":2527313191},{"type":"v1","addr":"172.21.15.155:6835","nonce":2527313191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1607756751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1607756751},{"type":"v1","addr":"172.21.15.155:6837","nonce":1607756751}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-11-16T16:47:09.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:09.201+0000 7f64aa3ff700 1 -- 172.21.15.2:0/2358173349 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f648c0a1150 msgr2=0x7f648c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:09.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:09.201+0000 7f64aa3ff700 1 --2- 172.21.15.2:0/2358173349 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f648c0a1150 0x7f648c0a3600 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6498009fd0 tx=0x7f6498009260 comp rx=0 tx=0).stop 2023-11-16T16:47:09.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:09.201+0000 7f64aa3ff700 1 -- 172.21.15.2:0/2358173349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64a4100290 msgr2=0x7f64a4198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:09.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:09.201+0000 7f64aa3ff700 1 --2- 172.21.15.2:0/2358173349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64a4100290 0x7f64a4198160 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f64940095c0 tx=0x7f6494005720 comp rx=0 tx=0).stop 2023-11-16T16:47:09.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:09.201+0000 7f64aa3ff700 1 -- 172.21.15.2:0/2358173349 shutdown_connections 2023-11-16T16:47:09.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:09.201+0000 7f64aa3ff700 1 --2- 172.21.15.2:0/2358173349 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f648c0a1150 0x7f648c0a3600 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:09.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:09.201+0000 7f64aa3ff700 1 --2- 172.21.15.2:0/2358173349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64a4100290 0x7f64a4198160 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:09.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:09.201+0000 7f64aa3ff700 1 --2- 172.21.15.2:0/2358173349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64a4101e10 0x7f64a419f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:09.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:09.201+0000 7f64aa3ff700 1 --2- 172.21.15.2:0/2358173349 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f64a4101480 0x7f64a41986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:09.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:09.201+0000 7f64aa3ff700 1 -- 172.21.15.2:0/2358173349 >> 172.21.15.2:0/2358173349 conn(0x7f64a40fb870 msgr2=0x7f64a4106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:09.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:09.201+0000 7f64aa3ff700 1 -- 172.21.15.2:0/2358173349 shutdown_connections 2023-11-16T16:47:09.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:09.201+0000 7f64aa3ff700 1 -- 172.21.15.2:0/2358173349 wait complete. 2023-11-16T16:47:09.205 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 8 2023-11-16T16:47:09.434 INFO:journalctl@ceph.mds.h.smithi141.stdout:Nov 16 16:47:09 smithi141 podman[156056]: 2023-11-16T16:47:09.765 INFO:journalctl@ceph.mds.h.smithi141.stdout:Nov 16 16:47:09 smithi141 bash[156056]: 8ca4ca040eacdd64e67947708b0ed9c5bb2da3ea7ba0e3ead76ea13d47f87a6b 2023-11-16T16:47:09.765 INFO:journalctl@ceph.mds.h.smithi141.stdout:Nov 16 16:47:09 smithi141 ceph-mds[156097]: -- 172.21.15.141:0/2104233136 <== mon.2 v2:172.21.15.141:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c47acb7520 con 0x55c47ac8d000 2023-11-16T16:47:09.765 INFO:journalctl@ceph.mds.h.smithi141.stdout:Nov 16 16:47:09 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-h[156072]: starting mds.h at 2023-11-16T16:47:09.765 INFO:journalctl@ceph.mds.h.smithi141.stdout:Nov 16 16:47:09 smithi141 systemd[1]: Started Ceph mds.h for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:47:10.262 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:09 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2358173349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:10.262 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:09 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:10.262 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:09 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:10.262 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:09 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:10.263 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:09 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:10.263 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:09 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:10.263 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:09 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:10.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:09 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2358173349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:10.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:09 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:10.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:09 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:10.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:09 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:10.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:09 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:10.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:09 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:10.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:09 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:10.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:09 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2358173349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:10.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:09 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:10.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:09 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:10.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:09 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:10.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:09 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:10.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:09 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:10.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:09 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:11.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:11 smithi155 ceph-mon[131607]: Deploying daemon mds.i on smithi155 2023-11-16T16:47:11.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:11 smithi155 ceph-mon[131607]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:11.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:11 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] up:boot 2023-11-16T16:47:11.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:11 smithi155 ceph-mon[131607]: fsmap 8 up:standby 2023-11-16T16:47:11.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:11 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T16:47:11.120 INFO:journalctl@ceph.mds.i.smithi155.stdout:Nov 16 16:47:11 smithi155 systemd[1]: Starting Ceph mds.i for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:47:11.223 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:11.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:11 smithi141 ceph-mon[131184]: Deploying daemon mds.i on smithi155 2023-11-16T16:47:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:11 smithi141 ceph-mon[131184]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:11 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] up:boot 2023-11-16T16:47:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:11 smithi141 ceph-mon[131184]: fsmap 8 up:standby 2023-11-16T16:47:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:11 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T16:47:11.292 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:10 smithi002 ceph-mon[128712]: Deploying daemon mds.i on smithi155 2023-11-16T16:47:11.292 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:10 smithi002 ceph-mon[128712]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:11.292 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:11 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] up:boot 2023-11-16T16:47:11.292 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:11 smithi002 ceph-mon[128712]: fsmap 8 up:standby 2023-11-16T16:47:11.292 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:11 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T16:47:11.439 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.437+0000 7fef947ff700 1 -- 172.21.15.2:0/1065906484 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef8c100290 msgr2=0x7fef8c1006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:11.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.437+0000 7fef947ff700 1 --2- 172.21.15.2:0/1065906484 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef8c100290 0x7fef8c1006a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fef88009990 tx=0x7fef88009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:11.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.438+0000 7fef947ff700 1 -- 172.21.15.2:0/1065906484 shutdown_connections 2023-11-16T16:47:11.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.438+0000 7fef947ff700 1 --2- 172.21.15.2:0/1065906484 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef8c101e10 0x7fef8c1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:11.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.438+0000 7fef947ff700 1 --2- 172.21.15.2:0/1065906484 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef8c100290 0x7fef8c1006a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:11.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.438+0000 7fef947ff700 1 --2- 172.21.15.2:0/1065906484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef8c101480 0x7fef8c1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:11.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.438+0000 7fef947ff700 1 -- 172.21.15.2:0/1065906484 >> 172.21.15.2:0/1065906484 conn(0x7fef8c0fb870 msgr2=0x7fef8c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:11.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.438+0000 7fef947ff700 1 -- 172.21.15.2:0/1065906484 shutdown_connections 2023-11-16T16:47:11.440 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.438+0000 7fef947ff700 1 -- 172.21.15.2:0/1065906484 wait complete. 2023-11-16T16:47:11.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.439+0000 7fef947ff700 1 Processor -- start 2023-11-16T16:47:11.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.439+0000 7fef947ff700 1 -- start start 2023-11-16T16:47:11.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.439+0000 7fef947ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef8c100290 0x7fef8c196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:11.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.439+0000 7fef947ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef8c101480 0x7fef8c196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:11.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.439+0000 7fef947ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef8c101e10 0x7fef8c19d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:11.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.439+0000 7fef947ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8c19dba0 con 0x7fef8c101e10 2023-11-16T16:47:11.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.439+0000 7fef947ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8c19dce0 con 0x7fef8c101480 2023-11-16T16:47:11.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.439+0000 7fef947ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8c19de70 con 0x7fef8c100290 2023-11-16T16:47:11.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.439+0000 7fef9259b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef8c100290 0x7fef8c196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:11.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.439+0000 7fef9259b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef8c100290 0x7fef8c196020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:49700/0 (socket says 172.21.15.2:49700) 2023-11-16T16:47:11.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.439+0000 7fef9259b700 1 -- 172.21.15.2:0/3782544166 learned_addr learned my addr 172.21.15.2:0/3782544166 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:11.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.439+0000 7fef92d9c700 1 --2- 172.21.15.2:0/3782544166 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef8c101e10 0x7fef8c19d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:11.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.440+0000 7fef91d9a700 1 --2- 172.21.15.2:0/3782544166 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef8c101480 0x7fef8c196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:11.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.440+0000 7fef9259b700 1 -- 172.21.15.2:0/3782544166 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef8c101480 msgr2=0x7fef8c196560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:11.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.440+0000 7fef9259b700 1 --2- 172.21.15.2:0/3782544166 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef8c101480 0x7fef8c196560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:11.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.440+0000 7fef9259b700 1 -- 172.21.15.2:0/3782544166 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef8c101e10 msgr2=0x7fef8c19d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:11.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.440+0000 7fef9259b700 1 --2- 172.21.15.2:0/3782544166 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef8c101e10 0x7fef8c19d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:11.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.440+0000 7fef9259b700 1 -- 172.21.15.2:0/3782544166 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c009520 con 0x7fef8c100290 2023-11-16T16:47:11.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.440+0000 7fef92d9c700 1 --2- 172.21.15.2:0/3782544166 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef8c101e10 0x7fef8c19d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:11.443 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.440+0000 7fef9259b700 1 --2- 172.21.15.2:0/3782544166 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef8c100290 0x7fef8c196020 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fef88005740 tx=0x7fef88005770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:11.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.440+0000 7fef837fe700 1 -- 172.21.15.2:0/3782544166 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8801d070 con 0x7fef8c100290 2023-11-16T16:47:11.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.440+0000 7fef947ff700 1 -- 172.21.15.2:0/3782544166 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef84009c40 con 0x7fef8c100290 2023-11-16T16:47:11.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.440+0000 7fef837fe700 1 -- 172.21.15.2:0/3782544166 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef88021410 con 0x7fef8c100290 2023-11-16T16:47:11.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.440+0000 7fef837fe700 1 -- 172.21.15.2:0/3782544166 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88017650 con 0x7fef8c100290 2023-11-16T16:47:11.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.441+0000 7fef947ff700 1 -- 172.21.15.2:0/3782544166 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef880095f0 con 0x7fef8c100290 2023-11-16T16:47:11.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.442+0000 7fef947ff700 1 -- 172.21.15.2:0/3782544166 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef8c10bab0 con 0x7fef8c100290 2023-11-16T16:47:11.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.443+0000 7fef837fe700 1 -- 172.21.15.2:0/3782544166 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fef88021580 con 0x7fef8c100290 2023-11-16T16:47:11.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.445+0000 7fef837fe700 1 --2- 172.21.15.2:0/3782544166 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fef780a1150 0x7fef780a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:11.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.445+0000 7fef837fe700 1 -- 172.21.15.2:0/3782544166 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fef880cd9e0 con 0x7fef8c100290 2023-11-16T16:47:11.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.445+0000 7fef91d9a700 1 --2- 172.21.15.2:0/3782544166 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fef780a1150 0x7fef780a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:11.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.445+0000 7fef91d9a700 1 --2- 172.21.15.2:0/3782544166 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fef780a1150 0x7fef780a3600 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009df0 tx=0x7fef7c012040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:11.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.447+0000 7fef837fe700 1 -- 172.21.15.2:0/3782544166 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fef88097650 con 0x7fef8c100290 2023-11-16T16:47:11.617 INFO:journalctl@ceph.mds.i.smithi155.stdout:Nov 16 16:47:11 smithi155 podman[157645]: 2023-11-16T16:47:11.618 INFO:journalctl@ceph.mds.i.smithi155.stdout:Nov 16 16:47:11 smithi155 bash[157645]: 00ca969bfe73c16cf70641b9f4c2e8f92c8a5cdc923573b5ba3ebe8cee2ac5e9 2023-11-16T16:47:11.618 INFO:journalctl@ceph.mds.i.smithi155.stdout:Nov 16 16:47:11 smithi155 ceph-mds[157685]: -- 172.21.15.155:0/2958418361 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563ae5d0d520 con 0x563ae5ce3000 2023-11-16T16:47:11.618 INFO:journalctl@ceph.mds.i.smithi155.stdout:Nov 16 16:47:11 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-i[157661]: starting mds.i at 2023-11-16T16:47:11.704 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.702+0000 7fef947ff700 1 -- 172.21.15.2:0/3782544166 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef8c0734c0 con 0x7fef8c100290 2023-11-16T16:47:11.705 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.703+0000 7fef837fe700 1 -- 172.21.15.2:0/3782544166 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6568 (secure 0 0 0) 0x7fef8809ac70 con 0x7fef8c100290 2023-11-16T16:47:11.705 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:11.705 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/1610211225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":1610211225},{"type":"v1","addr":"172.21.15.2:6837","nonce":1610211225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3516869188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3516869188},{"type":"v1","addr":"172.21.15.141:6835","nonce":3516869188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2527313191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":2527313191},{"type":"v1","addr":"172.21.15.155:6835","nonce":2527313191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1607756751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1607756751},{"type":"v1","addr":"172.21.15.155:6837","nonce":1607756751}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6837/2934891475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6836","nonce":2934891475},{"type":"v1","addr":"172.21.15.141:6837","nonce":2934891475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-11-16T16:47:11.707 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.705+0000 7fef947ff700 1 -- 172.21.15.2:0/3782544166 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fef780a1150 msgr2=0x7fef780a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:11.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.705+0000 7fef947ff700 1 --2- 172.21.15.2:0/3782544166 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fef780a1150 0x7fef780a3600 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009df0 tx=0x7fef7c012040 comp rx=0 tx=0).stop 2023-11-16T16:47:11.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.705+0000 7fef947ff700 1 -- 172.21.15.2:0/3782544166 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef8c100290 msgr2=0x7fef8c196020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:11.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.705+0000 7fef947ff700 1 --2- 172.21.15.2:0/3782544166 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef8c100290 0x7fef8c196020 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fef88005740 tx=0x7fef88005770 comp rx=0 tx=0).stop 2023-11-16T16:47:11.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.705+0000 7fef947ff700 1 -- 172.21.15.2:0/3782544166 shutdown_connections 2023-11-16T16:47:11.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.705+0000 7fef947ff700 1 --2- 172.21.15.2:0/3782544166 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fef780a1150 0x7fef780a3600 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:11.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.705+0000 7fef947ff700 1 --2- 172.21.15.2:0/3782544166 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fef8c100290 0x7fef8c196020 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:11.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.705+0000 7fef947ff700 1 --2- 172.21.15.2:0/3782544166 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef8c101e10 0x7fef8c19d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:11.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.705+0000 7fef947ff700 1 --2- 172.21.15.2:0/3782544166 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fef8c101480 0x7fef8c196560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:11.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.706+0000 7fef947ff700 1 -- 172.21.15.2:0/3782544166 >> 172.21.15.2:0/3782544166 conn(0x7fef8c0fb870 msgr2=0x7fef8c0fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:11.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.706+0000 7fef947ff700 1 -- 172.21.15.2:0/3782544166 shutdown_connections 2023-11-16T16:47:11.708 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:11.706+0000 7fef947ff700 1 -- 172.21.15.2:0/3782544166 wait complete. 2023-11-16T16:47:11.709 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 9 2023-11-16T16:47:12.118 INFO:journalctl@ceph.mds.i.smithi155.stdout:Nov 16 16:47:11 smithi155 systemd[1]: Started Ceph mds.i for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:47:12.977 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:12 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:12.978 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:12 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:12.978 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:12 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3782544166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:12.978 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:12 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:12.978 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:12 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:12.978 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:12 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:12.978 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:12 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:12.978 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:12 smithi002 ceph-mon[128712]: Deploying daemon mds.g on smithi002 2023-11-16T16:47:12.978 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:12 smithi002 ceph-mon[128712]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:12.978 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:12 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] up:boot 2023-11-16T16:47:12.978 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:12 smithi002 ceph-mon[128712]: fsmap 9 up:standby 2023-11-16T16:47:12.978 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:12 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T16:47:13.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:12 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:13.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:12 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:13.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:12 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3782544166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:13.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:12 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:13.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:12 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:13.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:12 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:13.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:12 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:13.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:12 smithi141 ceph-mon[131184]: Deploying daemon mds.g on smithi002 2023-11-16T16:47:13.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:12 smithi141 ceph-mon[131184]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:13.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:12 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] up:boot 2023-11-16T16:47:13.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:12 smithi141 ceph-mon[131184]: fsmap 9 up:standby 2023-11-16T16:47:13.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:12 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T16:47:13.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:12 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:13.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:12 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:13.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:12 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3782544166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:13.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:12 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:13.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:12 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:13.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:12 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:13.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:12 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:13.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:12 smithi155 ceph-mon[131607]: Deploying daemon mds.g on smithi002 2023-11-16T16:47:13.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:12 smithi155 ceph-mon[131607]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:13.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:12 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] up:boot 2023-11-16T16:47:13.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:12 smithi155 ceph-mon[131607]: fsmap 9 up:standby 2023-11-16T16:47:13.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:12 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T16:47:13.727 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:14.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.147+0000 7fc29d3ff700 1 -- 172.21.15.2:0/1032752200 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc298101e10 msgr2=0x7fc2981086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:14.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.147+0000 7fc29d3ff700 1 --2- 172.21.15.2:0/1032752200 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc298101e10 0x7fc2981086d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc29000b0a0 tx=0x7fc29000b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.147+0000 7fc29d3ff700 1 -- 172.21.15.2:0/1032752200 shutdown_connections 2023-11-16T16:47:14.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.147+0000 7fc29d3ff700 1 --2- 172.21.15.2:0/1032752200 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc298101e10 0x7fc2981086d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.147+0000 7fc29d3ff700 1 --2- 172.21.15.2:0/1032752200 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc298100290 0x7fc2981006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.147+0000 7fc29d3ff700 1 --2- 172.21.15.2:0/1032752200 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc298101480 0x7fc2981018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.147+0000 7fc29d3ff700 1 -- 172.21.15.2:0/1032752200 >> 172.21.15.2:0/1032752200 conn(0x7fc2980fb870 msgr2=0x7fc2980fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:14.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.147+0000 7fc29d3ff700 1 -- 172.21.15.2:0/1032752200 shutdown_connections 2023-11-16T16:47:14.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.147+0000 7fc29d3ff700 1 -- 172.21.15.2:0/1032752200 wait complete. 2023-11-16T16:47:14.152 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.148+0000 7fc29d3ff700 1 Processor -- start 2023-11-16T16:47:14.152 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.149+0000 7fc29d3ff700 1 -- start start 2023-11-16T16:47:14.152 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.149+0000 7fc29d3ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc298100290 0x7fc298196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:14.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.149+0000 7fc29d3ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc298101480 0x7fc2981965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:14.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.149+0000 7fc29d3ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc298101e10 0x7fc29819d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:14.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.149+0000 7fc29d3ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29819dc10 con 0x7fc298100290 2023-11-16T16:47:14.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.149+0000 7fc29d3ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29819dd50 con 0x7fc298101e10 2023-11-16T16:47:14.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.149+0000 7fc29d3ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29819dee0 con 0x7fc298101480 2023-11-16T16:47:14.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.149+0000 7fc296ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc298100290 0x7fc298196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:14.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.149+0000 7fc296ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc298100290 0x7fc298196090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54672/0 (socket says 172.21.15.2:54672) 2023-11-16T16:47:14.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.149+0000 7fc296ffd700 1 -- 172.21.15.2:0/2256152299 learned_addr learned my addr 172.21.15.2:0/2256152299 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:14.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.149+0000 7fc2967fc700 1 --2- 172.21.15.2:0/2256152299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc298101480 0x7fc2981965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:14.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.150+0000 7fc2977fe700 1 --2- 172.21.15.2:0/2256152299 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc298101e10 0x7fc29819d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:14.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.150+0000 7fc296ffd700 1 -- 172.21.15.2:0/2256152299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc298101480 msgr2=0x7fc2981965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:14.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.150+0000 7fc296ffd700 1 --2- 172.21.15.2:0/2256152299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc298101480 0x7fc2981965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.150+0000 7fc296ffd700 1 -- 172.21.15.2:0/2256152299 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc298101e10 msgr2=0x7fc29819d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:14.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.150+0000 7fc296ffd700 1 --2- 172.21.15.2:0/2256152299 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc298101e10 0x7fc29819d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.155 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.150+0000 7fc296ffd700 1 -- 172.21.15.2:0/2256152299 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc290009d00 con 0x7fc298100290 2023-11-16T16:47:14.155 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.150+0000 7fc296ffd700 1 --2- 172.21.15.2:0/2256152299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc298100290 0x7fc298196090 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc28800e970 tx=0x7fc28800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:14.155 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.150+0000 7fc27ffff700 1 -- 172.21.15.2:0/2256152299 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288009790 con 0x7fc298100290 2023-11-16T16:47:14.155 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.150+0000 7fc27ffff700 1 -- 172.21.15.2:0/2256152299 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc28800c7e0 con 0x7fc298100290 2023-11-16T16:47:14.155 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.150+0000 7fc29d3ff700 1 -- 172.21.15.2:0/2256152299 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc29810bc70 con 0x7fc298100290 2023-11-16T16:47:14.155 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.151+0000 7fc27ffff700 1 -- 172.21.15.2:0/2256152299 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288003c70 con 0x7fc298100290 2023-11-16T16:47:14.155 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.151+0000 7fc29d3ff700 1 -- 172.21.15.2:0/2256152299 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29810bae0 con 0x7fc298100290 2023-11-16T16:47:14.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.152+0000 7fc29d3ff700 1 -- 172.21.15.2:0/2256152299 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc29819e680 con 0x7fc298100290 2023-11-16T16:47:14.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.154+0000 7fc27ffff700 1 -- 172.21.15.2:0/2256152299 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fc28800cc50 con 0x7fc298100290 2023-11-16T16:47:14.157 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.155+0000 7fc27ffff700 1 --2- 172.21.15.2:0/2256152299 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2800a1250 0x7fc2800a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:14.157 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.155+0000 7fc27ffff700 1 -- 172.21.15.2:0/2256152299 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc288014070 con 0x7fc298100290 2023-11-16T16:47:14.157 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.155+0000 7fc2967fc700 1 --2- 172.21.15.2:0/2256152299 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2800a1250 0x7fc2800a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:14.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.156+0000 7fc2967fc700 1 --2- 172.21.15.2:0/2256152299 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2800a1250 0x7fc2800a3700 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc28c009c60 tx=0x7fc28c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:14.160 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.158+0000 7fc27ffff700 1 -- 172.21.15.2:0/2256152299 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc288096290 con 0x7fc298100290 2023-11-16T16:47:14.405 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 16:47:14 smithi002 systemd[1]: Starting Ceph mds.g for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:47:14.433 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.432+0000 7fc29d3ff700 1 -- 172.21.15.2:0/2256152299 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc298107100 con 0x7fc298100290 2023-11-16T16:47:14.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.432+0000 7fc27ffff700 1 -- 172.21.15.2:0/2256152299 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7330 (secure 0 0 0) 0x7fc2880998b0 con 0x7fc298100290 2023-11-16T16:47:14.434 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:14.435 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/1610211225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":1610211225},{"type":"v1","addr":"172.21.15.2:6837","nonce":1610211225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3516869188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3516869188},{"type":"v1","addr":"172.21.15.141:6835","nonce":3516869188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2527313191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":2527313191},{"type":"v1","addr":"172.21.15.155:6835","nonce":2527313191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1607756751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1607756751},{"type":"v1","addr":"172.21.15.155:6837","nonce":1607756751}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6837/2934891475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6836","nonce":2934891475},{"type":"v1","addr":"172.21.15.141:6837","nonce":2934891475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/1158457009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1158457009},{"type":"v1","addr":"172.21.15.155:6839","nonce":1158457009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-11-16T16:47:14.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.435+0000 7fc29d3ff700 1 -- 172.21.15.2:0/2256152299 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2800a1250 msgr2=0x7fc2800a3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:14.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.435+0000 7fc29d3ff700 1 --2- 172.21.15.2:0/2256152299 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2800a1250 0x7fc2800a3700 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc28c009c60 tx=0x7fc28c009260 comp rx=0 tx=0).stop 2023-11-16T16:47:14.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.435+0000 7fc29d3ff700 1 -- 172.21.15.2:0/2256152299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc298100290 msgr2=0x7fc298196090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:14.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.435+0000 7fc29d3ff700 1 --2- 172.21.15.2:0/2256152299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc298100290 0x7fc298196090 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc28800e970 tx=0x7fc28800ec80 comp rx=0 tx=0).stop 2023-11-16T16:47:14.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.435+0000 7fc29d3ff700 1 -- 172.21.15.2:0/2256152299 shutdown_connections 2023-11-16T16:47:14.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.435+0000 7fc29d3ff700 1 --2- 172.21.15.2:0/2256152299 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2800a1250 0x7fc2800a3700 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.435+0000 7fc29d3ff700 1 --2- 172.21.15.2:0/2256152299 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc298101e10 0x7fc29819d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.435+0000 7fc29d3ff700 1 --2- 172.21.15.2:0/2256152299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc298100290 0x7fc298196090 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.435+0000 7fc29d3ff700 1 --2- 172.21.15.2:0/2256152299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc298101480 0x7fc2981965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:14.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.435+0000 7fc29d3ff700 1 -- 172.21.15.2:0/2256152299 >> 172.21.15.2:0/2256152299 conn(0x7fc2980fb870 msgr2=0x7fc2980fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:14.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.435+0000 7fc29d3ff700 1 -- 172.21.15.2:0/2256152299 shutdown_connections 2023-11-16T16:47:14.438 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:14.435+0000 7fc29d3ff700 1 -- 172.21.15.2:0/2256152299 wait complete. 2023-11-16T16:47:14.438 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 10 2023-11-16T16:47:14.811 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 16:47:14 smithi002 podman[162558]: 2023-11-16T16:47:15.155 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 16:47:14 smithi002 ceph-mds[162600]: -- 172.21.15.2:0/2854456542 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ba65753520 con 0x55ba6572b000 2023-11-16T16:47:15.156 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 16:47:14 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-g[162576]: starting mds.g at 2023-11-16T16:47:15.156 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 16:47:14 smithi002 bash[162558]: e82d6215a6d0e70b2fa55133650fce39f337c1eb97aad4b65c33a509106ccd7e 2023-11-16T16:47:15.156 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 16:47:14 smithi002 systemd[1]: Started Ceph mds.g for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:47:15.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:14 smithi002 ceph-mon[128712]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:15.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:14 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2256152299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:15.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:14 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:14 smithi141 ceph-mon[131184]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:14 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2256152299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:14 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:15.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:14 smithi155 ceph-mon[131607]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:15.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:14 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2256152299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:15.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:14 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:16.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:15 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:16.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:15 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.2:6838/2583261818,v1:172.21.15.2:6839/2583261818] up:boot 2023-11-16T16:47:16.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:15 smithi141 ceph-mon[131184]: fsmap 10 up:standby 2023-11-16T16:47:16.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:15 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T16:47:16.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:15 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:16.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:15 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:16.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:15 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:16.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:15 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:16.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:15 smithi141 ceph-mon[131184]: Deploying daemon mds.a on smithi002 2023-11-16T16:47:16.351 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:15 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:16.351 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:15 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.2:6838/2583261818,v1:172.21.15.2:6839/2583261818] up:boot 2023-11-16T16:47:16.351 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:15 smithi002 ceph-mon[128712]: fsmap 10 up:standby 2023-11-16T16:47:16.351 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:15 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T16:47:16.351 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:15 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:16.351 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:15 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:16.351 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:15 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:16.351 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:15 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:16.351 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:15 smithi002 ceph-mon[128712]: Deploying daemon mds.a on smithi002 2023-11-16T16:47:16.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:15 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:16.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:15 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.2:6838/2583261818,v1:172.21.15.2:6839/2583261818] up:boot 2023-11-16T16:47:16.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:15 smithi155 ceph-mon[131607]: fsmap 10 up:standby 2023-11-16T16:47:16.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:15 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T16:47:16.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:15 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:16.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:15 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:16.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:15 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:16.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:15 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:16.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:15 smithi155 ceph-mon[131607]: Deploying daemon mds.a on smithi002 2023-11-16T16:47:16.459 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:16.693 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.692+0000 7f3038dff700 1 -- 172.21.15.2:0/3941120725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3034074b30 msgr2=0x7f3034072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:16.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.692+0000 7f3038dff700 1 --2- 172.21.15.2:0/3941120725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3034074b30 0x7f3034072f80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3024009990 tx=0x7f3024009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.692+0000 7f3038dff700 1 -- 172.21.15.2:0/3941120725 shutdown_connections 2023-11-16T16:47:16.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.692+0000 7f3038dff700 1 --2- 172.21.15.2:0/3941120725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f303410c860 0x7f303410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.692+0000 7f3038dff700 1 --2- 172.21.15.2:0/3941120725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3034074b30 0x7f3034072f80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.692+0000 7f3038dff700 1 --2- 172.21.15.2:0/3941120725 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3034073550 0x7f30340739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.692+0000 7f3038dff700 1 -- 172.21.15.2:0/3941120725 >> 172.21.15.2:0/3941120725 conn(0x7f30340fb870 msgr2=0x7f30340fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:16.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.692+0000 7f3038dff700 1 -- 172.21.15.2:0/3941120725 shutdown_connections 2023-11-16T16:47:16.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.692+0000 7f3038dff700 1 -- 172.21.15.2:0/3941120725 wait complete. 2023-11-16T16:47:16.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.693+0000 7f3038dff700 1 Processor -- start 2023-11-16T16:47:16.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.693+0000 7f3038dff700 1 -- start start 2023-11-16T16:47:16.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.694+0000 7f3038dff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3034073550 0x7f303419c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:16.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.694+0000 7f3038dff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3034074b30 0x7f303419cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:16.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.694+0000 7f3038dff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f303410c860 0x7f30341a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:16.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.694+0000 7f3038dff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30341a4180 con 0x7f3034073550 2023-11-16T16:47:16.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.694+0000 7f3038dff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30341a42c0 con 0x7f303410c860 2023-11-16T16:47:16.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.694+0000 7f3038dff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30341a4450 con 0x7f3034074b30 2023-11-16T16:47:16.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.694+0000 7f303259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3034073550 0x7f303419c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:16.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.694+0000 7f303259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3034073550 0x7f303419c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39238/0 (socket says 172.21.15.2:39238) 2023-11-16T16:47:16.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.694+0000 7f303259c700 1 -- 172.21.15.2:0/334758509 learned_addr learned my addr 172.21.15.2:0/334758509 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:16.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.694+0000 7f3031d9b700 1 --2- 172.21.15.2:0/334758509 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3034074b30 0x7f303419cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:16.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.694+0000 7f3032d9d700 1 --2- 172.21.15.2:0/334758509 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f303410c860 0x7f30341a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:16.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.694+0000 7f303259c700 1 -- 172.21.15.2:0/334758509 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3034074b30 msgr2=0x7f303419cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:16.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.694+0000 7f303259c700 1 --2- 172.21.15.2:0/334758509 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3034074b30 0x7f303419cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.694+0000 7f303259c700 1 -- 172.21.15.2:0/334758509 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f303410c860 msgr2=0x7f30341a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:16.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.694+0000 7f303259c700 1 --2- 172.21.15.2:0/334758509 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f303410c860 0x7f30341a3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.694+0000 7f303259c700 1 -- 172.21.15.2:0/334758509 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30240095f0 con 0x7f3034073550 2023-11-16T16:47:16.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.694+0000 7f3032d9d700 1 --2- 172.21.15.2:0/334758509 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f303410c860 0x7f30341a3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:16.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.695+0000 7f303259c700 1 --2- 172.21.15.2:0/334758509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3034073550 0x7f303419c600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3024005720 tx=0x7f3024009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:16.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.695+0000 7f30237fe700 1 -- 172.21.15.2:0/334758509 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302401d070 con 0x7f3034073550 2023-11-16T16:47:16.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.695+0000 7f30237fe700 1 -- 172.21.15.2:0/334758509 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3024021410 con 0x7f3034073550 2023-11-16T16:47:16.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.695+0000 7f30237fe700 1 -- 172.21.15.2:0/334758509 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3024017650 con 0x7f3034073550 2023-11-16T16:47:16.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.695+0000 7f3038dff700 1 -- 172.21.15.2:0/334758509 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3034112250 con 0x7f3034073550 2023-11-16T16:47:16.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.695+0000 7f3038dff700 1 -- 172.21.15.2:0/334758509 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30341120c0 con 0x7f3034073550 2023-11-16T16:47:16.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.696+0000 7f3038dff700 1 -- 172.21.15.2:0/334758509 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30341023b0 con 0x7f3034073550 2023-11-16T16:47:16.700 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.698+0000 7f30237fe700 1 -- 172.21.15.2:0/334758509 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f3024021580 con 0x7f3034073550 2023-11-16T16:47:16.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.699+0000 7f30237fe700 1 --2- 172.21.15.2:0/334758509 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f301c0a1150 0x7f301c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:16.701 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.700+0000 7f30237fe700 1 -- 172.21.15.2:0/334758509 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f30240cdc30 con 0x7f3034073550 2023-11-16T16:47:16.702 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.700+0000 7f3031d9b700 1 --2- 172.21.15.2:0/334758509 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f301c0a1150 0x7f301c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:16.702 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.700+0000 7f3031d9b700 1 --2- 172.21.15.2:0/334758509 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f301c0a1150 0x7f301c0a3600 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3034074850 tx=0x7f3028009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:16.703 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.701+0000 7f30237fe700 1 -- 172.21.15.2:0/334758509 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3024097810 con 0x7f3034073550 2023-11-16T16:47:16.980 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.978+0000 7f3038dff700 1 -- 172.21.15.2:0/334758509 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3034061700 con 0x7f3034073550 2023-11-16T16:47:16.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.979+0000 7f30237fe700 1 -- 172.21.15.2:0/334758509 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8085 (secure 0 0 0) 0x7f302409ae30 con 0x7f3034073550 2023-11-16T16:47:16.981 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:16.981 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/1610211225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":1610211225},{"type":"v1","addr":"172.21.15.2:6837","nonce":1610211225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14538,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2583261818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2583261818},{"type":"v1","addr":"172.21.15.2:6839","nonce":2583261818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3516869188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3516869188},{"type":"v1","addr":"172.21.15.141:6835","nonce":3516869188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2527313191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":2527313191},{"type":"v1","addr":"172.21.15.155:6835","nonce":2527313191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1607756751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1607756751},{"type":"v1","addr":"172.21.15.155:6837","nonce":1607756751}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6837/2934891475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6836","nonce":2934891475},{"type":"v1","addr":"172.21.15.141:6837","nonce":2934891475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/1158457009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1158457009},{"type":"v1","addr":"172.21.15.155:6839","nonce":1158457009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-11-16T16:47:16.983 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.981+0000 7f3038dff700 1 -- 172.21.15.2:0/334758509 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f301c0a1150 msgr2=0x7f301c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:16.983 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.981+0000 7f3038dff700 1 --2- 172.21.15.2:0/334758509 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f301c0a1150 0x7f301c0a3600 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3034074850 tx=0x7f3028009260 comp rx=0 tx=0).stop 2023-11-16T16:47:16.983 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.981+0000 7f3038dff700 1 -- 172.21.15.2:0/334758509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3034073550 msgr2=0x7f303419c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:16.983 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.981+0000 7f3038dff700 1 --2- 172.21.15.2:0/334758509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3034073550 0x7f303419c600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3024005720 tx=0x7f3024009580 comp rx=0 tx=0).stop 2023-11-16T16:47:16.984 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.981+0000 7f3038dff700 1 -- 172.21.15.2:0/334758509 shutdown_connections 2023-11-16T16:47:16.984 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.981+0000 7f3038dff700 1 --2- 172.21.15.2:0/334758509 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f301c0a1150 0x7f301c0a3600 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.984 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.981+0000 7f3038dff700 1 --2- 172.21.15.2:0/334758509 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f303410c860 0x7f30341a3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.984 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.981+0000 7f3038dff700 1 --2- 172.21.15.2:0/334758509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3034073550 0x7f303419c600 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.984 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.981+0000 7f3038dff700 1 --2- 172.21.15.2:0/334758509 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3034074b30 0x7f303419cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:16.984 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.981+0000 7f3038dff700 1 -- 172.21.15.2:0/334758509 >> 172.21.15.2:0/334758509 conn(0x7f30340fb870 msgr2=0x7f30340fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:16.984 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.981+0000 7f3038dff700 1 -- 172.21.15.2:0/334758509 shutdown_connections 2023-11-16T16:47:16.984 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:16.982+0000 7f3038dff700 1 -- 172.21.15.2:0/334758509 wait complete. 2023-11-16T16:47:16.984 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 11 2023-11-16T16:47:17.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:17 smithi002 ceph-mon[128712]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:17.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:17 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/334758509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:17.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:17 smithi155 ceph-mon[131607]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:17.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:17 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/334758509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:17.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:17 smithi141 ceph-mon[131184]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:17.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:17 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/334758509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:17.655 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 16:47:17 smithi002 systemd[1]: Starting Ceph mds.a for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:47:18.048 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 16:47:17 smithi002 podman[162949]: 2023-11-16T16:47:18.405 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 16:47:18 smithi002 bash[162949]: 816b8eda420f17e19ca9d25b0b7ad17bd9a7d3d8bfb8f0c44d418959da52c08b 2023-11-16T16:47:18.406 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 16:47:18 smithi002 ceph-mds[162989]: -- 172.21.15.2:0/2886825181 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564198595520 con 0x56419856cc00 2023-11-16T16:47:18.406 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 16:47:18 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-a[162965]: starting mds.a at 2023-11-16T16:47:18.406 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 16:47:18 smithi002 systemd[1]: Started Ceph mds.a for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:47:19.010 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:19.234 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.232+0000 7fe9293ff700 1 -- 172.21.15.2:0/1098647362 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe924101e30 msgr2=0x7fe9241086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:19.234 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.232+0000 7fe9293ff700 1 --2- 172.21.15.2:0/1098647362 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe924101e30 0x7fe9241086f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe91c00b0a0 tx=0x7fe91c00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.232+0000 7fe9293ff700 1 -- 172.21.15.2:0/1098647362 shutdown_connections 2023-11-16T16:47:19.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.232+0000 7fe9293ff700 1 --2- 172.21.15.2:0/1098647362 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe924101e30 0x7fe9241086f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.232+0000 7fe9293ff700 1 --2- 172.21.15.2:0/1098647362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe9241002b0 0x7fe9241006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.232+0000 7fe9293ff700 1 --2- 172.21.15.2:0/1098647362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9241014a0 0x7fe9241018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.232+0000 7fe9293ff700 1 -- 172.21.15.2:0/1098647362 >> 172.21.15.2:0/1098647362 conn(0x7fe9240fb890 msgr2=0x7fe9240fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:19.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.232+0000 7fe9293ff700 1 -- 172.21.15.2:0/1098647362 shutdown_connections 2023-11-16T16:47:19.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.232+0000 7fe9293ff700 1 -- 172.21.15.2:0/1098647362 wait complete. 2023-11-16T16:47:19.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.233+0000 7fe9293ff700 1 Processor -- start 2023-11-16T16:47:19.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.233+0000 7fe9293ff700 1 -- start start 2023-11-16T16:47:19.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.233+0000 7fe9293ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9241002b0 0x7fe9241982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:19.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.234+0000 7fe9293ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9241014a0 0x7fe9241987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:19.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.234+0000 7fe9293ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe924101e30 0x7fe92419f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:19.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.234+0000 7fe9293ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92419fe30 con 0x7fe9241002b0 2023-11-16T16:47:19.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.234+0000 7fe9293ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92419ff70 con 0x7fe924101e30 2023-11-16T16:47:19.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.234+0000 7fe9293ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9241a0100 con 0x7fe9241014a0 2023-11-16T16:47:19.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.234+0000 7fe922ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9241002b0 0x7fe9241982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:19.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.234+0000 7fe922ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9241002b0 0x7fe9241982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39290/0 (socket says 172.21.15.2:39290) 2023-11-16T16:47:19.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.234+0000 7fe922ffd700 1 -- 172.21.15.2:0/1451521441 learned_addr learned my addr 172.21.15.2:0/1451521441 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:19.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.234+0000 7fe9237fe700 1 --2- 172.21.15.2:0/1451521441 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe924101e30 0x7fe92419f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:19.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.234+0000 7fe91bfff700 1 --2- 172.21.15.2:0/1451521441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9241014a0 0x7fe9241987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:19.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.234+0000 7fe922ffd700 1 -- 172.21.15.2:0/1451521441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9241014a0 msgr2=0x7fe9241987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:19.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.234+0000 7fe922ffd700 1 --2- 172.21.15.2:0/1451521441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9241014a0 0x7fe9241987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.234+0000 7fe922ffd700 1 -- 172.21.15.2:0/1451521441 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe924101e30 msgr2=0x7fe92419f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:19.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.234+0000 7fe922ffd700 1 --2- 172.21.15.2:0/1451521441 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe924101e30 0x7fe92419f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.234+0000 7fe922ffd700 1 -- 172.21.15.2:0/1451521441 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c009d00 con 0x7fe9241002b0 2023-11-16T16:47:19.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.234+0000 7fe922ffd700 1 --2- 172.21.15.2:0/1451521441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9241002b0 0x7fe9241982b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe91400e970 tx=0x7fe91400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:19.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.235+0000 7fe920ff9700 1 -- 172.21.15.2:0/1451521441 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe914009790 con 0x7fe9241002b0 2023-11-16T16:47:19.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.235+0000 7fe920ff9700 1 -- 172.21.15.2:0/1451521441 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe91400c7e0 con 0x7fe9241002b0 2023-11-16T16:47:19.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.235+0000 7fe9293ff700 1 -- 172.21.15.2:0/1451521441 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92410bc80 con 0x7fe9241002b0 2023-11-16T16:47:19.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.235+0000 7fe920ff9700 1 -- 172.21.15.2:0/1451521441 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe914003c70 con 0x7fe9241002b0 2023-11-16T16:47:19.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.235+0000 7fe9293ff700 1 -- 172.21.15.2:0/1451521441 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92410baf0 con 0x7fe9241002b0 2023-11-16T16:47:19.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.236+0000 7fe9293ff700 1 -- 172.21.15.2:0/1451521441 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9241a0850 con 0x7fe9241002b0 2023-11-16T16:47:19.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.238+0000 7fe920ff9700 1 -- 172.21.15.2:0/1451521441 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fe91400cc50 con 0x7fe9241002b0 2023-11-16T16:47:19.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.239+0000 7fe920ff9700 1 --2- 172.21.15.2:0/1451521441 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe9100a1150 0x7fe9100a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:19.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.239+0000 7fe920ff9700 1 -- 172.21.15.2:0/1451521441 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe914014070 con 0x7fe9241002b0 2023-11-16T16:47:19.241 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.240+0000 7fe91bfff700 1 --2- 172.21.15.2:0/1451521441 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe9100a1150 0x7fe9100a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:19.242 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.240+0000 7fe91bfff700 1 --2- 172.21.15.2:0/1451521441 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe9100a1150 0x7fe9100a3600 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe90c009c60 tx=0x7fe90c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:19.243 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.242+0000 7fe920ff9700 1 -- 172.21.15.2:0/1451521441 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe914096370 con 0x7fe9241002b0 2023-11-16T16:47:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:19 smithi141 ceph-mon[131184]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:19 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:19 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:19 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:19 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:19 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:19 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:19.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:19 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.2:6840/3170991923,v1:172.21.15.2:6841/3170991923] up:boot 2023-11-16T16:47:19.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:19 smithi141 ceph-mon[131184]: fsmap 11 up:standby 2023-11-16T16:47:19.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:19 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T16:47:19.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:19 smithi155 ceph-mon[131607]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:19.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:19 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:19.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:19 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:19.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:19 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:19.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:19 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:19.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:19 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:19.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:19 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:19.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:19 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.2:6840/3170991923,v1:172.21.15.2:6841/3170991923] up:boot 2023-11-16T16:47:19.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:19 smithi155 ceph-mon[131607]: fsmap 11 up:standby 2023-11-16T16:47:19.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:19 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T16:47:19.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:19 smithi002 ceph-mon[128712]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:19.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:19 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:19.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:19 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:19.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:19 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:19.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:19 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-11-16T16:47:19.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:19 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-11-16T16:47:19.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:19 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:19.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:19 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.2:6840/3170991923,v1:172.21.15.2:6841/3170991923] up:boot 2023-11-16T16:47:19.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:19 smithi002 ceph-mon[128712]: fsmap 11 up:standby 2023-11-16T16:47:19.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:19 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T16:47:19.499 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.498+0000 7fe9293ff700 1 -- 172.21.15.2:0/1451521441 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9241a0e00 con 0x7fe9241002b0 2023-11-16T16:47:19.500 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.499+0000 7fe920ff9700 1 -- 172.21.15.2:0/1451521441 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8840 (secure 0 0 0) 0x7fe914099990 con 0x7fe9241002b0 2023-11-16T16:47:19.500 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:19.501 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/1610211225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":1610211225},{"type":"v1","addr":"172.21.15.2:6837","nonce":1610211225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14538,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2583261818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2583261818},{"type":"v1","addr":"172.21.15.2:6839","nonce":2583261818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14550,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3170991923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3170991923},{"type":"v1","addr":"172.21.15.2:6841","nonce":3170991923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3516869188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3516869188},{"type":"v1","addr":"172.21.15.141:6835","nonce":3516869188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2527313191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":2527313191},{"type":"v1","addr":"172.21.15.155:6835","nonce":2527313191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1607756751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1607756751},{"type":"v1","addr":"172.21.15.155:6837","nonce":1607756751}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6837/2934891475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6836","nonce":2934891475},{"type":"v1","addr":"172.21.15.141:6837","nonce":2934891475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/1158457009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1158457009},{"type":"v1","addr":"172.21.15.155:6839","nonce":1158457009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-11-16T16:47:19.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.501+0000 7fe9293ff700 1 -- 172.21.15.2:0/1451521441 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe9100a1150 msgr2=0x7fe9100a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:19.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.501+0000 7fe9293ff700 1 --2- 172.21.15.2:0/1451521441 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe9100a1150 0x7fe9100a3600 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe90c009c60 tx=0x7fe90c009260 comp rx=0 tx=0).stop 2023-11-16T16:47:19.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.501+0000 7fe9293ff700 1 -- 172.21.15.2:0/1451521441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9241002b0 msgr2=0x7fe9241982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:19.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.501+0000 7fe9293ff700 1 --2- 172.21.15.2:0/1451521441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9241002b0 0x7fe9241982b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe91400e970 tx=0x7fe91400ec80 comp rx=0 tx=0).stop 2023-11-16T16:47:19.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.501+0000 7fe9293ff700 1 -- 172.21.15.2:0/1451521441 shutdown_connections 2023-11-16T16:47:19.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.501+0000 7fe9293ff700 1 --2- 172.21.15.2:0/1451521441 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe9100a1150 0x7fe9100a3600 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.501+0000 7fe9293ff700 1 --2- 172.21.15.2:0/1451521441 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe924101e30 0x7fe92419f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.501+0000 7fe9293ff700 1 --2- 172.21.15.2:0/1451521441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9241002b0 0x7fe9241982b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.501+0000 7fe9293ff700 1 --2- 172.21.15.2:0/1451521441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe9241014a0 0x7fe9241987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:19.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.501+0000 7fe9293ff700 1 -- 172.21.15.2:0/1451521441 >> 172.21.15.2:0/1451521441 conn(0x7fe9240fb890 msgr2=0x7fe924106d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:19.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.501+0000 7fe9293ff700 1 -- 172.21.15.2:0/1451521441 shutdown_connections 2023-11-16T16:47:19.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:19.501+0000 7fe9293ff700 1 -- 172.21.15.2:0/1451521441 wait complete. 2023-11-16T16:47:19.504 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 12 2023-11-16T16:47:20.015 INFO:journalctl@ceph.mds.b.smithi141.stdout:Nov 16 16:47:19 smithi141 systemd[1]: Starting Ceph mds.b for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:47:20.299 INFO:journalctl@ceph.mds.b.smithi141.stdout:Nov 16 16:47:20 smithi141 podman[156398]: 2023-11-16T16:47:20.299 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:20 smithi141 ceph-mon[131184]: Deploying daemon mds.b on smithi141 2023-11-16T16:47:20.299 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:20 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1451521441' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:20.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:20 smithi155 ceph-mon[131607]: Deploying daemon mds.b on smithi141 2023-11-16T16:47:20.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:20 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1451521441' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:20.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:20 smithi002 ceph-mon[128712]: Deploying daemon mds.b on smithi141 2023-11-16T16:47:20.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:20 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1451521441' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:20.764 INFO:journalctl@ceph.mds.b.smithi141.stdout:Nov 16 16:47:20 smithi141 bash[156398]: 971ae64327c4b505583578c91ed1973da65a219e2af6ca13af329b382298adc0 2023-11-16T16:47:20.765 INFO:journalctl@ceph.mds.b.smithi141.stdout:Nov 16 16:47:20 smithi141 ceph-mds[156438]: -- 172.21.15.141:0/1233588564 <== mon.2 v2:172.21.15.141:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5586df16d520 con 0x5586df142c00 2023-11-16T16:47:20.765 INFO:journalctl@ceph.mds.b.smithi141.stdout:Nov 16 16:47:20 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-b[156414]: starting mds.b at 2023-11-16T16:47:20.765 INFO:journalctl@ceph.mds.b.smithi141.stdout:Nov 16 16:47:20 smithi141 systemd[1]: Started Ceph mds.b for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:47:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:21 smithi155 ceph-mon[131607]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:21 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:21 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:21 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:21 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:21 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:21 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:47:21.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:21 smithi002 ceph-mon[128712]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:21.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:21 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:21.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:21 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:21.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:21 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:21.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:21 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:21.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:21 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:21.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:21 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:47:21.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:21 smithi141 ceph-mon[131184]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:21.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:21 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:21.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:21 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:21.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:21 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:21.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:21 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:21.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:21 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:21.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:21 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:47:21.523 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:21.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.755+0000 7f40393ff700 1 -- 172.21.15.2:0/234575709 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40340feb80 msgr2=0x7f4034100fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:21.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.756+0000 7f40393ff700 1 --2- 172.21.15.2:0/234575709 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40340feb80 0x7f4034100fa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4024009990 tx=0x7f4024009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:21.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.756+0000 7f40393ff700 1 -- 172.21.15.2:0/234575709 shutdown_connections 2023-11-16T16:47:21.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.756+0000 7f40393ff700 1 --2- 172.21.15.2:0/234575709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4034103ea0 0x7f4034104320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:21.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.756+0000 7f40393ff700 1 --2- 172.21.15.2:0/234575709 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40340feb80 0x7f4034100fa0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:21.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.756+0000 7f40393ff700 1 --2- 172.21.15.2:0/234575709 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40341014e0 0x7f4034103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:21.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.756+0000 7f40393ff700 1 -- 172.21.15.2:0/234575709 >> 172.21.15.2:0/234575709 conn(0x7f40340fa750 msgr2=0x7f40340fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:21.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.756+0000 7f40393ff700 1 -- 172.21.15.2:0/234575709 shutdown_connections 2023-11-16T16:47:21.758 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.756+0000 7f40393ff700 1 -- 172.21.15.2:0/234575709 wait complete. 2023-11-16T16:47:21.760 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.757+0000 7f40393ff700 1 Processor -- start 2023-11-16T16:47:21.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.757+0000 7f40393ff700 1 -- start start 2023-11-16T16:47:21.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.757+0000 7f40393ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40340feb80 0x7f403419c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:21.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.757+0000 7f40393ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40341014e0 0x7f403419caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:21.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.757+0000 7f40393ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4034103ea0 0x7f40341a3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:21.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.757+0000 7f40393ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40341a4130 con 0x7f4034103ea0 2023-11-16T16:47:21.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.757+0000 7f40393ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40341a4270 con 0x7f40340feb80 2023-11-16T16:47:21.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.757+0000 7f40393ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40341a4400 con 0x7f40341014e0 2023-11-16T16:47:21.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.757+0000 7f4032ffd700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40340feb80 0x7f403419c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:21.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.757+0000 7f4032ffd700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40340feb80 0x7f403419c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:45998/0 (socket says 172.21.15.2:45998) 2023-11-16T16:47:21.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.757+0000 7f4032ffd700 1 -- 172.21.15.2:0/1432076778 learned_addr learned my addr 172.21.15.2:0/1432076778 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:21.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.758+0000 7f40327fc700 1 --2- 172.21.15.2:0/1432076778 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40341014e0 0x7f403419caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:21.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.758+0000 7f40337fe700 1 --2- 172.21.15.2:0/1432076778 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4034103ea0 0x7f40341a3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:21.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.758+0000 7f4032ffd700 1 -- 172.21.15.2:0/1432076778 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40341014e0 msgr2=0x7f403419caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:21.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.758+0000 7f4032ffd700 1 --2- 172.21.15.2:0/1432076778 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40341014e0 0x7f403419caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:21.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.758+0000 7f4032ffd700 1 -- 172.21.15.2:0/1432076778 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4034103ea0 msgr2=0x7f40341a3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:21.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.758+0000 7f4032ffd700 1 --2- 172.21.15.2:0/1432076778 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4034103ea0 0x7f40341a3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:21.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.758+0000 7f4032ffd700 1 -- 172.21.15.2:0/1432076778 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40240095f0 con 0x7f40340feb80 2023-11-16T16:47:21.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.758+0000 7f4032ffd700 1 --2- 172.21.15.2:0/1432076778 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40340feb80 0x7f403419c5b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f402400b680 tx=0x7f4024005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:21.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.758+0000 7f401bfff700 1 -- 172.21.15.2:0/1432076778 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402401d070 con 0x7f40340feb80 2023-11-16T16:47:21.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.759+0000 7f401bfff700 1 -- 172.21.15.2:0/1432076778 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4024021410 con 0x7f40340feb80 2023-11-16T16:47:21.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.759+0000 7f40393ff700 1 -- 172.21.15.2:0/1432076778 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4034112230 con 0x7f40340feb80 2023-11-16T16:47:21.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.759+0000 7f401bfff700 1 -- 172.21.15.2:0/1432076778 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4024017650 con 0x7f40340feb80 2023-11-16T16:47:21.762 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.759+0000 7f40393ff700 1 -- 172.21.15.2:0/1432076778 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40341120a0 con 0x7f40340feb80 2023-11-16T16:47:21.763 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.760+0000 7f40393ff700 1 -- 172.21.15.2:0/1432076778 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40340fc2c0 con 0x7f40340feb80 2023-11-16T16:47:21.763 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.762+0000 7f401bfff700 1 -- 172.21.15.2:0/1432076778 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f4024021580 con 0x7f40340feb80 2023-11-16T16:47:21.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.763+0000 7f401bfff700 1 --2- 172.21.15.2:0/1432076778 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f401c0a11d0 0x7f401c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:21.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.763+0000 7f401bfff700 1 -- 172.21.15.2:0/1432076778 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f40240cda30 con 0x7f40340feb80 2023-11-16T16:47:21.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.763+0000 7f40327fc700 1 --2- 172.21.15.2:0/1432076778 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f401c0a11d0 0x7f401c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:21.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.764+0000 7f40327fc700 1 --2- 172.21.15.2:0/1432076778 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f401c0a11d0 0x7f401c0a3680 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4028007c60 tx=0x7f40280073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:21.767 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:21.765+0000 7f401bfff700 1 -- 172.21.15.2:0/1432076778 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4024097610 con 0x7f40340feb80 2023-11-16T16:47:22.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.028+0000 7f40393ff700 1 -- 172.21.15.2:0/1432076778 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f403402cfe0 con 0x7f40340feb80 2023-11-16T16:47:22.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.028+0000 7f401bfff700 1 -- 172.21.15.2:0/1432076778 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9601 (secure 0 0 0) 0x7f402409ac30 con 0x7f40340feb80 2023-11-16T16:47:22.030 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:22.031 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/1610211225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":1610211225},{"type":"v1","addr":"172.21.15.2:6837","nonce":1610211225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14538,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2583261818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2583261818},{"type":"v1","addr":"172.21.15.2:6839","nonce":2583261818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14550,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3170991923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3170991923},{"type":"v1","addr":"172.21.15.2:6841","nonce":3170991923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3516869188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3516869188},{"type":"v1","addr":"172.21.15.141:6835","nonce":3516869188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2527313191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":2527313191},{"type":"v1","addr":"172.21.15.155:6835","nonce":2527313191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1607756751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1607756751},{"type":"v1","addr":"172.21.15.155:6837","nonce":1607756751}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6837/2934891475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6836","nonce":2934891475},{"type":"v1","addr":"172.21.15.141:6837","nonce":2934891475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/1158457009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1158457009},{"type":"v1","addr":"172.21.15.155:6839","nonce":1158457009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24590,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6839/3916358272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6838","nonce":3916358272},{"type":"v1","addr":"172.21.15.141:6839","nonce":3916358272}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-11-16T16:47:22.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.031+0000 7f40393ff700 1 -- 172.21.15.2:0/1432076778 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f401c0a11d0 msgr2=0x7f401c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:22.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.031+0000 7f40393ff700 1 --2- 172.21.15.2:0/1432076778 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f401c0a11d0 0x7f401c0a3680 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4028007c60 tx=0x7f40280073d0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.031+0000 7f40393ff700 1 -- 172.21.15.2:0/1432076778 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40340feb80 msgr2=0x7f403419c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:22.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.031+0000 7f40393ff700 1 --2- 172.21.15.2:0/1432076778 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40340feb80 0x7f403419c5b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f402400b680 tx=0x7f4024005740 comp rx=0 tx=0).stop 2023-11-16T16:47:22.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.031+0000 7f40393ff700 1 -- 172.21.15.2:0/1432076778 shutdown_connections 2023-11-16T16:47:22.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.031+0000 7f40393ff700 1 --2- 172.21.15.2:0/1432076778 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f401c0a11d0 0x7f401c0a3680 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.031+0000 7f40393ff700 1 --2- 172.21.15.2:0/1432076778 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4034103ea0 0x7f40341a3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.031+0000 7f40393ff700 1 --2- 172.21.15.2:0/1432076778 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f40340feb80 0x7f403419c5b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.031+0000 7f40393ff700 1 --2- 172.21.15.2:0/1432076778 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40341014e0 0x7f403419caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.031+0000 7f40393ff700 1 -- 172.21.15.2:0/1432076778 >> 172.21.15.2:0/1432076778 conn(0x7f40340fa750 msgr2=0x7f40341021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:22.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.031+0000 7f40393ff700 1 -- 172.21.15.2:0/1432076778 shutdown_connections 2023-11-16T16:47:22.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.031+0000 7f40393ff700 1 -- 172.21.15.2:0/1432076778 wait complete. 2023-11-16T16:47:22.034 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 13 2023-11-16T16:47:22.055 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:47:22.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.269+0000 7fc29fbff700 1 -- 172.21.15.2:0/3413106017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc298100290 msgr2=0x7fc2981006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:22.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.269+0000 7fc29fbff700 1 --2- 172.21.15.2:0/3413106017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc298100290 0x7fc2981006a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc294009990 tx=0x7fc294009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.269+0000 7fc29fbff700 1 -- 172.21.15.2:0/3413106017 shutdown_connections 2023-11-16T16:47:22.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.269+0000 7fc29fbff700 1 --2- 172.21.15.2:0/3413106017 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc298101e10 0x7fc2981086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.269+0000 7fc29fbff700 1 --2- 172.21.15.2:0/3413106017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc298100290 0x7fc2981006a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.269+0000 7fc29fbff700 1 --2- 172.21.15.2:0/3413106017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc298101480 0x7fc2981018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.269+0000 7fc29fbff700 1 -- 172.21.15.2:0/3413106017 >> 172.21.15.2:0/3413106017 conn(0x7fc2980fb870 msgr2=0x7fc2980fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:22.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.269+0000 7fc29fbff700 1 -- 172.21.15.2:0/3413106017 shutdown_connections 2023-11-16T16:47:22.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.269+0000 7fc29fbff700 1 -- 172.21.15.2:0/3413106017 wait complete. 2023-11-16T16:47:22.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.270+0000 7fc29fbff700 1 Processor -- start 2023-11-16T16:47:22.275 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.270+0000 7fc29fbff700 1 -- start start 2023-11-16T16:47:22.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.270+0000 7fc29fbff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc298100290 0x7fc2981981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:22.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.270+0000 7fc29fbff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc298101480 0x7fc298198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:22.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.270+0000 7fc29fbff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc298101e10 0x7fc29819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:22.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.270+0000 7fc29fbff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29819fd50 con 0x7fc298101e10 2023-11-16T16:47:22.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.270+0000 7fc29fbff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29819fe90 con 0x7fc298101480 2023-11-16T16:47:22.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.270+0000 7fc29fbff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2981a0020 con 0x7fc298100290 2023-11-16T16:47:22.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.271+0000 7fc29d19a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc298101480 0x7fc298198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:22.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.271+0000 7fc29e19c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc298101e10 0x7fc29819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:22.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.271+0000 7fc29d19a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc298101480 0x7fc298198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:46008/0 (socket says 172.21.15.2:46008) 2023-11-16T16:47:22.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.271+0000 7fc29d19a700 1 -- 172.21.15.2:0/225568116 learned_addr learned my addr 172.21.15.2:0/225568116 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:22.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.271+0000 7fc29d99b700 1 --2- 172.21.15.2:0/225568116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc298100290 0x7fc2981981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:22.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.271+0000 7fc29d19a700 1 -- 172.21.15.2:0/225568116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc298100290 msgr2=0x7fc2981981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:22.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.271+0000 7fc29d19a700 1 --2- 172.21.15.2:0/225568116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc298100290 0x7fc2981981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.271+0000 7fc29d19a700 1 -- 172.21.15.2:0/225568116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc298101e10 msgr2=0x7fc29819f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:22.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.271+0000 7fc29d19a700 1 --2- 172.21.15.2:0/225568116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc298101e10 0x7fc29819f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.271+0000 7fc29d19a700 1 -- 172.21.15.2:0/225568116 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2940095f0 con 0x7fc298101480 2023-11-16T16:47:22.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.271+0000 7fc29e19c700 1 --2- 172.21.15.2:0/225568116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc298101e10 0x7fc29819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:22.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.271+0000 7fc29d19a700 1 --2- 172.21.15.2:0/225568116 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc298101480 0x7fc298198710 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc28800e9a0 tx=0x7fc28800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:22.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.272+0000 7fc28effd700 1 -- 172.21.15.2:0/225568116 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288009790 con 0x7fc298101480 2023-11-16T16:47:22.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.272+0000 7fc28effd700 1 -- 172.21.15.2:0/225568116 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc288010430 con 0x7fc298101480 2023-11-16T16:47:22.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.272+0000 7fc29fbff700 1 -- 172.21.15.2:0/225568116 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc29810bc40 con 0x7fc298101480 2023-11-16T16:47:22.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.272+0000 7fc28effd700 1 -- 172.21.15.2:0/225568116 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288018680 con 0x7fc298101480 2023-11-16T16:47:22.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.272+0000 7fc29fbff700 1 -- 172.21.15.2:0/225568116 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29810bab0 con 0x7fc298101480 2023-11-16T16:47:22.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.273+0000 7fc29fbff700 1 -- 172.21.15.2:0/225568116 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2981928d0 con 0x7fc298101480 2023-11-16T16:47:22.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.275+0000 7fc28effd700 1 -- 172.21.15.2:0/225568116 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fc2880105a0 con 0x7fc298101480 2023-11-16T16:47:22.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.276+0000 7fc28effd700 1 --2- 172.21.15.2:0/225568116 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2840a11d0 0x7fc2840a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:22.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.276+0000 7fc28effd700 1 -- 172.21.15.2:0/225568116 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc288014070 con 0x7fc298101480 2023-11-16T16:47:22.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.277+0000 7fc29d99b700 1 --2- 172.21.15.2:0/225568116 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2840a11d0 0x7fc2840a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:22.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.277+0000 7fc29d99b700 1 --2- 172.21.15.2:0/225568116 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2840a11d0 0x7fc2840a3680 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc29400b2e0 tx=0x7fc294005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:22.281 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.280+0000 7fc28effd700 1 -- 172.21.15.2:0/225568116 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc288051730 con 0x7fc298101480 2023-11-16T16:47:22.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:22 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] up:boot 2023-11-16T16:47:22.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:22 smithi155 ceph-mon[131607]: fsmap 12 up:standby 2023-11-16T16:47:22.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:22 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T16:47:22.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:22 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1432076778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:22.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:22 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] up:boot 2023-11-16T16:47:22.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:22 smithi002 ceph-mon[128712]: fsmap 12 up:standby 2023-11-16T16:47:22.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:22 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T16:47:22.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:22 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1432076778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:22.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.497+0000 7fc29fbff700 1 -- 172.21.15.2:0/225568116 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc29802cfe0 con 0x7fc298101480 2023-11-16T16:47:22.500 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.498+0000 7fc28effd700 1 -- 172.21.15.2:0/225568116 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18374 (secure 0 0 0) 0x7fc2880961d0 con 0x7fc298101480 2023-11-16T16:47:22.500 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:22.500 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":81,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:46:37.203188+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:47:22.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.500+0000 7fc29fbff700 1 -- 172.21.15.2:0/225568116 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2840a11d0 msgr2=0x7fc2840a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:22.502 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.500+0000 7fc29fbff700 1 --2- 172.21.15.2:0/225568116 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2840a11d0 0x7fc2840a3680 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc29400b2e0 tx=0x7fc294005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.500+0000 7fc29fbff700 1 -- 172.21.15.2:0/225568116 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc298101480 msgr2=0x7fc298198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:22.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.500+0000 7fc29fbff700 1 --2- 172.21.15.2:0/225568116 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc298101480 0x7fc298198710 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc28800e9a0 tx=0x7fc28800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.501+0000 7fc29fbff700 1 -- 172.21.15.2:0/225568116 shutdown_connections 2023-11-16T16:47:22.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.501+0000 7fc29fbff700 1 --2- 172.21.15.2:0/225568116 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2840a11d0 0x7fc2840a3680 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.501+0000 7fc29fbff700 1 --2- 172.21.15.2:0/225568116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc298100290 0x7fc2981981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.501+0000 7fc29fbff700 1 --2- 172.21.15.2:0/225568116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc298101e10 0x7fc29819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.501+0000 7fc29fbff700 1 --2- 172.21.15.2:0/225568116 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc298101480 0x7fc298198710 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.501+0000 7fc29fbff700 1 -- 172.21.15.2:0/225568116 >> 172.21.15.2:0/225568116 conn(0x7fc2980fb870 msgr2=0x7fc298106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:22.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.501+0000 7fc29fbff700 1 -- 172.21.15.2:0/225568116 shutdown_connections 2023-11-16T16:47:22.503 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.501+0000 7fc29fbff700 1 -- 172.21.15.2:0/225568116 wait complete. 2023-11-16T16:47:22.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:22 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] up:boot 2023-11-16T16:47:22.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:22 smithi141 ceph-mon[131184]: fsmap 12 up:standby 2023-11-16T16:47:22.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:22 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T16:47:22.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:22 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1432076778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:22.523 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:04.306584+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-16T16:47:22.523 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:47:22.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.742+0000 7f2bebfff700 1 -- 172.21.15.2:0/863802827 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2be4104060 msgr2=0x7f2be410a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:22.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.742+0000 7f2bebfff700 1 --2- 172.21.15.2:0/863802827 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2be4104060 0x7f2be410a920 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00b0a0 tx=0x7f2bdc00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.742+0000 7f2bebfff700 1 -- 172.21.15.2:0/863802827 shutdown_connections 2023-11-16T16:47:22.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.742+0000 7f2bebfff700 1 --2- 172.21.15.2:0/863802827 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2be4104060 0x7f2be410a920 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.742+0000 7f2bebfff700 1 --2- 172.21.15.2:0/863802827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2be41024e0 0x7f2be41028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.742+0000 7f2bebfff700 1 --2- 172.21.15.2:0/863802827 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2be41036d0 0x7f2be4103b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.742+0000 7f2bebfff700 1 -- 172.21.15.2:0/863802827 >> 172.21.15.2:0/863802827 conn(0x7f2be40fdb20 msgr2=0x7f2be40fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:22.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.742+0000 7f2bebfff700 1 -- 172.21.15.2:0/863802827 shutdown_connections 2023-11-16T16:47:22.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.742+0000 7f2bebfff700 1 -- 172.21.15.2:0/863802827 wait complete. 2023-11-16T16:47:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.743+0000 7f2bebfff700 1 Processor -- start 2023-11-16T16:47:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.743+0000 7f2bebfff700 1 -- start start 2023-11-16T16:47:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.744+0000 7f2bebfff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2be41024e0 0x7f2be4198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.744+0000 7f2bebfff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2be41036d0 0x7f2be4198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.744+0000 7f2bebfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2be4104060 0x7f2be419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.744+0000 7f2bebfff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be419fcb0 con 0x7f2be4104060 2023-11-16T16:47:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.744+0000 7f2bebfff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be419fdf0 con 0x7f2be41024e0 2023-11-16T16:47:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.744+0000 7f2bebfff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be419ff80 con 0x7f2be41036d0 2023-11-16T16:47:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.744+0000 7f2be959a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2be41036d0 0x7f2be4198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.744+0000 7f2be9d9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2be41024e0 0x7f2be4198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.744+0000 7f2bea59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2be4104060 0x7f2be419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.744+0000 7f2be959a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2be41036d0 0x7f2be4198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:33206/0 (socket says 172.21.15.2:33206) 2023-11-16T16:47:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.744+0000 7f2be959a700 1 -- 172.21.15.2:0/1316425762 learned_addr learned my addr 172.21.15.2:0/1316425762 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.744+0000 7f2be959a700 1 -- 172.21.15.2:0/1316425762 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2be41024e0 msgr2=0x7f2be4198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.744+0000 7f2be959a700 1 --2- 172.21.15.2:0/1316425762 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2be41024e0 0x7f2be4198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.744+0000 7f2be959a700 1 -- 172.21.15.2:0/1316425762 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2be4104060 msgr2=0x7f2be419f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.744+0000 7f2be959a700 1 --2- 172.21.15.2:0/1316425762 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2be4104060 0x7f2be419f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.744+0000 7f2be959a700 1 -- 172.21.15.2:0/1316425762 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd4009520 con 0x7f2be41036d0 2023-11-16T16:47:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.744+0000 7f2bea59c700 1 --2- 172.21.15.2:0/1316425762 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2be4104060 0x7f2be419f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.745+0000 7f2be959a700 1 --2- 172.21.15.2:0/1316425762 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2be41036d0 0x7f2be4198670 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400eb30 tx=0x7f2bd400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.745+0000 7f2bdaffd700 1 -- 172.21.15.2:0/1316425762 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4009880 con 0x7f2be41036d0 2023-11-16T16:47:22.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.745+0000 7f2bdaffd700 1 -- 172.21.15.2:0/1316425762 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2bd40044e0 con 0x7f2be41036d0 2023-11-16T16:47:22.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.745+0000 7f2bdaffd700 1 -- 172.21.15.2:0/1316425762 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4003d80 con 0x7f2be41036d0 2023-11-16T16:47:22.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.745+0000 7f2bebfff700 1 -- 172.21.15.2:0/1316425762 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bdc009d00 con 0x7f2be41036d0 2023-11-16T16:47:22.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.745+0000 7f2bebfff700 1 -- 172.21.15.2:0/1316425762 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be410dde0 con 0x7f2be41036d0 2023-11-16T16:47:22.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.746+0000 7f2bebfff700 1 -- 172.21.15.2:0/1316425762 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be4067f50 con 0x7f2be41036d0 2023-11-16T16:47:22.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.748+0000 7f2bdaffd700 1 -- 172.21.15.2:0/1316425762 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f2bd4003f20 con 0x7f2be41036d0 2023-11-16T16:47:22.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.749+0000 7f2bdaffd700 1 --2- 172.21.15.2:0/1316425762 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2bd00a1150 0x7f2bd00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:22.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.750+0000 7f2bdaffd700 1 -- 172.21.15.2:0/1316425762 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2bd4014070 con 0x7f2be41036d0 2023-11-16T16:47:22.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.750+0000 7f2be9d9b700 1 --2- 172.21.15.2:0/1316425762 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2bd00a1150 0x7f2bd00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:22.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.750+0000 7f2be9d9b700 1 --2- 172.21.15.2:0/1316425762 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2bd00a1150 0x7f2bd00a3600 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2be4103530 tx=0x7f2be0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:22.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.752+0000 7f2bdaffd700 1 -- 172.21.15.2:0/1316425762 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2bd4096b70 con 0x7f2be41036d0 2023-11-16T16:47:22.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.964+0000 7f2bebfff700 1 -- 172.21.15.2:0/1316425762 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2be4109360 con 0x7f2be41036d0 2023-11-16T16:47:22.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.965+0000 7f2bdaffd700 1 -- 172.21.15.2:0/1316425762 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2bd40b8e60 con 0x7f2be41036d0 2023-11-16T16:47:22.966 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T16:47:22.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.967+0000 7f2bebfff700 1 -- 172.21.15.2:0/1316425762 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2bd00a1150 msgr2=0x7f2bd00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:22.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.967+0000 7f2bebfff700 1 --2- 172.21.15.2:0/1316425762 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2bd00a1150 0x7f2bd00a3600 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2be4103530 tx=0x7f2be0009260 comp rx=0 tx=0).stop 2023-11-16T16:47:22.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.967+0000 7f2bebfff700 1 -- 172.21.15.2:0/1316425762 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2be41036d0 msgr2=0x7f2be4198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:22.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.967+0000 7f2bebfff700 1 --2- 172.21.15.2:0/1316425762 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2be41036d0 0x7f2be4198670 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400eb30 tx=0x7f2bd400ee40 comp rx=0 tx=0).stop 2023-11-16T16:47:22.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.967+0000 7f2bebfff700 1 -- 172.21.15.2:0/1316425762 shutdown_connections 2023-11-16T16:47:22.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.967+0000 7f2bebfff700 1 --2- 172.21.15.2:0/1316425762 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2bd00a1150 0x7f2bd00a3600 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.967+0000 7f2bebfff700 1 --2- 172.21.15.2:0/1316425762 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2be4104060 0x7f2be419f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.967+0000 7f2bebfff700 1 --2- 172.21.15.2:0/1316425762 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2be41024e0 0x7f2be4198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.967+0000 7f2bebfff700 1 --2- 172.21.15.2:0/1316425762 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2be41036d0 0x7f2be4198670 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:22.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.967+0000 7f2bebfff700 1 -- 172.21.15.2:0/1316425762 >> 172.21.15.2:0/1316425762 conn(0x7f2be40fdb20 msgr2=0x7f2be40ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:22.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.967+0000 7f2bebfff700 1 -- 172.21.15.2:0/1316425762 shutdown_connections 2023-11-16T16:47:22.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:22.967+0000 7f2bebfff700 1 -- 172.21.15.2:0/1316425762 wait complete. 2023-11-16T16:47:22.988 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-16T16:47:23.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.197+0000 7f6b581ff700 1 -- 172.21.15.2:0/93091460 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b50100290 msgr2=0x7f6b501006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:23.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.197+0000 7f6b581ff700 1 --2- 172.21.15.2:0/93091460 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b50100290 0x7f6b501006a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c009990 tx=0x7f6b4c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.198+0000 7f6b581ff700 1 -- 172.21.15.2:0/93091460 shutdown_connections 2023-11-16T16:47:23.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.198+0000 7f6b581ff700 1 --2- 172.21.15.2:0/93091460 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b50101e10 0x7f6b501086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.198+0000 7f6b581ff700 1 --2- 172.21.15.2:0/93091460 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b50100290 0x7f6b501006a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.198+0000 7f6b581ff700 1 --2- 172.21.15.2:0/93091460 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b50101480 0x7f6b501018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.198+0000 7f6b581ff700 1 -- 172.21.15.2:0/93091460 >> 172.21.15.2:0/93091460 conn(0x7f6b500fb870 msgr2=0x7f6b500fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:23.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.198+0000 7f6b581ff700 1 -- 172.21.15.2:0/93091460 shutdown_connections 2023-11-16T16:47:23.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.198+0000 7f6b581ff700 1 -- 172.21.15.2:0/93091460 wait complete. 2023-11-16T16:47:23.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.199+0000 7f6b581ff700 1 Processor -- start 2023-11-16T16:47:23.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.199+0000 7f6b581ff700 1 -- start start 2023-11-16T16:47:23.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.199+0000 7f6b581ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b50100290 0x7f6b501981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:23.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.199+0000 7f6b581ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b50101480 0x7f6b50198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:23.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.199+0000 7f6b581ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b50101e10 0x7f6b5019f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:23.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.199+0000 7f6b581ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5019fd70 con 0x7f6b50100290 2023-11-16T16:47:23.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.199+0000 7f6b581ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5019feb0 con 0x7f6b50101480 2023-11-16T16:47:23.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.199+0000 7f6b581ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b501a0040 con 0x7f6b50101e10 2023-11-16T16:47:23.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.199+0000 7f6b5579a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b50101480 0x7f6b50198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:23.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.199+0000 7f6b55f9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b50100290 0x7f6b501981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:23.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.200+0000 7f6b5579a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b50101480 0x7f6b50198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:46040/0 (socket says 172.21.15.2:46040) 2023-11-16T16:47:23.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.200+0000 7f6b5579a700 1 -- 172.21.15.2:0/2395937648 learned_addr learned my addr 172.21.15.2:0/2395937648 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:23.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.200+0000 7f6b5679c700 1 --2- 172.21.15.2:0/2395937648 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b50101e10 0x7f6b5019f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:23.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.200+0000 7f6b5579a700 1 -- 172.21.15.2:0/2395937648 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b50101e10 msgr2=0x7f6b5019f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:23.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.200+0000 7f6b5579a700 1 --2- 172.21.15.2:0/2395937648 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b50101e10 0x7f6b5019f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.200+0000 7f6b5579a700 1 -- 172.21.15.2:0/2395937648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b50100290 msgr2=0x7f6b501981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:23.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.200+0000 7f6b5579a700 1 --2- 172.21.15.2:0/2395937648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b50100290 0x7f6b501981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.200+0000 7f6b5579a700 1 -- 172.21.15.2:0/2395937648 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4c0095f0 con 0x7f6b50101480 2023-11-16T16:47:23.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.200+0000 7f6b55f9b700 1 --2- 172.21.15.2:0/2395937648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b50100290 0x7f6b501981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:23.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.200+0000 7f6b5579a700 1 --2- 172.21.15.2:0/2395937648 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b50101480 0x7f6b50198730 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6b4000e9a0 tx=0x7f6b4000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:23.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.200+0000 7f6b46ffd700 1 -- 172.21.15.2:0/2395937648 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b40009790 con 0x7f6b50101480 2023-11-16T16:47:23.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.200+0000 7f6b581ff700 1 -- 172.21.15.2:0/2395937648 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b5010bc40 con 0x7f6b50101480 2023-11-16T16:47:23.204 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.200+0000 7f6b46ffd700 1 -- 172.21.15.2:0/2395937648 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b40010430 con 0x7f6b50101480 2023-11-16T16:47:23.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.200+0000 7f6b46ffd700 1 -- 172.21.15.2:0/2395937648 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b40018680 con 0x7f6b50101480 2023-11-16T16:47:23.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.201+0000 7f6b581ff700 1 -- 172.21.15.2:0/2395937648 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b5010bab0 con 0x7f6b50101480 2023-11-16T16:47:23.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.201+0000 7f6b581ff700 1 -- 172.21.15.2:0/2395937648 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b501928d0 con 0x7f6b50101480 2023-11-16T16:47:23.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.203+0000 7f6b46ffd700 1 -- 172.21.15.2:0/2395937648 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f6b400105a0 con 0x7f6b50101480 2023-11-16T16:47:23.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.205+0000 7f6b46ffd700 1 --2- 172.21.15.2:0/2395937648 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6b3c0a1150 0x7f6b3c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:23.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.205+0000 7f6b46ffd700 1 -- 172.21.15.2:0/2395937648 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6b40014070 con 0x7f6b50101480 2023-11-16T16:47:23.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.205+0000 7f6b55f9b700 1 --2- 172.21.15.2:0/2395937648 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6b3c0a1150 0x7f6b3c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:23.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.206+0000 7f6b55f9b700 1 --2- 172.21.15.2:0/2395937648 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6b3c0a1150 0x7f6b3c0a3600 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c00b2e0 tx=0x7f6b4c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:23.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.208+0000 7f6b46ffd700 1 -- 172.21.15.2:0/2395937648 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6b400961d0 con 0x7f6b50101480 2023-11-16T16:47:23.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:23 smithi155 ceph-mon[131607]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:23.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:23 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/225568116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:23.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:23 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1316425762' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:47:23.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:23 smithi002 ceph-mon[128712]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:23.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:23 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/225568116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:23.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:23 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1316425762' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:47:23.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.429+0000 7f6b581ff700 1 -- 172.21.15.2:0/2395937648 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6b50067a60 con 0x7f6b50101480 2023-11-16T16:47:23.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.430+0000 7f6b46ffd700 1 -- 172.21.15.2:0/2395937648 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7f6b400997f0 con 0x7f6b50101480 2023-11-16T16:47:23.432 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:23.432 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2023-11-16T16:47:23.432 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T16:47:23.432 INFO:teuthology.orchestra.run.smithi002.stdout: "poolnum": 1, 2023-11-16T16:47:23.432 INFO:teuthology.orchestra.run.smithi002.stdout: "poolname": ".mgr" 2023-11-16T16:47:23.432 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T16:47:23.432 INFO:teuthology.orchestra.run.smithi002.stdout:] 2023-11-16T16:47:23.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.432+0000 7f6b581ff700 1 -- 172.21.15.2:0/2395937648 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6b3c0a1150 msgr2=0x7f6b3c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:23.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.432+0000 7f6b581ff700 1 --2- 172.21.15.2:0/2395937648 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6b3c0a1150 0x7f6b3c0a3600 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c00b2e0 tx=0x7f6b4c005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.432+0000 7f6b581ff700 1 -- 172.21.15.2:0/2395937648 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b50101480 msgr2=0x7f6b50198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:23.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.432+0000 7f6b581ff700 1 --2- 172.21.15.2:0/2395937648 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b50101480 0x7f6b50198730 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6b4000e9a0 tx=0x7f6b4000ecb0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.432+0000 7f6b581ff700 1 -- 172.21.15.2:0/2395937648 shutdown_connections 2023-11-16T16:47:23.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.432+0000 7f6b581ff700 1 --2- 172.21.15.2:0/2395937648 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6b3c0a1150 0x7f6b3c0a3600 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c00b2e0 tx=0x7f6b4c005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.435 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.432+0000 7f6b581ff700 1 --2- 172.21.15.2:0/2395937648 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b50101e10 0x7f6b5019f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.435 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.432+0000 7f6b581ff700 1 --2- 172.21.15.2:0/2395937648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b50100290 0x7f6b501981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.435 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.432+0000 7f6b581ff700 1 --2- 172.21.15.2:0/2395937648 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6b50101480 0x7f6b50198730 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.435 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.432+0000 7f6b581ff700 1 -- 172.21.15.2:0/2395937648 >> 172.21.15.2:0/2395937648 conn(0x7f6b500fb870 msgr2=0x7f6b50106e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:23.435 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.432+0000 7f6b581ff700 1 -- 172.21.15.2:0/2395937648 shutdown_connections 2023-11-16T16:47:23.435 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.432+0000 7f6b581ff700 1 -- 172.21.15.2:0/2395937648 wait complete. 2023-11-16T16:47:23.458 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-16T16:47:23.458 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-16T16:47:23.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:23 smithi141 ceph-mon[131184]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:23.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:23 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/225568116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:23.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:23 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1316425762' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:47:23.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.669+0000 7fd3aabff700 1 -- 172.21.15.2:0/3983840076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a41002d0 msgr2=0x7fd3a41006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:23.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.669+0000 7fd3aabff700 1 --2- 172.21.15.2:0/3983840076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a41002d0 0x7fd3a41006e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd398009990 tx=0x7fd398009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.669+0000 7fd3aabff700 1 -- 172.21.15.2:0/3983840076 shutdown_connections 2023-11-16T16:47:23.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.670+0000 7fd3aabff700 1 --2- 172.21.15.2:0/3983840076 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3a4101e50 0x7fd3a4108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.670+0000 7fd3aabff700 1 --2- 172.21.15.2:0/3983840076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a41002d0 0x7fd3a41006e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.670+0000 7fd3aabff700 1 --2- 172.21.15.2:0/3983840076 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd3a41014c0 0x7fd3a4101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.670+0000 7fd3aabff700 1 -- 172.21.15.2:0/3983840076 >> 172.21.15.2:0/3983840076 conn(0x7fd3a40fb8b0 msgr2=0x7fd3a40fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:23.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.670+0000 7fd3aabff700 1 -- 172.21.15.2:0/3983840076 shutdown_connections 2023-11-16T16:47:23.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.670+0000 7fd3aabff700 1 -- 172.21.15.2:0/3983840076 wait complete. 2023-11-16T16:47:23.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.671+0000 7fd3aabff700 1 Processor -- start 2023-11-16T16:47:23.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.671+0000 7fd3aabff700 1 -- start start 2023-11-16T16:47:23.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.671+0000 7fd3aabff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a41002d0 0x7fd3a41982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:23.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.671+0000 7fd3aabff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3a41014c0 0x7fd3a4198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:23.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.671+0000 7fd3aabff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd3a4101e50 0x7fd3a419f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:23.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.671+0000 7fd3aabff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a419fe50 con 0x7fd3a41002d0 2023-11-16T16:47:23.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.671+0000 7fd3aabff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a419ff90 con 0x7fd3a4101e50 2023-11-16T16:47:23.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.671+0000 7fd3aabff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a41a0120 con 0x7fd3a41014c0 2023-11-16T16:47:23.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.671+0000 7fd3a899b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a41002d0 0x7fd3a41982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:23.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.671+0000 7fd3a899b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a41002d0 0x7fd3a41982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39376/0 (socket says 172.21.15.2:39376) 2023-11-16T16:47:23.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.671+0000 7fd3a899b700 1 -- 172.21.15.2:0/283901192 learned_addr learned my addr 172.21.15.2:0/283901192 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:23.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.671+0000 7fd3a3fff700 1 --2- 172.21.15.2:0/283901192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3a41014c0 0x7fd3a4198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:23.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.672+0000 7fd3a919c700 1 --2- 172.21.15.2:0/283901192 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd3a4101e50 0x7fd3a419f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:23.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.672+0000 7fd3a899b700 1 -- 172.21.15.2:0/283901192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3a41014c0 msgr2=0x7fd3a4198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:23.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.672+0000 7fd3a899b700 1 --2- 172.21.15.2:0/283901192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3a41014c0 0x7fd3a4198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.672+0000 7fd3a899b700 1 -- 172.21.15.2:0/283901192 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd3a4101e50 msgr2=0x7fd3a419f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:23.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.672+0000 7fd3a899b700 1 --2- 172.21.15.2:0/283901192 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd3a4101e50 0x7fd3a419f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:23.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.672+0000 7fd3a899b700 1 -- 172.21.15.2:0/283901192 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3980095f0 con 0x7fd3a41002d0 2023-11-16T16:47:23.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.672+0000 7fd3a3fff700 1 --2- 172.21.15.2:0/283901192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3a41014c0 0x7fd3a4198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:23.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.672+0000 7fd3a899b700 1 --2- 172.21.15.2:0/283901192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a41002d0 0x7fd3a41982d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd398004730 tx=0x7fd398004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:23.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.672+0000 7fd3a1ffb700 1 -- 172.21.15.2:0/283901192 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39801d070 con 0x7fd3a41002d0 2023-11-16T16:47:23.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.672+0000 7fd3a1ffb700 1 -- 172.21.15.2:0/283901192 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd398021410 con 0x7fd3a41002d0 2023-11-16T16:47:23.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.672+0000 7fd3aabff700 1 -- 172.21.15.2:0/283901192 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a410bca0 con 0x7fd3a41002d0 2023-11-16T16:47:23.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.672+0000 7fd3a1ffb700 1 -- 172.21.15.2:0/283901192 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398017650 con 0x7fd3a41002d0 2023-11-16T16:47:23.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.673+0000 7fd3aabff700 1 -- 172.21.15.2:0/283901192 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a410bb10 con 0x7fd3a41002d0 2023-11-16T16:47:23.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.674+0000 7fd3aabff700 1 -- 172.21.15.2:0/283901192 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a4192a00 con 0x7fd3a41002d0 2023-11-16T16:47:23.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.676+0000 7fd3a1ffb700 1 -- 172.21.15.2:0/283901192 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fd398021580 con 0x7fd3a41002d0 2023-11-16T16:47:23.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.678+0000 7fd3a1ffb700 1 --2- 172.21.15.2:0/283901192 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd3940a11d0 0x7fd3940a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:23.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.678+0000 7fd3a1ffb700 1 -- 172.21.15.2:0/283901192 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd3980cdf00 con 0x7fd3a41002d0 2023-11-16T16:47:23.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.678+0000 7fd3a3fff700 1 --2- 172.21.15.2:0/283901192 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd3940a11d0 0x7fd3940a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:23.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.679+0000 7fd3a3fff700 1 --2- 172.21.15.2:0/283901192 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd3940a11d0 0x7fd3940a3680 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd390011490 tx=0x7fd390009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:23.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.680+0000 7fd3a1ffb700 1 -- 172.21.15.2:0/283901192 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd398097ae0 con 0x7fd3a41002d0 2023-11-16T16:47:23.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:23.901+0000 7fd3aabff700 1 -- 172.21.15.2:0/283901192 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd3a402cfe0 con 0x7fd3a41002d0 2023-11-16T16:47:24.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:24 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:24.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:24 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:24.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:24 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:24.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:24 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:24.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:24 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2395937648' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-11-16T16:47:24.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:24 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:24.793 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:24.791+0000 7fd3a1ffb700 1 -- 172.21.15.2:0/283901192 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd39809b100 con 0x7fd3a41002d0 2023-11-16T16:47:24.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:24.797+0000 7fd3aabff700 1 -- 172.21.15.2:0/283901192 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd3940a11d0 msgr2=0x7fd3940a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:24.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:24.797+0000 7fd3aabff700 1 --2- 172.21.15.2:0/283901192 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd3940a11d0 0x7fd3940a3680 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd390011490 tx=0x7fd390009260 comp rx=0 tx=0).stop 2023-11-16T16:47:24.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:24.797+0000 7fd3aabff700 1 -- 172.21.15.2:0/283901192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a41002d0 msgr2=0x7fd3a41982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:24.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:24.797+0000 7fd3aabff700 1 --2- 172.21.15.2:0/283901192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a41002d0 0x7fd3a41982d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd398004730 tx=0x7fd398004760 comp rx=0 tx=0).stop 2023-11-16T16:47:24.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:24.797+0000 7fd3aabff700 1 -- 172.21.15.2:0/283901192 shutdown_connections 2023-11-16T16:47:24.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:24.797+0000 7fd3aabff700 1 --2- 172.21.15.2:0/283901192 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd3940a11d0 0x7fd3940a3680 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:24.797+0000 7fd3aabff700 1 --2- 172.21.15.2:0/283901192 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd3a4101e50 0x7fd3a419f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:24.797+0000 7fd3aabff700 1 --2- 172.21.15.2:0/283901192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a41002d0 0x7fd3a41982d0 secure :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd398004730 tx=0x7fd398004760 comp rx=0 tx=0).stop 2023-11-16T16:47:24.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:24.797+0000 7fd3aabff700 1 --2- 172.21.15.2:0/283901192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3a41014c0 0x7fd3a4198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:24.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:24.797+0000 7fd3aabff700 1 -- 172.21.15.2:0/283901192 >> 172.21.15.2:0/283901192 conn(0x7fd3a40fb8b0 msgr2=0x7fd3a4106e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:24.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:24.797+0000 7fd3aabff700 1 -- 172.21.15.2:0/283901192 shutdown_connections 2023-11-16T16:47:24.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:24.798+0000 7fd3aabff700 1 -- 172.21.15.2:0/283901192 wait complete. 2023-11-16T16:47:24.801 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_metadata' created 2023-11-16T16:47:24.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:24 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:24.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:24 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:24.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:24 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:24.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:24 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:24.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:24 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2395937648' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-11-16T16:47:24.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:24 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:24.819 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-11-16T16:47:24.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:24 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:24.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:24 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:24.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:24 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:24.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:24 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:24.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:24 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2395937648' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-11-16T16:47:24.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:24 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:25.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.041+0000 7f6964bff700 1 -- 172.21.15.2:0/4238398229 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6960103720 msgr2=0x7f6960105b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.041+0000 7f6964bff700 1 --2- 172.21.15.2:0/4238398229 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6960103720 0x7f6960105b00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6948009990 tx=0x7f6948009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.041+0000 7f6964bff700 1 -- 172.21.15.2:0/4238398229 shutdown_connections 2023-11-16T16:47:25.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.041+0000 7f6964bff700 1 --2- 172.21.15.2:0/4238398229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6960106040 0x7f6960108420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.041+0000 7f6964bff700 1 --2- 172.21.15.2:0/4238398229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6960100e00 0x7f69601031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.041+0000 7f6964bff700 1 --2- 172.21.15.2:0/4238398229 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6960103720 0x7f6960105b00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.041+0000 7f6964bff700 1 -- 172.21.15.2:0/4238398229 >> 172.21.15.2:0/4238398229 conn(0x7f69600fa7c0 msgr2=0x7f69600fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:25.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.041+0000 7f6964bff700 1 -- 172.21.15.2:0/4238398229 shutdown_connections 2023-11-16T16:47:25.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.041+0000 7f6964bff700 1 -- 172.21.15.2:0/4238398229 wait complete. 2023-11-16T16:47:25.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.042+0000 7f6964bff700 1 Processor -- start 2023-11-16T16:47:25.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.042+0000 7f6964bff700 1 -- start start 2023-11-16T16:47:25.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.043+0000 7f6964bff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6960100e00 0x7f6960198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.043+0000 7f6964bff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6960103720 0x7f69601987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.043+0000 7f6964bff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6960106040 0x7f696019f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.043+0000 7f6964bff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696019fe10 con 0x7f6960106040 2023-11-16T16:47:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.043+0000 7f6964bff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696019ff50 con 0x7f6960100e00 2023-11-16T16:47:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.043+0000 7f6964bff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69601a00e0 con 0x7f6960103720 2023-11-16T16:47:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.043+0000 7f695e59c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6960100e00 0x7f6960198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.043+0000 7f6957fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6960103720 0x7f69601987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.043+0000 7f695e59c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6960100e00 0x7f6960198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:42118/0 (socket says 172.21.15.2:42118) 2023-11-16T16:47:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.043+0000 7f695e59c700 1 -- 172.21.15.2:0/1972804875 learned_addr learned my addr 172.21.15.2:0/1972804875 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.043+0000 7f695ed9d700 1 --2- 172.21.15.2:0/1972804875 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6960106040 0x7f696019f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.043+0000 7f695e59c700 1 -- 172.21.15.2:0/1972804875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6960103720 msgr2=0x7f69601987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.043+0000 7f695e59c700 1 --2- 172.21.15.2:0/1972804875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6960103720 0x7f69601987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.043+0000 7f695e59c700 1 -- 172.21.15.2:0/1972804875 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6960106040 msgr2=0x7f696019f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.043+0000 7f695e59c700 1 --2- 172.21.15.2:0/1972804875 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6960106040 0x7f696019f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.043+0000 7f695e59c700 1 -- 172.21.15.2:0/1972804875 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69480095f0 con 0x7f6960100e00 2023-11-16T16:47:25.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.044+0000 7f695e59c700 1 --2- 172.21.15.2:0/1972804875 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6960100e00 0x7f6960198290 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f695000e9a0 tx=0x7f695000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:25.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.044+0000 7f69577fe700 1 -- 172.21.15.2:0/1972804875 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6950009790 con 0x7f6960100e00 2023-11-16T16:47:25.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.044+0000 7f69577fe700 1 -- 172.21.15.2:0/1972804875 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6950010430 con 0x7f6960100e00 2023-11-16T16:47:25.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.044+0000 7f69577fe700 1 -- 172.21.15.2:0/1972804875 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6950018680 con 0x7f6960100e00 2023-11-16T16:47:25.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.044+0000 7f6964bff700 1 -- 172.21.15.2:0/1972804875 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696010bcc0 con 0x7f6960100e00 2023-11-16T16:47:25.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.044+0000 7f6964bff700 1 -- 172.21.15.2:0/1972804875 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696010bb30 con 0x7f6960100e00 2023-11-16T16:47:25.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.045+0000 7f6964bff700 1 -- 172.21.15.2:0/1972804875 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6960107bb0 con 0x7f6960100e00 2023-11-16T16:47:25.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.047+0000 7f69577fe700 1 -- 172.21.15.2:0/1972804875 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f69500105a0 con 0x7f6960100e00 2023-11-16T16:47:25.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.048+0000 7f69577fe700 1 --2- 172.21.15.2:0/1972804875 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f694c0a11d0 0x7f694c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:25.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.049+0000 7f69577fe700 1 -- 172.21.15.2:0/1972804875 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f6950014070 con 0x7f6960100e00 2023-11-16T16:47:25.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.049+0000 7f6957fff700 1 --2- 172.21.15.2:0/1972804875 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f694c0a11d0 0x7f694c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:25.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.049+0000 7f6957fff700 1 --2- 172.21.15.2:0/1972804875 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f694c0a11d0 0x7f694c0a3680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6948000c00 tx=0x7f6948005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:25.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.052+0000 7f69577fe700 1 -- 172.21.15.2:0/1972804875 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6950051730 con 0x7f6960100e00 2023-11-16T16:47:25.263 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.261+0000 7f6964bff700 1 -- 172.21.15.2:0/1972804875 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f696002cfe0 con 0x7f6960100e00 2023-11-16T16:47:25.807 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.805+0000 7f69577fe700 1 -- 172.21.15.2:0/1972804875 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) v1 ==== 171+0+0 (secure 0 0 0) 0x7f6950099c60 con 0x7f6960100e00 2023-11-16T16:47:25.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.814+0000 7f6964bff700 1 -- 172.21.15.2:0/1972804875 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f694c0a11d0 msgr2=0x7f694c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.814+0000 7f6964bff700 1 --2- 172.21.15.2:0/1972804875 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f694c0a11d0 0x7f694c0a3680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6948000c00 tx=0x7f6948005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.814+0000 7f6964bff700 1 -- 172.21.15.2:0/1972804875 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6960100e00 msgr2=0x7f6960198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:25.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.814+0000 7f6964bff700 1 --2- 172.21.15.2:0/1972804875 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6960100e00 0x7f6960198290 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f695000e9a0 tx=0x7f695000ecb0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.814+0000 7f6964bff700 1 -- 172.21.15.2:0/1972804875 shutdown_connections 2023-11-16T16:47:25.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.814+0000 7f6964bff700 1 --2- 172.21.15.2:0/1972804875 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f694c0a11d0 0x7f694c0a3680 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6948000c00 tx=0x7f6948005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.814+0000 7f6964bff700 1 --2- 172.21.15.2:0/1972804875 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6960106040 0x7f696019f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.814+0000 7f6964bff700 1 --2- 172.21.15.2:0/1972804875 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6960100e00 0x7f6960198290 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.814+0000 7f6964bff700 1 --2- 172.21.15.2:0/1972804875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6960103720 0x7f69601987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:25.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.815+0000 7f6964bff700 1 -- 172.21.15.2:0/1972804875 >> 172.21.15.2:0/1972804875 conn(0x7f69600fa7c0 msgr2=0x7f69600ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:25.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.815+0000 7f6964bff700 1 -- 172.21.15.2:0/1972804875 shutdown_connections 2023-11-16T16:47:25.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:25.815+0000 7f6964bff700 1 -- 172.21.15.2:0/1972804875 wait complete. 2023-11-16T16:47:25.820 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data' created 2023-11-16T16:47:25.849 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-16T16:47:26.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.067+0000 7f061e39d700 1 -- 172.21.15.2:0/3803535025 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0618100050 msgr2=0x7f0618100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:26.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.067+0000 7f061e39d700 1 --2- 172.21.15.2:0/3803535025 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0618100050 0x7f0618100460 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0614009990 tx=0x7f0614009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:26.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.068+0000 7f061e39d700 1 -- 172.21.15.2:0/3803535025 shutdown_connections 2023-11-16T16:47:26.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.068+0000 7f061e39d700 1 --2- 172.21.15.2:0/3803535025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0618101c90 0x7f0618108530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:26.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.068+0000 7f061e39d700 1 --2- 172.21.15.2:0/3803535025 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0618100050 0x7f0618100460 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:26.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.068+0000 7f061e39d700 1 --2- 172.21.15.2:0/3803535025 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06181012e0 0x7f0618101750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:26.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.068+0000 7f061e39d700 1 -- 172.21.15.2:0/3803535025 >> 172.21.15.2:0/3803535025 conn(0x7f06180fb650 msgr2=0x7f06180fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:26.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.068+0000 7f061e39d700 1 -- 172.21.15.2:0/3803535025 shutdown_connections 2023-11-16T16:47:26.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.068+0000 7f061e39d700 1 -- 172.21.15.2:0/3803535025 wait complete. 2023-11-16T16:47:26.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.069+0000 7f061e39d700 1 Processor -- start 2023-11-16T16:47:26.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.069+0000 7f061e39d700 1 -- start start 2023-11-16T16:47:26.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.069+0000 7f061e39d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0618100050 0x7f06180748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:26.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.069+0000 7f061e39d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06181012e0 0x7f0618072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:26.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.069+0000 7f061e39d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0618101c90 0x7f0618073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:26.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.069+0000 7f061e39d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0618073bc0 con 0x7f06181012e0 2023-11-16T16:47:26.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.069+0000 7f061e39d700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0618073d30 con 0x7f0618100050 2023-11-16T16:47:26.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.069+0000 7f061e39d700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061819fe30 con 0x7f0618101c90 2023-11-16T16:47:26.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.069+0000 7f061d39b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0618100050 0x7f06180748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:26.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.069+0000 7f061cb9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06181012e0 0x7f0618072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:26.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.069+0000 7f061d39b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0618100050 0x7f06180748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:42130/0 (socket says 172.21.15.2:42130) 2023-11-16T16:47:26.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.069+0000 7f061d39b700 1 -- 172.21.15.2:0/2196440852 learned_addr learned my addr 172.21.15.2:0/2196440852 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:26.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.070+0000 7f061db9c700 1 --2- 172.21.15.2:0/2196440852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0618101c90 0x7f0618073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:26.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.070+0000 7f061d39b700 1 -- 172.21.15.2:0/2196440852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0618101c90 msgr2=0x7f0618073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:26.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.070+0000 7f061d39b700 1 --2- 172.21.15.2:0/2196440852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0618101c90 0x7f0618073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:26.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.070+0000 7f061d39b700 1 -- 172.21.15.2:0/2196440852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06181012e0 msgr2=0x7f0618072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:26.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.070+0000 7f061d39b700 1 --2- 172.21.15.2:0/2196440852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06181012e0 0x7f0618072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:26.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.070+0000 7f061d39b700 1 -- 172.21.15.2:0/2196440852 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06140095f0 con 0x7f0618100050 2023-11-16T16:47:26.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.070+0000 7f061d39b700 1 --2- 172.21.15.2:0/2196440852 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0618100050 0x7f06180748a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0614004730 tx=0x7f0614004760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:26.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.070+0000 7f060e7fc700 1 -- 172.21.15.2:0/2196440852 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061401d070 con 0x7f0618100050 2023-11-16T16:47:26.076 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.070+0000 7f060e7fc700 1 -- 172.21.15.2:0/2196440852 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0614004c90 con 0x7f0618100050 2023-11-16T16:47:26.076 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.070+0000 7f061e39d700 1 -- 172.21.15.2:0/2196440852 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061810ba80 con 0x7f0618100050 2023-11-16T16:47:26.076 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.070+0000 7f060e7fc700 1 -- 172.21.15.2:0/2196440852 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0614005570 con 0x7f0618100050 2023-11-16T16:47:26.076 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.070+0000 7f061e39d700 1 -- 172.21.15.2:0/2196440852 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f061810b8f0 con 0x7f0618100050 2023-11-16T16:47:26.077 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.071+0000 7f061e39d700 1 -- 172.21.15.2:0/2196440852 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0618067980 con 0x7f0618100050 2023-11-16T16:47:26.077 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.073+0000 7f060e7fc700 1 -- 172.21.15.2:0/2196440852 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f061400fc70 con 0x7f0618100050 2023-11-16T16:47:26.078 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.076+0000 7f060e7fc700 1 --2- 172.21.15.2:0/2196440852 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f06040a11d0 0x7f06040a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:26.078 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.076+0000 7f060e7fc700 1 -- 172.21.15.2:0/2196440852 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f06140cdae0 con 0x7f0618100050 2023-11-16T16:47:26.078 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.076+0000 7f061cb9a700 1 --2- 172.21.15.2:0/2196440852 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f06040a11d0 0x7f06040a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:26.079 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.077+0000 7f061cb9a700 1 --2- 172.21.15.2:0/2196440852 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f06040a11d0 0x7f06040a3680 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0608009c60 tx=0x7f0608009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:26.079 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.077+0000 7f060e7fc700 1 -- 172.21.15.2:0/2196440852 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f06140973d0 con 0x7f0618100050 2023-11-16T16:47:26.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:25 smithi155 ceph-mon[131607]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:26.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:25 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/283901192' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-11-16T16:47:26.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:25 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:26.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:25 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:26.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:25 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:26.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:25 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:47:26.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:25 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:47:26.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:25 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:26.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:25 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:47:26.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:25 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:47:26.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:25 smithi155 ceph-mon[131607]: Service mds.all has no related daemon already placed. 2023-11-16T16:47:26.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:25 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/283901192' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-11-16T16:47:26.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:25 smithi155 ceph-mon[131607]: osdmap e82: 12 total, 12 up, 12 in 2023-11-16T16:47:26.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:25 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:47:26.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:25 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1972804875' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-11-16T16:47:26.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:25 smithi002 ceph-mon[128712]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:26.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:25 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/283901192' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-11-16T16:47:26.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:25 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:26.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:25 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:26.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:25 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:26.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:25 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:47:26.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:25 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:47:26.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:25 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:26.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:25 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:47:26.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:25 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:47:26.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:25 smithi002 ceph-mon[128712]: Service mds.all has no related daemon already placed. 2023-11-16T16:47:26.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:25 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/283901192' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-11-16T16:47:26.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:25 smithi002 ceph-mon[128712]: osdmap e82: 12 total, 12 up, 12 in 2023-11-16T16:47:26.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:25 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:47:26.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:25 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1972804875' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-11-16T16:47:26.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:25 smithi141 ceph-mon[131184]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:26.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:25 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/283901192' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-11-16T16:47:26.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:25 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:26.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:25 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:26.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:25 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:26.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:25 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:47:26.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:25 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:47:26.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:25 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:26.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:25 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:47:26.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:25 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:47:26.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:25 smithi141 ceph-mon[131184]: Service mds.all has no related daemon already placed. 2023-11-16T16:47:26.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:25 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/283901192' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-11-16T16:47:26.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:25 smithi141 ceph-mon[131184]: osdmap e82: 12 total, 12 up, 12 in 2023-11-16T16:47:26.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:25 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:47:26.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:25 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1972804875' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-11-16T16:47:26.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.346+0000 7f061e39d700 1 -- 172.21.15.2:0/2196440852 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0618106f70 con 0x7f0618100050 2023-11-16T16:47:26.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.356+0000 7f060e7fc700 1 -- 172.21.15.2:0/2196440852 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f061409a9f0 con 0x7f0618100050 2023-11-16T16:47:26.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.361+0000 7f061e39d700 1 -- 172.21.15.2:0/2196440852 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f06040a11d0 msgr2=0x7f06040a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:26.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.361+0000 7f061e39d700 1 --2- 172.21.15.2:0/2196440852 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f06040a11d0 0x7f06040a3680 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0608009c60 tx=0x7f0608009260 comp rx=0 tx=0).stop 2023-11-16T16:47:26.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.361+0000 7f061e39d700 1 -- 172.21.15.2:0/2196440852 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0618100050 msgr2=0x7f06180748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:26.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.361+0000 7f061e39d700 1 --2- 172.21.15.2:0/2196440852 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0618100050 0x7f06180748a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0614004730 tx=0x7f0614004760 comp rx=0 tx=0).stop 2023-11-16T16:47:26.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.361+0000 7f061e39d700 1 -- 172.21.15.2:0/2196440852 shutdown_connections 2023-11-16T16:47:26.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.361+0000 7f061e39d700 1 --2- 172.21.15.2:0/2196440852 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f06040a11d0 0x7f06040a3680 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:26.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.361+0000 7f061e39d700 1 --2- 172.21.15.2:0/2196440852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0618101c90 0x7f0618073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:26.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.361+0000 7f061e39d700 1 --2- 172.21.15.2:0/2196440852 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0618100050 0x7f06180748a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:26.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.361+0000 7f061e39d700 1 --2- 172.21.15.2:0/2196440852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06181012e0 0x7f0618072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:26.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.361+0000 7f061e39d700 1 -- 172.21.15.2:0/2196440852 >> 172.21.15.2:0/2196440852 conn(0x7f06180fb650 msgr2=0x7f06180fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:26.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.362+0000 7f061e39d700 1 -- 172.21.15.2:0/2196440852 shutdown_connections 2023-11-16T16:47:26.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.363+0000 7f061e39d700 1 -- 172.21.15.2:0/2196440852 wait complete. 2023-11-16T16:47:26.370 INFO:teuthology.orchestra.run.smithi002.stderr:new fs with metadata pool 2 and data pool 3 2023-11-16T16:47:26.399 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-11-16T16:47:26.399 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-11-16T16:47:26.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.618+0000 7f083b59e700 1 -- 172.21.15.2:0/530948633 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f083c101c90 msgr2=0x7f083c108530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:26.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.618+0000 7f083b59e700 1 --2- 172.21.15.2:0/530948633 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f083c101c90 0x7f083c108530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f083400b0a0 tx=0x7f083400b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:26.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.618+0000 7f083b59e700 1 -- 172.21.15.2:0/530948633 shutdown_connections 2023-11-16T16:47:26.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.618+0000 7f083b59e700 1 --2- 172.21.15.2:0/530948633 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f083c100050 0x7f083c100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:26.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.618+0000 7f083b59e700 1 --2- 172.21.15.2:0/530948633 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f083c101c90 0x7f083c108530 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:26.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.618+0000 7f083b59e700 1 --2- 172.21.15.2:0/530948633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f083c1012e0 0x7f083c101750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:26.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.618+0000 7f083b59e700 1 -- 172.21.15.2:0/530948633 >> 172.21.15.2:0/530948633 conn(0x7f083c0fb650 msgr2=0x7f083c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:26.621 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.618+0000 7f083b59e700 1 -- 172.21.15.2:0/530948633 shutdown_connections 2023-11-16T16:47:26.621 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.618+0000 7f083b59e700 1 -- 172.21.15.2:0/530948633 wait complete. 2023-11-16T16:47:26.621 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.619+0000 7f083b59e700 1 Processor -- start 2023-11-16T16:47:26.622 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.619+0000 7f083b59e700 1 -- start start 2023-11-16T16:47:26.622 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.619+0000 7f083b59e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f083c100050 0x7f083c195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:26.622 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.619+0000 7f083b59e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f083c1012e0 0x7f083c196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:26.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.620+0000 7f083b59e700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f083c101c90 0x7f083c19d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:26.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.620+0000 7f083b59e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c19da70 con 0x7f083c1012e0 2023-11-16T16:47:26.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.620+0000 7f083b59e700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c19dbb0 con 0x7f083c101c90 2023-11-16T16:47:26.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.620+0000 7f083b59e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c19dd40 con 0x7f083c100050 2023-11-16T16:47:26.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.620+0000 7f0839d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f083c1012e0 0x7f083c196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:26.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.620+0000 7f0839d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f083c1012e0 0x7f083c196430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50068/0 (socket says 172.21.15.2:50068) 2023-11-16T16:47:26.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.620+0000 7f0839d9b700 1 -- 172.21.15.2:0/2901087705 learned_addr learned my addr 172.21.15.2:0/2901087705 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:26.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.620+0000 7f083ad9d700 1 --2- 172.21.15.2:0/2901087705 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f083c101c90 0x7f083c19d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:26.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.620+0000 7f083a59c700 1 --2- 172.21.15.2:0/2901087705 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f083c100050 0x7f083c195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:26.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.620+0000 7f0839d9b700 1 -- 172.21.15.2:0/2901087705 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f083c100050 msgr2=0x7f083c195ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:26.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.620+0000 7f0839d9b700 1 --2- 172.21.15.2:0/2901087705 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f083c100050 0x7f083c195ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:26.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.621+0000 7f0839d9b700 1 -- 172.21.15.2:0/2901087705 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f083c101c90 msgr2=0x7f083c19d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:26.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.621+0000 7f0839d9b700 1 --2- 172.21.15.2:0/2901087705 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f083c101c90 0x7f083c19d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:26.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.621+0000 7f0839d9b700 1 -- 172.21.15.2:0/2901087705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0834009d00 con 0x7f083c1012e0 2023-11-16T16:47:26.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.621+0000 7f0839d9b700 1 --2- 172.21.15.2:0/2901087705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f083c1012e0 0x7f083c196430 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f083000b570 tx=0x7f083000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:26.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.621+0000 7f082b7fe700 1 -- 172.21.15.2:0/2901087705 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0830019070 con 0x7f083c1012e0 2023-11-16T16:47:26.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.621+0000 7f083b59e700 1 -- 172.21.15.2:0/2901087705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083c10ba10 con 0x7f083c1012e0 2023-11-16T16:47:26.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.621+0000 7f082b7fe700 1 -- 172.21.15.2:0/2901087705 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f083001d8d0 con 0x7f083c1012e0 2023-11-16T16:47:26.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.621+0000 7f082b7fe700 1 -- 172.21.15.2:0/2901087705 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083001c910 con 0x7f083c1012e0 2023-11-16T16:47:26.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.621+0000 7f083b59e700 1 -- 172.21.15.2:0/2901087705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f083c10b880 con 0x7f083c1012e0 2023-11-16T16:47:26.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.622+0000 7f083b59e700 1 -- 172.21.15.2:0/2901087705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f083c073310 con 0x7f083c1012e0 2023-11-16T16:47:26.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.625+0000 7f082b7fe700 1 -- 172.21.15.2:0/2901087705 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f083001d620 con 0x7f083c1012e0 2023-11-16T16:47:26.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.626+0000 7f082b7fe700 1 --2- 172.21.15.2:0/2901087705 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08240a1150 0x7f08240a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:26.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.626+0000 7f082b7fe700 1 -- 172.21.15.2:0/2901087705 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f08300d4a50 con 0x7f083c1012e0 2023-11-16T16:47:26.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.626+0000 7f083a59c700 1 --2- 172.21.15.2:0/2901087705 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08240a1150 0x7f08240a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:26.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.628+0000 7f083a59c700 1 --2- 172.21.15.2:0/2901087705 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08240a1150 0x7f08240a3600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f082c009c60 tx=0x7f082c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:26.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.630+0000 7f082b7fe700 1 -- 172.21.15.2:0/2901087705 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f083009e2f0 con 0x7f083c1012e0 2023-11-16T16:47:26.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:26.857+0000 7f083b59e700 1 -- 172.21.15.2:0/2901087705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f083c0738c0 con 0x7f083c1012e0 2023-11-16T16:47:27.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:26 smithi155 ceph-mon[131607]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-11-16T16:47:27.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:26 smithi155 ceph-mon[131607]: pgmap v273: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:27.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:26 smithi155 ceph-mon[131607]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-11-16T16:47:27.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:26 smithi155 ceph-mon[131607]: osdmap e83: 12 total, 12 up, 12 in 2023-11-16T16:47:27.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:26 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2196440852' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-11-16T16:47:27.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:26 smithi155 ceph-mon[131607]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-11-16T16:47:27.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:26 smithi155 ceph-mon[131607]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-11-16T16:47:27.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:26 smithi155 ceph-mon[131607]: osdmap e84: 12 total, 12 up, 12 in 2023-11-16T16:47:27.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:26 smithi155 ceph-mon[131607]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-11-16T16:47:27.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:26 smithi155 ceph-mon[131607]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-11-16T16:47:27.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:26 smithi155 ceph-mon[131607]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-11-16T16:47:27.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:26 smithi002 ceph-mon[128712]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-11-16T16:47:27.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:26 smithi002 ceph-mon[128712]: pgmap v273: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:27.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:26 smithi002 ceph-mon[128712]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-11-16T16:47:27.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:26 smithi002 ceph-mon[128712]: osdmap e83: 12 total, 12 up, 12 in 2023-11-16T16:47:27.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:26 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2196440852' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-11-16T16:47:27.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:26 smithi002 ceph-mon[128712]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-11-16T16:47:27.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:26 smithi002 ceph-mon[128712]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-11-16T16:47:27.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:26 smithi002 ceph-mon[128712]: osdmap e84: 12 total, 12 up, 12 in 2023-11-16T16:47:27.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:26 smithi002 ceph-mon[128712]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-11-16T16:47:27.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:26 smithi002 ceph-mon[128712]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-11-16T16:47:27.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:26 smithi002 ceph-mon[128712]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-11-16T16:47:27.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:26 smithi141 ceph-mon[131184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-11-16T16:47:27.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:26 smithi141 ceph-mon[131184]: pgmap v273: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:27.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:26 smithi141 ceph-mon[131184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-11-16T16:47:27.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:26 smithi141 ceph-mon[131184]: osdmap e83: 12 total, 12 up, 12 in 2023-11-16T16:47:27.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:26 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2196440852' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-11-16T16:47:27.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:26 smithi141 ceph-mon[131184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-11-16T16:47:27.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:26 smithi141 ceph-mon[131184]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-11-16T16:47:27.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:26 smithi141 ceph-mon[131184]: osdmap e84: 12 total, 12 up, 12 in 2023-11-16T16:47:27.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:26 smithi141 ceph-mon[131184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-11-16T16:47:27.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:26 smithi141 ceph-mon[131184]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-11-16T16:47:27.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:26 smithi141 ceph-mon[131184]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-11-16T16:47:27.356 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.354+0000 7f082b7fe700 1 -- 172.21.15.2:0/2901087705 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) v1 ==== 157+0+0 (secure 0 0 0) 0x7f083002c0e0 con 0x7f083c1012e0 2023-11-16T16:47:27.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.357+0000 7f083b59e700 1 -- 172.21.15.2:0/2901087705 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08240a1150 msgr2=0x7f08240a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:27.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.357+0000 7f083b59e700 1 --2- 172.21.15.2:0/2901087705 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08240a1150 0x7f08240a3600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f082c009c60 tx=0x7f082c009260 comp rx=0 tx=0).stop 2023-11-16T16:47:27.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.357+0000 7f083b59e700 1 -- 172.21.15.2:0/2901087705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f083c1012e0 msgr2=0x7f083c196430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:27.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.357+0000 7f083b59e700 1 --2- 172.21.15.2:0/2901087705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f083c1012e0 0x7f083c196430 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f083000b570 tx=0x7f083000b880 comp rx=0 tx=0).stop 2023-11-16T16:47:27.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.357+0000 7f083b59e700 1 -- 172.21.15.2:0/2901087705 shutdown_connections 2023-11-16T16:47:27.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.357+0000 7f083b59e700 1 --2- 172.21.15.2:0/2901087705 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08240a1150 0x7f08240a3600 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:27.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.357+0000 7f083b59e700 1 --2- 172.21.15.2:0/2901087705 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f083c100050 0x7f083c195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:27.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.357+0000 7f083b59e700 1 --2- 172.21.15.2:0/2901087705 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f083c101c90 0x7f083c19d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:27.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.357+0000 7f083b59e700 1 --2- 172.21.15.2:0/2901087705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f083c1012e0 0x7f083c196430 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:27.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.357+0000 7f083b59e700 1 -- 172.21.15.2:0/2901087705 >> 172.21.15.2:0/2901087705 conn(0x7f083c0fb650 msgr2=0x7f083c0fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:27.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.357+0000 7f083b59e700 1 -- 172.21.15.2:0/2901087705 shutdown_connections 2023-11-16T16:47:27.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.357+0000 7f083b59e700 1 -- 172.21.15.2:0/2901087705 wait complete. 2023-11-16T16:47:27.380 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-11-16T16:47:27.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.602+0000 7fe2d8dff700 1 -- 172.21.15.2:0/3433395198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2d4101e50 msgr2=0x7fe2d4108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:27.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.602+0000 7fe2d8dff700 1 --2- 172.21.15.2:0/3433395198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2d4101e50 0x7fe2d4108710 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00b0a0 tx=0x7fe2cc00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:27.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.602+0000 7fe2d8dff700 1 -- 172.21.15.2:0/3433395198 shutdown_connections 2023-11-16T16:47:27.604 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.602+0000 7fe2d8dff700 1 --2- 172.21.15.2:0/3433395198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2d4101e50 0x7fe2d4108710 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:27.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.602+0000 7fe2d8dff700 1 --2- 172.21.15.2:0/3433395198 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2d41002d0 0x7fe2d41006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:27.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.602+0000 7fe2d8dff700 1 --2- 172.21.15.2:0/3433395198 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2d41014c0 0x7fe2d4101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:27.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.602+0000 7fe2d8dff700 1 -- 172.21.15.2:0/3433395198 >> 172.21.15.2:0/3433395198 conn(0x7fe2d40fb8b0 msgr2=0x7fe2d40fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:27.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.602+0000 7fe2d8dff700 1 -- 172.21.15.2:0/3433395198 shutdown_connections 2023-11-16T16:47:27.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.602+0000 7fe2d8dff700 1 -- 172.21.15.2:0/3433395198 wait complete. 2023-11-16T16:47:27.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.603+0000 7fe2d8dff700 1 Processor -- start 2023-11-16T16:47:27.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.603+0000 7fe2d8dff700 1 -- start start 2023-11-16T16:47:27.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.604+0000 7fe2d8dff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2d41002d0 0x7fe2d40748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:27.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.604+0000 7fe2d8dff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2d41014c0 0x7fe2d4072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:27.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.604+0000 7fe2d8dff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2d4101e50 0x7fe2d4073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:27.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.604+0000 7fe2d8dff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d4073bc0 con 0x7fe2d41002d0 2023-11-16T16:47:27.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.604+0000 7fe2d8dff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d4073d30 con 0x7fe2d41014c0 2023-11-16T16:47:27.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.604+0000 7fe2d8dff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d41a0210 con 0x7fe2d4101e50 2023-11-16T16:47:27.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.604+0000 7fe2d259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2d41002d0 0x7fe2d40748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:27.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.604+0000 7fe2d259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2d41002d0 0x7fe2d40748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50090/0 (socket says 172.21.15.2:50090) 2023-11-16T16:47:27.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.604+0000 7fe2d259c700 1 -- 172.21.15.2:0/547598403 learned_addr learned my addr 172.21.15.2:0/547598403 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:27.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.604+0000 7fe2d1d9b700 1 --2- 172.21.15.2:0/547598403 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2d41014c0 0x7fe2d4072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:27.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.604+0000 7fe2d2d9d700 1 --2- 172.21.15.2:0/547598403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2d4101e50 0x7fe2d4073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:27.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.604+0000 7fe2d259c700 1 -- 172.21.15.2:0/547598403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2d4101e50 msgr2=0x7fe2d4073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:27.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.604+0000 7fe2d259c700 1 --2- 172.21.15.2:0/547598403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2d4101e50 0x7fe2d4073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:27.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.604+0000 7fe2d259c700 1 -- 172.21.15.2:0/547598403 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2d41014c0 msgr2=0x7fe2d4072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:27.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.604+0000 7fe2d259c700 1 --2- 172.21.15.2:0/547598403 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2d41014c0 0x7fe2d4072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:27.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.604+0000 7fe2d259c700 1 -- 172.21.15.2:0/547598403 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc009d00 con 0x7fe2d41002d0 2023-11-16T16:47:27.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.604+0000 7fe2d2d9d700 1 --2- 172.21.15.2:0/547598403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2d4101e50 0x7fe2d4073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:27.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.605+0000 7fe2d259c700 1 --2- 172.21.15.2:0/547598403 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2d41002d0 0x7fe2d40748a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400e970 tx=0x7fe2c400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:27.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.605+0000 7fe2c37fe700 1 -- 172.21.15.2:0/547598403 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4009790 con 0x7fe2d41002d0 2023-11-16T16:47:27.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.605+0000 7fe2c37fe700 1 -- 172.21.15.2:0/547598403 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe2c400eec0 con 0x7fe2d41002d0 2023-11-16T16:47:27.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.605+0000 7fe2d8dff700 1 -- 172.21.15.2:0/547598403 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d410bca0 con 0x7fe2d41002d0 2023-11-16T16:47:27.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.605+0000 7fe2c37fe700 1 -- 172.21.15.2:0/547598403 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4003cf0 con 0x7fe2d41002d0 2023-11-16T16:47:27.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.605+0000 7fe2d8dff700 1 -- 172.21.15.2:0/547598403 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d410bb10 con 0x7fe2d41002d0 2023-11-16T16:47:27.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.606+0000 7fe2d8dff700 1 -- 172.21.15.2:0/547598403 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2d41077c0 con 0x7fe2d41002d0 2023-11-16T16:47:27.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.609+0000 7fe2c37fe700 1 -- 172.21.15.2:0/547598403 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fe2c400ccd0 con 0x7fe2d41002d0 2023-11-16T16:47:27.612 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.610+0000 7fe2c37fe700 1 --2- 172.21.15.2:0/547598403 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe2bc0a1150 0x7fe2bc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:27.612 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.610+0000 7fe2c37fe700 1 -- 172.21.15.2:0/547598403 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8205+0+0 (secure 0 0 0) 0x7fe2c4014070 con 0x7fe2d41002d0 2023-11-16T16:47:27.612 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.610+0000 7fe2d1d9b700 1 --2- 172.21.15.2:0/547598403 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe2bc0a1150 0x7fe2bc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:27.613 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.611+0000 7fe2d1d9b700 1 --2- 172.21.15.2:0/547598403 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe2bc0a1150 0x7fe2bc0a3600 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8009be0 tx=0x7fe2c8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:27.613 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.611+0000 7fe2c37fe700 1 -- 172.21.15.2:0/547598403 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe2c40963f0 con 0x7fe2d41002d0 2023-11-16T16:47:27.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:27.829+0000 7fe2d8dff700 1 -- 172.21.15.2:0/547598403 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe2d4061930 con 0x7fe2d41002d0 2023-11-16T16:47:28.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:27 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2901087705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-11-16T16:47:28.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:27 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2901087705' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-11-16T16:47:28.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:27 smithi155 ceph-mon[131607]: osdmap e85: 12 total, 12 up, 12 in 2023-11-16T16:47:28.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:27 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] up:active 2023-11-16T16:47:28.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:27 smithi155 ceph-mon[131607]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:28.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:27 smithi155 ceph-mon[131607]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:28.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:27 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:28.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:27 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:28.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:27 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:28.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:27 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:28.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:27 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2901087705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-11-16T16:47:28.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:27 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2901087705' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-11-16T16:47:28.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:27 smithi002 ceph-mon[128712]: osdmap e85: 12 total, 12 up, 12 in 2023-11-16T16:47:28.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:27 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] up:active 2023-11-16T16:47:28.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:27 smithi002 ceph-mon[128712]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:28.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:27 smithi002 ceph-mon[128712]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:28.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:27 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:28.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:27 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:28.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:27 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:28.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:27 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:28.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:27 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2901087705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-11-16T16:47:28.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:27 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2901087705' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-11-16T16:47:28.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:27 smithi141 ceph-mon[131184]: osdmap e85: 12 total, 12 up, 12 in 2023-11-16T16:47:28.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:27 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] up:active 2023-11-16T16:47:28.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:27 smithi141 ceph-mon[131184]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:28.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:27 smithi141 ceph-mon[131184]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:28.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:27 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:28.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:27 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:28.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:27 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:28.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:27 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:29.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:28 smithi155 ceph-mon[131607]: pgmap v277: 97 pgs: 18 creating+peering, 27 active+clean, 52 unknown; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:29.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:28 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/547598403' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-11-16T16:47:29.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:28 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:29.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:28 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:29.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:28 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:29.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:28 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:47:29.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:28 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:29.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:28 smithi002 ceph-mon[128712]: pgmap v277: 97 pgs: 18 creating+peering, 27 active+clean, 52 unknown; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:29.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:28 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/547598403' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-11-16T16:47:29.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:28 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:29.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:28 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:29.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:28 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:29.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:28 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:47:29.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:28 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:29.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:28 smithi141 ceph-mon[131184]: pgmap v277: 97 pgs: 18 creating+peering, 27 active+clean, 52 unknown; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:47:29.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:28 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/547598403' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-11-16T16:47:29.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:28 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:29.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:28 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:29.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:28 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:47:29.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:28 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:47:29.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:28 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:47:29.806 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:29.804+0000 7fe2c37fe700 1 -- 172.21.15.2:0/547598403 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) v1 ==== 228+0+0 (secure 0 0 0) 0x7fe2c4099a10 con 0x7fe2d41002d0 2023-11-16T16:47:29.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:29.810+0000 7fe2d8dff700 1 -- 172.21.15.2:0/547598403 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe2bc0a1150 msgr2=0x7fe2bc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:29.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:29.810+0000 7fe2d8dff700 1 --2- 172.21.15.2:0/547598403 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe2bc0a1150 0x7fe2bc0a3600 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8009be0 tx=0x7fe2c8009260 comp rx=0 tx=0).stop 2023-11-16T16:47:29.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:29.810+0000 7fe2d8dff700 1 -- 172.21.15.2:0/547598403 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2d41002d0 msgr2=0x7fe2d40748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:29.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:29.810+0000 7fe2d8dff700 1 --2- 172.21.15.2:0/547598403 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2d41002d0 0x7fe2d40748a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400e970 tx=0x7fe2c400ec80 comp rx=0 tx=0).stop 2023-11-16T16:47:29.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:29.810+0000 7fe2d8dff700 1 -- 172.21.15.2:0/547598403 shutdown_connections 2023-11-16T16:47:29.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:29.810+0000 7fe2d8dff700 1 --2- 172.21.15.2:0/547598403 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe2bc0a1150 0x7fe2bc0a3600 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:29.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:29.810+0000 7fe2d8dff700 1 --2- 172.21.15.2:0/547598403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2d4101e50 0x7fe2d4073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:29.813 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:29.810+0000 7fe2d8dff700 1 --2- 172.21.15.2:0/547598403 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2d41002d0 0x7fe2d40748a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:29.813 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:29.810+0000 7fe2d8dff700 1 --2- 172.21.15.2:0/547598403 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe2d41014c0 0x7fe2d4072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:29.813 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:29.810+0000 7fe2d8dff700 1 -- 172.21.15.2:0/547598403 >> 172.21.15.2:0/547598403 conn(0x7fe2d40fb8b0 msgr2=0x7fe2d4104900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:29.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:29.812+0000 7fe2d8dff700 1 -- 172.21.15.2:0/547598403 shutdown_connections 2023-11-16T16:47:29.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:29.812+0000 7fe2d8dff700 1 -- 172.21.15.2:0/547598403 wait complete. 2023-11-16T16:47:29.817 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data_ec' created 2023-11-16T16:47:29.837 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-11-16T16:47:30.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.044+0000 7f9556bff700 1 -- 172.21.15.2:0/2295622942 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95501002c0 msgr2=0x7f95501006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:30.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.044+0000 7f9556bff700 1 --2- 172.21.15.2:0/2295622942 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95501002c0 0x7f95501006d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9544009990 tx=0x7f9544009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:30.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.044+0000 7f9556bff700 1 -- 172.21.15.2:0/2295622942 shutdown_connections 2023-11-16T16:47:30.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.044+0000 7f9556bff700 1 --2- 172.21.15.2:0/2295622942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9550101e40 0x7f9550108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:30.046 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.044+0000 7f9556bff700 1 --2- 172.21.15.2:0/2295622942 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95501002c0 0x7f95501006d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:30.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.044+0000 7f9556bff700 1 --2- 172.21.15.2:0/2295622942 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f95501014b0 0x7f9550101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:30.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.044+0000 7f9556bff700 1 -- 172.21.15.2:0/2295622942 >> 172.21.15.2:0/2295622942 conn(0x7f95500fb8a0 msgr2=0x7f95500fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:30.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.044+0000 7f9556bff700 1 -- 172.21.15.2:0/2295622942 shutdown_connections 2023-11-16T16:47:30.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.044+0000 7f9556bff700 1 -- 172.21.15.2:0/2295622942 wait complete. 2023-11-16T16:47:30.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.045+0000 7f9556bff700 1 Processor -- start 2023-11-16T16:47:30.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.045+0000 7f9556bff700 1 -- start start 2023-11-16T16:47:30.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.045+0000 7f9556bff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f95501002c0 0x7f95500748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:30.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.045+0000 7f9556bff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95501014b0 0x7f9550072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:30.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.046+0000 7f9556bff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9550101e40 0x7f9550073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:30.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.046+0000 7f9556bff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9550073bc0 con 0x7f95501014b0 2023-11-16T16:47:30.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.046+0000 7f9556bff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9550073d30 con 0x7f9550101e40 2023-11-16T16:47:30.049 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.046+0000 7f9556bff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95501a00c0 con 0x7f95501002c0 2023-11-16T16:47:30.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.046+0000 7f954ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95501014b0 0x7f9550072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:30.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.046+0000 7f954ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95501014b0 0x7f9550072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50098/0 (socket says 172.21.15.2:50098) 2023-11-16T16:47:30.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.046+0000 7f954ffff700 1 -- 172.21.15.2:0/3138341428 learned_addr learned my addr 172.21.15.2:0/3138341428 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:30.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.046+0000 7f955499b700 1 --2- 172.21.15.2:0/3138341428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f95501002c0 0x7f95500748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:30.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.046+0000 7f955519c700 1 --2- 172.21.15.2:0/3138341428 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9550101e40 0x7f9550073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:30.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.046+0000 7f954ffff700 1 -- 172.21.15.2:0/3138341428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f95501002c0 msgr2=0x7f95500748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:30.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.046+0000 7f954ffff700 1 --2- 172.21.15.2:0/3138341428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f95501002c0 0x7f95500748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:30.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.046+0000 7f954ffff700 1 -- 172.21.15.2:0/3138341428 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9550101e40 msgr2=0x7f9550073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:30.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.046+0000 7f954ffff700 1 --2- 172.21.15.2:0/3138341428 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9550101e40 0x7f9550073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:30.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.046+0000 7f954ffff700 1 -- 172.21.15.2:0/3138341428 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95440095f0 con 0x7f95501014b0 2023-11-16T16:47:30.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.046+0000 7f954ffff700 1 --2- 172.21.15.2:0/3138341428 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95501014b0 0x7f9550072ee0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f953c00e9a0 tx=0x7f953c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:30.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.047+0000 7f954dffb700 1 -- 172.21.15.2:0/3138341428 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c009790 con 0x7f95501014b0 2023-11-16T16:47:30.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.047+0000 7f954dffb700 1 -- 172.21.15.2:0/3138341428 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f953c010430 con 0x7f95501014b0 2023-11-16T16:47:30.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.047+0000 7f9556bff700 1 -- 172.21.15.2:0/3138341428 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f955010bc90 con 0x7f95501014b0 2023-11-16T16:47:30.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.047+0000 7f954dffb700 1 -- 172.21.15.2:0/3138341428 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c018680 con 0x7f95501014b0 2023-11-16T16:47:30.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.047+0000 7f9556bff700 1 -- 172.21.15.2:0/3138341428 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f955010bb00 con 0x7f95501014b0 2023-11-16T16:47:30.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.048+0000 7f9556bff700 1 -- 172.21.15.2:0/3138341428 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95501a0800 con 0x7f95501014b0 2023-11-16T16:47:30.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.050+0000 7f954dffb700 1 -- 172.21.15.2:0/3138341428 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f953c0105a0 con 0x7f95501014b0 2023-11-16T16:47:30.053 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.051+0000 7f954dffb700 1 --2- 172.21.15.2:0/3138341428 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f95400a11d0 0x7f95400a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:30.053 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.052+0000 7f954dffb700 1 -- 172.21.15.2:0/3138341428 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f953c014070 con 0x7f95501014b0 2023-11-16T16:47:30.053 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.052+0000 7f955499b700 1 --2- 172.21.15.2:0/3138341428 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f95400a11d0 0x7f95400a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:30.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.052+0000 7f955499b700 1 --2- 172.21.15.2:0/3138341428 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f95400a11d0 0x7f95400a3680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f954400b2e0 tx=0x7f9544005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:30.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.056+0000 7f954dffb700 1 -- 172.21.15.2:0/3138341428 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f953c051730 con 0x7f95501014b0 2023-11-16T16:47:30.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:29 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:47:30.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:29 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:47:30.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:29 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:47:30.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:29 smithi155 ceph-mon[131607]: Service mds.all has no related daemon already placed. 2023-11-16T16:47:30.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:29 smithi155 ceph-mon[131607]: osdmap e86: 12 total, 12 up, 12 in 2023-11-16T16:47:30.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:29 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:47:30.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:29 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:47:30.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:29 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:47:30.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:29 smithi002 ceph-mon[128712]: Service mds.all has no related daemon already placed. 2023-11-16T16:47:30.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:29 smithi002 ceph-mon[128712]: osdmap e86: 12 total, 12 up, 12 in 2023-11-16T16:47:30.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:29 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:47:30.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:29 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:47:30.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:29 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:47:30.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:29 smithi141 ceph-mon[131184]: Service mds.all has no related daemon already placed. 2023-11-16T16:47:30.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:29 smithi141 ceph-mon[131184]: osdmap e86: 12 total, 12 up, 12 in 2023-11-16T16:47:30.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.267+0000 7f9556bff700 1 -- 172.21.15.2:0/3138341428 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f9550107170 con 0x7f95501014b0 2023-11-16T16:47:30.808 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.807+0000 7f954dffb700 1 -- 172.21.15.2:0/3138341428 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) v1 ==== 169+0+0 (secure 0 0 0) 0x7f953c099c60 con 0x7f95501014b0 2023-11-16T16:47:30.815 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.813+0000 7f9556bff700 1 -- 172.21.15.2:0/3138341428 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f95400a11d0 msgr2=0x7f95400a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:30.815 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.813+0000 7f9556bff700 1 --2- 172.21.15.2:0/3138341428 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f95400a11d0 0x7f95400a3680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f954400b2e0 tx=0x7f9544005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:30.815 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.813+0000 7f9556bff700 1 -- 172.21.15.2:0/3138341428 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95501014b0 msgr2=0x7f9550072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:30.815 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.813+0000 7f9556bff700 1 --2- 172.21.15.2:0/3138341428 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95501014b0 0x7f9550072ee0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f953c00e9a0 tx=0x7f953c00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:47:30.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.815+0000 7f9556bff700 1 -- 172.21.15.2:0/3138341428 shutdown_connections 2023-11-16T16:47:30.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.815+0000 7f9556bff700 1 --2- 172.21.15.2:0/3138341428 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f95400a11d0 0x7f95400a3680 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:30.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.815+0000 7f9556bff700 1 --2- 172.21.15.2:0/3138341428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f95501002c0 0x7f95500748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:30.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.815+0000 7f9556bff700 1 --2- 172.21.15.2:0/3138341428 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9550101e40 0x7f9550073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:30.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.815+0000 7f9556bff700 1 --2- 172.21.15.2:0/3138341428 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95501014b0 0x7f9550072ee0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:30.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.815+0000 7f9556bff700 1 -- 172.21.15.2:0/3138341428 >> 172.21.15.2:0/3138341428 conn(0x7f95500fb8a0 msgr2=0x7f95500fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:30.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.815+0000 7f9556bff700 1 -- 172.21.15.2:0/3138341428 shutdown_connections 2023-11-16T16:47:30.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:30.815+0000 7f9556bff700 1 -- 172.21.15.2:0/3138341428 wait complete. 2023-11-16T16:47:30.820 INFO:teuthology.orchestra.run.smithi002.stderr:set pool 4 allow_ec_overwrites to true 2023-11-16T16:47:30.851 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-11-16T16:47:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.057+0000 7f201b7ff700 1 -- 172.21.15.2:0/1466702936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20141014b0 msgr2=0x7f2014101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.057+0000 7f201b7ff700 1 --2- 172.21.15.2:0/1466702936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20141014b0 0x7f2014101900 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2004009990 tx=0x7f2004009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.057+0000 7f201b7ff700 1 -- 172.21.15.2:0/1466702936 shutdown_connections 2023-11-16T16:47:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.057+0000 7f201b7ff700 1 --2- 172.21.15.2:0/1466702936 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2014101e40 0x7f2014108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.057+0000 7f201b7ff700 1 --2- 172.21.15.2:0/1466702936 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20141002c0 0x7f20141006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.057+0000 7f201b7ff700 1 --2- 172.21.15.2:0/1466702936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20141014b0 0x7f2014101900 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.057+0000 7f201b7ff700 1 -- 172.21.15.2:0/1466702936 >> 172.21.15.2:0/1466702936 conn(0x7f20140fb8a0 msgr2=0x7f20140fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:31.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.057+0000 7f201b7ff700 1 -- 172.21.15.2:0/1466702936 shutdown_connections 2023-11-16T16:47:31.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.057+0000 7f201b7ff700 1 -- 172.21.15.2:0/1466702936 wait complete. 2023-11-16T16:47:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.058+0000 7f201b7ff700 1 Processor -- start 2023-11-16T16:47:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.058+0000 7f201b7ff700 1 -- start start 2023-11-16T16:47:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.059+0000 7f201b7ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20141002c0 0x7f2014198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.059+0000 7f201b7ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20141014b0 0x7f20141987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.059+0000 7f201b7ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2014101e40 0x7f201419f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.059+0000 7f201b7ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201419fde0 con 0x7f20141002c0 2023-11-16T16:47:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.059+0000 7f201b7ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201419ff20 con 0x7f20141014b0 2023-11-16T16:47:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.059+0000 7f201b7ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20141a00b0 con 0x7f2014101e40 2023-11-16T16:47:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.059+0000 7f201959b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20141002c0 0x7f2014198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.059+0000 7f201959b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20141002c0 0x7f2014198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50118/0 (socket says 172.21.15.2:50118) 2023-11-16T16:47:31.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.059+0000 7f201959b700 1 -- 172.21.15.2:0/4219006457 learned_addr learned my addr 172.21.15.2:0/4219006457 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:31.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.059+0000 7f2018d9a700 1 --2- 172.21.15.2:0/4219006457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20141014b0 0x7f20141987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:31.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.059+0000 7f2019d9c700 1 --2- 172.21.15.2:0/4219006457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2014101e40 0x7f201419f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:31.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.059+0000 7f201959b700 1 -- 172.21.15.2:0/4219006457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2014101e40 msgr2=0x7f201419f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:31.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.059+0000 7f201959b700 1 --2- 172.21.15.2:0/4219006457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2014101e40 0x7f201419f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.059+0000 7f201959b700 1 -- 172.21.15.2:0/4219006457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20141014b0 msgr2=0x7f20141987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:31.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.059+0000 7f201959b700 1 --2- 172.21.15.2:0/4219006457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20141014b0 0x7f20141987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.059+0000 7f201959b700 1 -- 172.21.15.2:0/4219006457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20040095f0 con 0x7f20141002c0 2023-11-16T16:47:31.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.060+0000 7f201959b700 1 --2- 172.21.15.2:0/4219006457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20141002c0 0x7f2014198260 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f201000e970 tx=0x7f201000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:31.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.060+0000 7f200a7fc700 1 -- 172.21.15.2:0/4219006457 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2010009790 con 0x7f20141002c0 2023-11-16T16:47:31.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.060+0000 7f200a7fc700 1 -- 172.21.15.2:0/4219006457 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f201000eec0 con 0x7f20141002c0 2023-11-16T16:47:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.060+0000 7f201b7ff700 1 -- 172.21.15.2:0/4219006457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f201410bc70 con 0x7f20141002c0 2023-11-16T16:47:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.060+0000 7f200a7fc700 1 -- 172.21.15.2:0/4219006457 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2010003cf0 con 0x7f20141002c0 2023-11-16T16:47:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.060+0000 7f201b7ff700 1 -- 172.21.15.2:0/4219006457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f201410bae0 con 0x7f20141002c0 2023-11-16T16:47:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.061+0000 7f201b7ff700 1 -- 172.21.15.2:0/4219006457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20141a0850 con 0x7f20141002c0 2023-11-16T16:47:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.063+0000 7f200a7fc700 1 -- 172.21.15.2:0/4219006457 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f201000ccd0 con 0x7f20141002c0 2023-11-16T16:47:31.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.064+0000 7f200a7fc700 1 --2- 172.21.15.2:0/4219006457 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f20000a11d0 0x7f20000a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:31.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.065+0000 7f200a7fc700 1 -- 172.21.15.2:0/4219006457 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f2010014070 con 0x7f20141002c0 2023-11-16T16:47:31.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.065+0000 7f2018d9a700 1 --2- 172.21.15.2:0/4219006457 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f20000a11d0 0x7f20000a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:31.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.065+0000 7f2018d9a700 1 --2- 172.21.15.2:0/4219006457 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f20000a11d0 0x7f20000a3680 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f200400b2e0 tx=0x7f2004005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:31.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.068+0000 7f200a7fc700 1 -- 172.21.15.2:0/4219006457 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2010096210 con 0x7f20141002c0 2023-11-16T16:47:31.082 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:30 smithi141 ceph-mon[131184]: pgmap v279: 97 pgs: 18 creating+peering, 79 active+clean; 579 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2023-11-16T16:47:31.082 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:30 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/547598403' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-11-16T16:47:31.082 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:30 smithi141 ceph-mon[131184]: osdmap e87: 12 total, 12 up, 12 in 2023-11-16T16:47:31.082 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:30 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3138341428' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-11-16T16:47:31.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:30 smithi155 ceph-mon[131607]: pgmap v279: 97 pgs: 18 creating+peering, 79 active+clean; 579 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2023-11-16T16:47:31.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:30 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/547598403' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-11-16T16:47:31.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:30 smithi155 ceph-mon[131607]: osdmap e87: 12 total, 12 up, 12 in 2023-11-16T16:47:31.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:30 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3138341428' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-11-16T16:47:31.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:30 smithi002 ceph-mon[128712]: pgmap v279: 97 pgs: 18 creating+peering, 79 active+clean; 579 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2023-11-16T16:47:31.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:30 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/547598403' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-11-16T16:47:31.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:30 smithi002 ceph-mon[128712]: osdmap e87: 12 total, 12 up, 12 in 2023-11-16T16:47:31.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:30 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3138341428' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-11-16T16:47:31.327 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.325+0000 7f201b7ff700 1 -- 172.21.15.2:0/4219006457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f20141a0e00 con 0x7f20141002c0 2023-11-16T16:47:31.333 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.331+0000 7f200a7fc700 1 -- 172.21.15.2:0/4219006457 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7f2010099830 con 0x7f20141002c0 2023-11-16T16:47:31.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.334+0000 7f201b7ff700 1 -- 172.21.15.2:0/4219006457 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f20000a11d0 msgr2=0x7f20000a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:31.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.334+0000 7f201b7ff700 1 --2- 172.21.15.2:0/4219006457 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f20000a11d0 0x7f20000a3680 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f200400b2e0 tx=0x7f2004005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.334+0000 7f201b7ff700 1 -- 172.21.15.2:0/4219006457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20141002c0 msgr2=0x7f2014198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:31.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.334+0000 7f201b7ff700 1 --2- 172.21.15.2:0/4219006457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20141002c0 0x7f2014198260 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f201000e970 tx=0x7f201000ec80 comp rx=0 tx=0).stop 2023-11-16T16:47:31.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.334+0000 7f201b7ff700 1 -- 172.21.15.2:0/4219006457 shutdown_connections 2023-11-16T16:47:31.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.334+0000 7f201b7ff700 1 --2- 172.21.15.2:0/4219006457 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f20000a11d0 0x7f20000a3680 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.334+0000 7f201b7ff700 1 --2- 172.21.15.2:0/4219006457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2014101e40 0x7f201419f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.334+0000 7f201b7ff700 1 --2- 172.21.15.2:0/4219006457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20141002c0 0x7f2014198260 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.334+0000 7f201b7ff700 1 --2- 172.21.15.2:0/4219006457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f20141014b0 0x7f20141987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.334+0000 7f201b7ff700 1 -- 172.21.15.2:0/4219006457 >> 172.21.15.2:0/4219006457 conn(0x7f20140fb8a0 msgr2=0x7f20140fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:31.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.334+0000 7f201b7ff700 1 -- 172.21.15.2:0/4219006457 shutdown_connections 2023-11-16T16:47:31.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.334+0000 7f201b7ff700 1 -- 172.21.15.2:0/4219006457 wait complete. 2023-11-16T16:47:31.337 INFO:teuthology.orchestra.run.smithi002.stderr:added data pool 4 to fsmap 2023-11-16T16:47:31.355 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:31.514 INFO:journalctl@ceph.mds.b.smithi141.stdout:Nov 16 16:47:31 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-b[156414]: 2023-11-16T16:47:31.079+0000 7fdde4a6c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-16T16:47:31.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.558+0000 7f8e715ff700 1 -- 172.21.15.2:0/612165805 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e6c101e10 msgr2=0x7f8e6c1086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:31.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.558+0000 7f8e715ff700 1 --2- 172.21.15.2:0/612165805 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e6c101e10 0x7f8e6c1086d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8e6400b0a0 tx=0x7f8e6400b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.559+0000 7f8e715ff700 1 -- 172.21.15.2:0/612165805 shutdown_connections 2023-11-16T16:47:31.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.559+0000 7f8e715ff700 1 --2- 172.21.15.2:0/612165805 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e6c100290 0x7f8e6c1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.559+0000 7f8e715ff700 1 --2- 172.21.15.2:0/612165805 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e6c101e10 0x7f8e6c1086d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.559+0000 7f8e715ff700 1 --2- 172.21.15.2:0/612165805 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e6c101480 0x7f8e6c1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.559+0000 7f8e715ff700 1 -- 172.21.15.2:0/612165805 >> 172.21.15.2:0/612165805 conn(0x7f8e6c0fb870 msgr2=0x7f8e6c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:31.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.559+0000 7f8e715ff700 1 -- 172.21.15.2:0/612165805 shutdown_connections 2023-11-16T16:47:31.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.559+0000 7f8e715ff700 1 -- 172.21.15.2:0/612165805 wait complete. 2023-11-16T16:47:31.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.560+0000 7f8e715ff700 1 Processor -- start 2023-11-16T16:47:31.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.560+0000 7f8e715ff700 1 -- start start 2023-11-16T16:47:31.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.561+0000 7f8e715ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e6c100290 0x7f8e6c0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:31.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.561+0000 7f8e715ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e6c101480 0x7f8e6c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:31.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.561+0000 7f8e715ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e6c101e10 0x7f8e6c073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:31.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.561+0000 7f8e715ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c073bc0 con 0x7f8e6c100290 2023-11-16T16:47:31.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.561+0000 7f8e715ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c073d30 con 0x7f8e6c101e10 2023-11-16T16:47:31.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.561+0000 7f8e715ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c1a00f0 con 0x7f8e6c101480 2023-11-16T16:47:31.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.561+0000 7f8e6affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e6c100290 0x7f8e6c0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:31.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.561+0000 7f8e6affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e6c100290 0x7f8e6c0748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50138/0 (socket says 172.21.15.2:50138) 2023-11-16T16:47:31.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.561+0000 7f8e6affd700 1 -- 172.21.15.2:0/1745037147 learned_addr learned my addr 172.21.15.2:0/1745037147 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:31.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.561+0000 7f8e6a7fc700 1 --2- 172.21.15.2:0/1745037147 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e6c101480 0x7f8e6c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:31.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.561+0000 7f8e6b7fe700 1 --2- 172.21.15.2:0/1745037147 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e6c101e10 0x7f8e6c073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:31.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.561+0000 7f8e6affd700 1 -- 172.21.15.2:0/1745037147 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e6c101480 msgr2=0x7f8e6c072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:31.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.561+0000 7f8e6affd700 1 --2- 172.21.15.2:0/1745037147 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e6c101480 0x7f8e6c072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.561+0000 7f8e6affd700 1 -- 172.21.15.2:0/1745037147 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e6c101e10 msgr2=0x7f8e6c073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:31.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.561+0000 7f8e6affd700 1 --2- 172.21.15.2:0/1745037147 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e6c101e10 0x7f8e6c073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.562+0000 7f8e6affd700 1 -- 172.21.15.2:0/1745037147 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e64009d00 con 0x7f8e6c100290 2023-11-16T16:47:31.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.562+0000 7f8e6affd700 1 --2- 172.21.15.2:0/1745037147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e6c100290 0x7f8e6c0748a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c00e970 tx=0x7f8e5c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:31.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.562+0000 7f8e53fff700 1 -- 172.21.15.2:0/1745037147 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c009790 con 0x7f8e6c100290 2023-11-16T16:47:31.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.562+0000 7f8e53fff700 1 -- 172.21.15.2:0/1745037147 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e5c00c7e0 con 0x7f8e6c100290 2023-11-16T16:47:31.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.562+0000 7f8e715ff700 1 -- 172.21.15.2:0/1745037147 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6c10bc70 con 0x7f8e6c100290 2023-11-16T16:47:31.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.562+0000 7f8e53fff700 1 -- 172.21.15.2:0/1745037147 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c003c70 con 0x7f8e6c100290 2023-11-16T16:47:31.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.562+0000 7f8e715ff700 1 -- 172.21.15.2:0/1745037147 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e6c10bae0 con 0x7f8e6c100290 2023-11-16T16:47:31.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.563+0000 7f8e715ff700 1 -- 172.21.15.2:0/1745037147 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e6c1a0860 con 0x7f8e6c100290 2023-11-16T16:47:31.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.565+0000 7f8e53fff700 1 -- 172.21.15.2:0/1745037147 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f8e5c00cc50 con 0x7f8e6c100290 2023-11-16T16:47:31.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.566+0000 7f8e53fff700 1 --2- 172.21.15.2:0/1745037147 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8e540a11d0 0x7f8e540a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:31.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.567+0000 7f8e53fff700 1 -- 172.21.15.2:0/1745037147 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8e5c014070 con 0x7f8e6c100290 2023-11-16T16:47:31.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.567+0000 7f8e6a7fc700 1 --2- 172.21.15.2:0/1745037147 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8e540a11d0 0x7f8e540a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:31.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.567+0000 7f8e6a7fc700 1 --2- 172.21.15.2:0/1745037147 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8e540a11d0 0x7f8e540a3680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009c60 tx=0x7f8e60009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:31.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.570+0000 7f8e53fff700 1 -- 172.21.15.2:0/1745037147 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8e5c096170 con 0x7f8e6c100290 2023-11-16T16:47:31.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.829+0000 7f8e715ff700 1 -- 172.21.15.2:0/1745037147 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e6c107100 con 0x7f8e6c100290 2023-11-16T16:47:31.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.830+0000 7f8e53fff700 1 -- 172.21.15.2:0/1745037147 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10747 (secure 0 0 0) 0x7f8e5c003dd0 con 0x7f8e6c100290 2023-11-16T16:47:31.832 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:31.832 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/1610211225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":1610211225},{"type":"v1","addr":"172.21.15.2:6837","nonce":1610211225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14538,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2583261818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2583261818},{"type":"v1","addr":"172.21.15.2:6839","nonce":2583261818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14550,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3170991923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3170991923},{"type":"v1","addr":"172.21.15.2:6841","nonce":3170991923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3516869188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3516869188},{"type":"v1","addr":"172.21.15.141:6835","nonce":3516869188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2527313191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":2527313191},{"type":"v1","addr":"172.21.15.155:6835","nonce":2527313191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1607756751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1607756751},{"type":"v1","addr":"172.21.15.155:6837","nonce":1607756751}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6837/2934891475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6836","nonce":2934891475},{"type":"v1","addr":"172.21.15.141:6837","nonce":2934891475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/1158457009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1158457009},{"type":"v1","addr":"172.21.15.155:6839","nonce":1158457009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-16T16:47:26.349176+0000","modified":"2023-11-16T16:47:31.327539+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6839/3916358272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6838","nonce":3916358272},{"type":"v1","addr":"172.21.15.141:6839","nonce":3916358272}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-11-16T16:47:31.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.832+0000 7f8e715ff700 1 -- 172.21.15.2:0/1745037147 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8e540a11d0 msgr2=0x7f8e540a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:31.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.832+0000 7f8e715ff700 1 --2- 172.21.15.2:0/1745037147 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8e540a11d0 0x7f8e540a3680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009c60 tx=0x7f8e60009260 comp rx=0 tx=0).stop 2023-11-16T16:47:31.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.832+0000 7f8e715ff700 1 -- 172.21.15.2:0/1745037147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e6c100290 msgr2=0x7f8e6c0748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:31.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.832+0000 7f8e715ff700 1 --2- 172.21.15.2:0/1745037147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e6c100290 0x7f8e6c0748a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c00e970 tx=0x7f8e5c00ec80 comp rx=0 tx=0).stop 2023-11-16T16:47:31.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.832+0000 7f8e715ff700 1 -- 172.21.15.2:0/1745037147 shutdown_connections 2023-11-16T16:47:31.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.832+0000 7f8e715ff700 1 --2- 172.21.15.2:0/1745037147 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8e540a11d0 0x7f8e540a3680 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.832+0000 7f8e715ff700 1 --2- 172.21.15.2:0/1745037147 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e6c101e10 0x7f8e6c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.832+0000 7f8e715ff700 1 --2- 172.21.15.2:0/1745037147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e6c100290 0x7f8e6c0748a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.832+0000 7f8e715ff700 1 --2- 172.21.15.2:0/1745037147 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e6c101480 0x7f8e6c072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:31.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.832+0000 7f8e715ff700 1 -- 172.21.15.2:0/1745037147 >> 172.21.15.2:0/1745037147 conn(0x7f8e6c0fb870 msgr2=0x7f8e6c0fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:31.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.833+0000 7f8e715ff700 1 -- 172.21.15.2:0/1745037147 shutdown_connections 2023-11-16T16:47:31.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:31.833+0000 7f8e715ff700 1 -- 172.21.15.2:0/1745037147 wait complete. 2023-11-16T16:47:31.836 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 17 2023-11-16T16:47:31.854 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:47:32.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.056+0000 7f3659bff700 1 -- 172.21.15.2:0/3039463832 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3654100290 msgr2=0x7f36541006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:32.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.056+0000 7f3659bff700 1 --2- 172.21.15.2:0/3039463832 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3654100290 0x7f36541006a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3644009990 tx=0x7f3644009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.056+0000 7f3659bff700 1 -- 172.21.15.2:0/3039463832 shutdown_connections 2023-11-16T16:47:32.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.056+0000 7f3659bff700 1 --2- 172.21.15.2:0/3039463832 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3654101e10 0x7f36541086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.058 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.056+0000 7f3659bff700 1 --2- 172.21.15.2:0/3039463832 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3654100290 0x7f36541006a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.056+0000 7f3659bff700 1 --2- 172.21.15.2:0/3039463832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3654101480 0x7f36541018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.056+0000 7f3659bff700 1 -- 172.21.15.2:0/3039463832 >> 172.21.15.2:0/3039463832 conn(0x7f36540fb870 msgr2=0x7f36540fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:32.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.056+0000 7f3659bff700 1 -- 172.21.15.2:0/3039463832 shutdown_connections 2023-11-16T16:47:32.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.056+0000 7f3659bff700 1 -- 172.21.15.2:0/3039463832 wait complete. 2023-11-16T16:47:32.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.057+0000 7f3659bff700 1 Processor -- start 2023-11-16T16:47:32.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.057+0000 7f3659bff700 1 -- start start 2023-11-16T16:47:32.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.058+0000 7f3659bff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3654100290 0x7f3654195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:32.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.058+0000 7f3659bff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3654101480 0x7f3654196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:32.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.058+0000 7f3659bff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3654101e10 0x7f365419d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:32.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.058+0000 7f3659bff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365419db50 con 0x7f3654101480 2023-11-16T16:47:32.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.058+0000 7f3659bff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365419dc90 con 0x7f3654100290 2023-11-16T16:47:32.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.058+0000 7f3659bff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365419de20 con 0x7f3654101e10 2023-11-16T16:47:32.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.058+0000 7f364bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3654101480 0x7f3654196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:32.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.058+0000 7f364bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3654101480 0x7f3654196510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50154/0 (socket says 172.21.15.2:50154) 2023-11-16T16:47:32.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.058+0000 7f364bfff700 1 -- 172.21.15.2:0/3317601128 learned_addr learned my addr 172.21.15.2:0/3317601128 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:32.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.058+0000 7f36537fe700 1 --2- 172.21.15.2:0/3317601128 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3654100290 0x7f3654195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:32.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.058+0000 7f3653fff700 1 --2- 172.21.15.2:0/3317601128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3654101e10 0x7f365419d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:32.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.058+0000 7f364bfff700 1 -- 172.21.15.2:0/3317601128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3654101e10 msgr2=0x7f365419d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:32.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.058+0000 7f364bfff700 1 --2- 172.21.15.2:0/3317601128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3654101e10 0x7f365419d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.058+0000 7f364bfff700 1 -- 172.21.15.2:0/3317601128 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3654100290 msgr2=0x7f3654195fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:32.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.058+0000 7f364bfff700 1 --2- 172.21.15.2:0/3317601128 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3654100290 0x7f3654195fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.058+0000 7f364bfff700 1 -- 172.21.15.2:0/3317601128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36440095f0 con 0x7f3654101480 2023-11-16T16:47:32.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.059+0000 7f364bfff700 1 --2- 172.21.15.2:0/3317601128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3654101480 0x7f3654196510 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f363c00e9a0 tx=0x7f363c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:32.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.059+0000 7f36517fa700 1 -- 172.21.15.2:0/3317601128 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c009790 con 0x7f3654101480 2023-11-16T16:47:32.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.059+0000 7f3659bff700 1 -- 172.21.15.2:0/3317601128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f365410bc40 con 0x7f3654101480 2023-11-16T16:47:32.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.059+0000 7f36517fa700 1 -- 172.21.15.2:0/3317601128 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f363c010430 con 0x7f3654101480 2023-11-16T16:47:32.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.059+0000 7f36517fa700 1 -- 172.21.15.2:0/3317601128 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c018680 con 0x7f3654101480 2023-11-16T16:47:32.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.059+0000 7f3659bff700 1 -- 172.21.15.2:0/3317601128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f365410bab0 con 0x7f3654101480 2023-11-16T16:47:32.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.060+0000 7f364a7fc700 1 -- 172.21.15.2:0/3317601128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3654061700 con 0x7f3654101480 2023-11-16T16:47:32.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.062+0000 7f36517fa700 1 -- 172.21.15.2:0/3317601128 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f363c0105a0 con 0x7f3654101480 2023-11-16T16:47:32.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.063+0000 7f36517fa700 1 --2- 172.21.15.2:0/3317601128 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f36400a11d0 0x7f36400a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:32.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.064+0000 7f36517fa700 1 -- 172.21.15.2:0/3317601128 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f363c014070 con 0x7f3654101480 2023-11-16T16:47:32.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.064+0000 7f36537fe700 1 --2- 172.21.15.2:0/3317601128 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f36400a11d0 0x7f36400a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:32.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.064+0000 7f36537fe700 1 --2- 172.21.15.2:0/3317601128 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f36400a11d0 0x7f36400a3680 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f364400b2e0 tx=0x7f3644005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:32.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.067+0000 7f36517fa700 1 -- 172.21.15.2:0/3317601128 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f363c0960a0 con 0x7f3654101480 2023-11-16T16:47:32.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:31 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3138341428' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-11-16T16:47:32.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:31 smithi155 ceph-mon[131607]: osdmap e88: 12 total, 12 up, 12 in 2023-11-16T16:47:32.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:31 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/4219006457' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-11-16T16:47:32.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:31 smithi155 ceph-mon[131607]: osdmap e89: 12 total, 12 up, 12 in 2023-11-16T16:47:32.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:31 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/4219006457' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-11-16T16:47:32.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:31 smithi155 ceph-mon[131607]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:32.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:31 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3138341428' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-11-16T16:47:32.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:31 smithi002 ceph-mon[128712]: osdmap e88: 12 total, 12 up, 12 in 2023-11-16T16:47:32.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:31 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/4219006457' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-11-16T16:47:32.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:31 smithi002 ceph-mon[128712]: osdmap e89: 12 total, 12 up, 12 in 2023-11-16T16:47:32.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:31 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/4219006457' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-11-16T16:47:32.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:31 smithi002 ceph-mon[128712]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:32.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:31 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3138341428' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-11-16T16:47:32.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:31 smithi141 ceph-mon[131184]: osdmap e88: 12 total, 12 up, 12 in 2023-11-16T16:47:32.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:31 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/4219006457' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-11-16T16:47:32.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:31 smithi141 ceph-mon[131184]: osdmap e89: 12 total, 12 up, 12 in 2023-11-16T16:47:32.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:31 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/4219006457' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-11-16T16:47:32.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:31 smithi141 ceph-mon[131184]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:32.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.281+0000 7f364a7fc700 1 -- 172.21.15.2:0/3317601128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3654061cb0 con 0x7f3654101480 2023-11-16T16:47:32.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.283+0000 7f36517fa700 1 -- 172.21.15.2:0/3317601128 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23912 (secure 0 0 0) 0x7f363c0996c0 con 0x7f3654101480 2023-11-16T16:47:32.285 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:32.285 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":89,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:47:31.327295+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:24.458452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:25.786005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:29.313181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:47:32.287 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.285+0000 7f364a7fc700 1 -- 172.21.15.2:0/3317601128 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f36400a11d0 msgr2=0x7f36400a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:32.287 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.285+0000 7f364a7fc700 1 --2- 172.21.15.2:0/3317601128 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f36400a11d0 0x7f36400a3680 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f364400b2e0 tx=0x7f3644005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.287 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.285+0000 7f364a7fc700 1 -- 172.21.15.2:0/3317601128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3654101480 msgr2=0x7f3654196510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:32.287 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.285+0000 7f364a7fc700 1 --2- 172.21.15.2:0/3317601128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3654101480 0x7f3654196510 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f363c00e9a0 tx=0x7f363c00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.287 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.285+0000 7f364a7fc700 1 -- 172.21.15.2:0/3317601128 shutdown_connections 2023-11-16T16:47:32.287 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.285+0000 7f364a7fc700 1 --2- 172.21.15.2:0/3317601128 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f36400a11d0 0x7f36400a3680 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.288 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.285+0000 7f364a7fc700 1 --2- 172.21.15.2:0/3317601128 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3654101e10 0x7f365419d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.288 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.285+0000 7f364a7fc700 1 --2- 172.21.15.2:0/3317601128 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3654100290 0x7f3654195fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.288 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.285+0000 7f364a7fc700 1 --2- 172.21.15.2:0/3317601128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3654101480 0x7f3654196510 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.288 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.285+0000 7f364a7fc700 1 -- 172.21.15.2:0/3317601128 >> 172.21.15.2:0/3317601128 conn(0x7f36540fb870 msgr2=0x7f36541049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:32.288 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.286+0000 7f364a7fc700 1 -- 172.21.15.2:0/3317601128 shutdown_connections 2023-11-16T16:47:32.288 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.286+0000 7f364a7fc700 1 -- 172.21.15.2:0/3317601128 wait complete. 2023-11-16T16:47:32.307 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:47:32.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.509+0000 7fe64efff700 1 -- 172.21.15.2:0/2902259950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe648100290 msgr2=0x7fe6481006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:32.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.509+0000 7fe64efff700 1 --2- 172.21.15.2:0/2902259950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe648100290 0x7fe6481006a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009990 tx=0x7fe63c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.509+0000 7fe64efff700 1 -- 172.21.15.2:0/2902259950 shutdown_connections 2023-11-16T16:47:32.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.509+0000 7fe64efff700 1 --2- 172.21.15.2:0/2902259950 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe648101e10 0x7fe6481086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.509+0000 7fe64efff700 1 --2- 172.21.15.2:0/2902259950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe648100290 0x7fe6481006a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.509+0000 7fe64efff700 1 --2- 172.21.15.2:0/2902259950 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe648101480 0x7fe6481018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.509+0000 7fe64efff700 1 -- 172.21.15.2:0/2902259950 >> 172.21.15.2:0/2902259950 conn(0x7fe6480fb870 msgr2=0x7fe6480fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:32.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.509+0000 7fe64efff700 1 -- 172.21.15.2:0/2902259950 shutdown_connections 2023-11-16T16:47:32.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.509+0000 7fe64efff700 1 -- 172.21.15.2:0/2902259950 wait complete. 2023-11-16T16:47:32.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.510+0000 7fe64efff700 1 Processor -- start 2023-11-16T16:47:32.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.511+0000 7fe64efff700 1 -- start start 2023-11-16T16:47:32.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.511+0000 7fe64efff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe648100290 0x7fe648195fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:32.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.511+0000 7fe64efff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe648101480 0x7fe6481964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:32.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.511+0000 7fe64efff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe648101e10 0x7fe64819d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:32.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.511+0000 7fe64efff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64819db20 con 0x7fe648101480 2023-11-16T16:47:32.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.511+0000 7fe64efff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64819dc60 con 0x7fe648100290 2023-11-16T16:47:32.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.511+0000 7fe64efff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64819ddf0 con 0x7fe648101e10 2023-11-16T16:47:32.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.511+0000 7fe647fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe648101480 0x7fe6481964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:32.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.511+0000 7fe647fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe648101480 0x7fe6481964e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50176/0 (socket says 172.21.15.2:50176) 2023-11-16T16:47:32.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.511+0000 7fe647fff700 1 -- 172.21.15.2:0/1606201584 learned_addr learned my addr 172.21.15.2:0/1606201584 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:32.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.511+0000 7fe64d59c700 1 --2- 172.21.15.2:0/1606201584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe648101e10 0x7fe64819d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:32.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.511+0000 7fe64cd9b700 1 --2- 172.21.15.2:0/1606201584 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe648100290 0x7fe648195fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:32.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.512+0000 7fe647fff700 1 -- 172.21.15.2:0/1606201584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe648101e10 msgr2=0x7fe64819d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:32.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.512+0000 7fe647fff700 1 --2- 172.21.15.2:0/1606201584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe648101e10 0x7fe64819d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.512+0000 7fe647fff700 1 -- 172.21.15.2:0/1606201584 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe648100290 msgr2=0x7fe648195fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:32.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.512+0000 7fe647fff700 1 --2- 172.21.15.2:0/1606201584 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe648100290 0x7fe648195fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.512+0000 7fe647fff700 1 -- 172.21.15.2:0/1606201584 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63c0095f0 con 0x7fe648101480 2023-11-16T16:47:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.512+0000 7fe647fff700 1 --2- 172.21.15.2:0/1606201584 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe648101480 0x7fe6481964e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe63400e9a0 tx=0x7fe63400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.512+0000 7fe645ffb700 1 -- 172.21.15.2:0/1606201584 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe634009790 con 0x7fe648101480 2023-11-16T16:47:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.512+0000 7fe645ffb700 1 -- 172.21.15.2:0/1606201584 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe634010430 con 0x7fe648101480 2023-11-16T16:47:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.512+0000 7fe64efff700 1 -- 172.21.15.2:0/1606201584 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe64810bc40 con 0x7fe648101480 2023-11-16T16:47:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.512+0000 7fe645ffb700 1 -- 172.21.15.2:0/1606201584 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe634018680 con 0x7fe648101480 2023-11-16T16:47:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.512+0000 7fe64efff700 1 -- 172.21.15.2:0/1606201584 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe64810bab0 con 0x7fe648101480 2023-11-16T16:47:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.513+0000 7fe64efff700 1 -- 172.21.15.2:0/1606201584 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe648107210 con 0x7fe648101480 2023-11-16T16:47:32.517 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.515+0000 7fe645ffb700 1 -- 172.21.15.2:0/1606201584 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fe6340105a0 con 0x7fe648101480 2023-11-16T16:47:32.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.517+0000 7fe645ffb700 1 --2- 172.21.15.2:0/1606201584 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe6380a11d0 0x7fe6380a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:32.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.517+0000 7fe645ffb700 1 -- 172.21.15.2:0/1606201584 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe634014070 con 0x7fe648101480 2023-11-16T16:47:32.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.517+0000 7fe64cd9b700 1 --2- 172.21.15.2:0/1606201584 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe6380a11d0 0x7fe6380a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:32.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.517+0000 7fe64cd9b700 1 --2- 172.21.15.2:0/1606201584 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe6380a11d0 0x7fe6380a3680 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe63c00b2e0 tx=0x7fe63c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:32.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.519+0000 7fe645ffb700 1 -- 172.21.15.2:0/1606201584 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe6340963d0 con 0x7fe648101480 2023-11-16T16:47:32.730 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.728+0000 7fe64efff700 1 -- 172.21.15.2:0/1606201584 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe64802cfe0 con 0x7fe648101480 2023-11-16T16:47:32.731 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.730+0000 7fe645ffb700 1 -- 172.21.15.2:0/1606201584 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23912 (secure 0 0 0) 0x7fe634026020 con 0x7fe648101480 2023-11-16T16:47:32.732 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:32.732 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":89,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:47:31.327295+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:24.458452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:25.786005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:29.313181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:47:32.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.732+0000 7fe64efff700 1 -- 172.21.15.2:0/1606201584 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe6380a11d0 msgr2=0x7fe6380a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:32.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.732+0000 7fe64efff700 1 --2- 172.21.15.2:0/1606201584 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe6380a11d0 0x7fe6380a3680 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe63c00b2e0 tx=0x7fe63c005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.732+0000 7fe64efff700 1 -- 172.21.15.2:0/1606201584 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe648101480 msgr2=0x7fe6481964e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:32.734 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.732+0000 7fe64efff700 1 --2- 172.21.15.2:0/1606201584 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe648101480 0x7fe6481964e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe63400e9a0 tx=0x7fe63400ecb0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.732+0000 7fe64efff700 1 -- 172.21.15.2:0/1606201584 shutdown_connections 2023-11-16T16:47:32.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.732+0000 7fe64efff700 1 --2- 172.21.15.2:0/1606201584 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe6380a11d0 0x7fe6380a3680 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.732+0000 7fe64efff700 1 --2- 172.21.15.2:0/1606201584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe648101e10 0x7fe64819d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.732+0000 7fe64efff700 1 --2- 172.21.15.2:0/1606201584 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe648100290 0x7fe648195fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.732+0000 7fe64efff700 1 --2- 172.21.15.2:0/1606201584 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe648101480 0x7fe6481964e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.732+0000 7fe64efff700 1 -- 172.21.15.2:0/1606201584 >> 172.21.15.2:0/1606201584 conn(0x7fe6480fb870 msgr2=0x7fe6480fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:32.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.732+0000 7fe64efff700 1 -- 172.21.15.2:0/1606201584 shutdown_connections 2023-11-16T16:47:32.735 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.732+0000 7fe64efff700 1 -- 172.21.15.2:0/1606201584 wait complete. 2023-11-16T16:47:32.768 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:32.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.971+0000 7fcaee3ff700 1 -- 172.21.15.2:0/854874970 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcae8074b30 msgr2=0x7fcae8072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:32.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.971+0000 7fcaee3ff700 1 --2- 172.21.15.2:0/854874970 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcae8074b30 0x7fcae8072f80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcad8009990 tx=0x7fcad8009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.971+0000 7fcaee3ff700 1 -- 172.21.15.2:0/854874970 shutdown_connections 2023-11-16T16:47:32.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.971+0000 7fcaee3ff700 1 --2- 172.21.15.2:0/854874970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcae810c860 0x7fcae810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.971+0000 7fcaee3ff700 1 --2- 172.21.15.2:0/854874970 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcae8074b30 0x7fcae8072f80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.971+0000 7fcaee3ff700 1 --2- 172.21.15.2:0/854874970 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcae8073550 0x7fcae80739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.971+0000 7fcaee3ff700 1 -- 172.21.15.2:0/854874970 >> 172.21.15.2:0/854874970 conn(0x7fcae80fb870 msgr2=0x7fcae80fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:32.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.971+0000 7fcaee3ff700 1 -- 172.21.15.2:0/854874970 shutdown_connections 2023-11-16T16:47:32.974 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.971+0000 7fcaee3ff700 1 -- 172.21.15.2:0/854874970 wait complete. 2023-11-16T16:47:32.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.973+0000 7fcaee3ff700 1 Processor -- start 2023-11-16T16:47:32.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.973+0000 7fcaee3ff700 1 -- start start 2023-11-16T16:47:32.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.973+0000 7fcaee3ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcae8073550 0x7fcae819c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:32.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.973+0000 7fcaee3ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcae8074b30 0x7fcae819cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:32.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.974+0000 7fcaee3ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcae810c860 0x7fcae81a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:32.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.974+0000 7fcaee3ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae81a4170 con 0x7fcae810c860 2023-11-16T16:47:32.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.974+0000 7fcaee3ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae81a42b0 con 0x7fcae8074b30 2023-11-16T16:47:32.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.974+0000 7fcaee3ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae81a4440 con 0x7fcae8073550 2023-11-16T16:47:32.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.974+0000 7fcae77fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcae8074b30 0x7fcae819cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:32.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.974+0000 7fcaec99c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcae810c860 0x7fcae81a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:32.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.974+0000 7fcae77fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcae8074b30 0x7fcae819cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:42280/0 (socket says 172.21.15.2:42280) 2023-11-16T16:47:32.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.974+0000 7fcae77fe700 1 -- 172.21.15.2:0/1882276720 learned_addr learned my addr 172.21.15.2:0/1882276720 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:32.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.974+0000 7fcae7fff700 1 --2- 172.21.15.2:0/1882276720 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcae8073550 0x7fcae819c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:32.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.974+0000 7fcae77fe700 1 -- 172.21.15.2:0/1882276720 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcae8073550 msgr2=0x7fcae819c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:32.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.974+0000 7fcae77fe700 1 --2- 172.21.15.2:0/1882276720 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcae8073550 0x7fcae819c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.974+0000 7fcae77fe700 1 -- 172.21.15.2:0/1882276720 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcae810c860 msgr2=0x7fcae81a3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:32.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.974+0000 7fcae77fe700 1 --2- 172.21.15.2:0/1882276720 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcae810c860 0x7fcae81a3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:32.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.974+0000 7fcae77fe700 1 -- 172.21.15.2:0/1882276720 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae0009c40 con 0x7fcae8074b30 2023-11-16T16:47:32.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.974+0000 7fcaec99c700 1 --2- 172.21.15.2:0/1882276720 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcae810c860 0x7fcae81a3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:32.980 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.974+0000 7fcae77fe700 1 --2- 172.21.15.2:0/1882276720 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcae8074b30 0x7fcae819cb30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcadc00e970 tx=0x7fcadc00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:32.980 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.975+0000 7fcae57fa700 1 -- 172.21.15.2:0/1882276720 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcadc009790 con 0x7fcae8074b30 2023-11-16T16:47:32.980 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.975+0000 7fcae57fa700 1 -- 172.21.15.2:0/1882276720 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcadc00c7c0 con 0x7fcae8074b30 2023-11-16T16:47:32.980 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.975+0000 7fcae57fa700 1 -- 172.21.15.2:0/1882276720 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcadc003c20 con 0x7fcae8074b30 2023-11-16T16:47:32.980 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.975+0000 7fcaee3ff700 1 -- 172.21.15.2:0/1882276720 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad80095f0 con 0x7fcae8074b30 2023-11-16T16:47:32.980 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.975+0000 7fcaee3ff700 1 -- 172.21.15.2:0/1882276720 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae8112250 con 0x7fcae8074b30 2023-11-16T16:47:32.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.976+0000 7fcaee3ff700 1 -- 172.21.15.2:0/1882276720 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae81023b0 con 0x7fcae8074b30 2023-11-16T16:47:32.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.978+0000 7fcae57fa700 1 -- 172.21.15.2:0/1882276720 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fcadc00c930 con 0x7fcae8074b30 2023-11-16T16:47:32.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.979+0000 7fcae57fa700 1 --2- 172.21.15.2:0/1882276720 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fcad00a1150 0x7fcad00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:32.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.979+0000 7fcae57fa700 1 -- 172.21.15.2:0/1882276720 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fcadc014070 con 0x7fcae8074b30 2023-11-16T16:47:32.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.979+0000 7fcae7fff700 1 --2- 172.21.15.2:0/1882276720 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fcad00a1150 0x7fcad00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:32.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.980+0000 7fcae7fff700 1 --2- 172.21.15.2:0/1882276720 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fcad00a1150 0x7fcad00a3600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcad800b2e0 tx=0x7fcad8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:32.985 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:32.983+0000 7fcae57fa700 1 -- 172.21.15.2:0/1882276720 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcadc0961f0 con 0x7fcae8074b30 2023-11-16T16:47:33.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:32 smithi155 ceph-mon[131607]: pgmap v283: 129 pgs: 24 unknown, 105 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2023-11-16T16:47:33.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:32 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1745037147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:33.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:32 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3317601128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:33.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:32 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1606201584' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:33.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:32 smithi002 ceph-mon[128712]: pgmap v283: 129 pgs: 24 unknown, 105 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2023-11-16T16:47:33.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:32 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1745037147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:33.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:32 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3317601128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:33.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:32 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1606201584' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:33.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.242+0000 7fcaee3ff700 1 -- 172.21.15.2:0/1882276720 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcae802cfe0 con 0x7fcae8074b30 2023-11-16T16:47:33.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.243+0000 7fcae57fa700 1 -- 172.21.15.2:0/1882276720 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10747 (secure 0 0 0) 0x7fcadc099810 con 0x7fcae8074b30 2023-11-16T16:47:33.245 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:33.245 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/1610211225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":1610211225},{"type":"v1","addr":"172.21.15.2:6837","nonce":1610211225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14538,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2583261818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2583261818},{"type":"v1","addr":"172.21.15.2:6839","nonce":2583261818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14550,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3170991923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3170991923},{"type":"v1","addr":"172.21.15.2:6841","nonce":3170991923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6835/3516869188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3516869188},{"type":"v1","addr":"172.21.15.141:6835","nonce":3516869188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/2527313191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":2527313191},{"type":"v1","addr":"172.21.15.155:6835","nonce":2527313191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1607756751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1607756751},{"type":"v1","addr":"172.21.15.155:6837","nonce":1607756751}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6837/2934891475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6836","nonce":2934891475},{"type":"v1","addr":"172.21.15.141:6837","nonce":2934891475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/1158457009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1158457009},{"type":"v1","addr":"172.21.15.155:6839","nonce":1158457009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-16T16:47:26.349176+0000","modified":"2023-11-16T16:47:31.327539+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6839/3916358272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6838","nonce":3916358272},{"type":"v1","addr":"172.21.15.141:6839","nonce":3916358272}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-11-16T16:47:33.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.245+0000 7fcaee3ff700 1 -- 172.21.15.2:0/1882276720 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fcad00a1150 msgr2=0x7fcad00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:33.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.245+0000 7fcaee3ff700 1 --2- 172.21.15.2:0/1882276720 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fcad00a1150 0x7fcad00a3600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcad800b2e0 tx=0x7fcad8005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.245+0000 7fcaee3ff700 1 -- 172.21.15.2:0/1882276720 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcae8074b30 msgr2=0x7fcae819cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:33.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.245+0000 7fcaee3ff700 1 --2- 172.21.15.2:0/1882276720 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcae8074b30 0x7fcae819cb30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcadc00e970 tx=0x7fcadc00ec80 comp rx=0 tx=0).stop 2023-11-16T16:47:33.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.245+0000 7fcaee3ff700 1 -- 172.21.15.2:0/1882276720 shutdown_connections 2023-11-16T16:47:33.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.245+0000 7fcaee3ff700 1 --2- 172.21.15.2:0/1882276720 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fcad00a1150 0x7fcad00a3600 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.245+0000 7fcaee3ff700 1 --2- 172.21.15.2:0/1882276720 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcae8073550 0x7fcae819c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.245+0000 7fcaee3ff700 1 --2- 172.21.15.2:0/1882276720 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcae810c860 0x7fcae81a3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.245+0000 7fcaee3ff700 1 --2- 172.21.15.2:0/1882276720 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcae8074b30 0x7fcae819cb30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.245+0000 7fcaee3ff700 1 -- 172.21.15.2:0/1882276720 >> 172.21.15.2:0/1882276720 conn(0x7fcae80fb870 msgr2=0x7fcae80fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:33.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.246+0000 7fcaee3ff700 1 -- 172.21.15.2:0/1882276720 shutdown_connections 2023-11-16T16:47:33.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.246+0000 7fcaee3ff700 1 -- 172.21.15.2:0/1882276720 wait complete. 2023-11-16T16:47:33.249 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 17 2023-11-16T16:47:33.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:32 smithi141 ceph-mon[131184]: pgmap v283: 129 pgs: 24 unknown, 105 active+clean; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2023-11-16T16:47:33.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:32 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1745037147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:33.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:32 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3317601128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:33.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:32 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1606201584' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:33.267 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:47:33.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.469+0000 7f3d2f3ff700 1 -- 172.21.15.2:0/1373083213 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d28103760 msgr2=0x7f3d28103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:33.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.469+0000 7f3d2f3ff700 1 --2- 172.21.15.2:0/1373083213 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d28103760 0x7f3d28103bd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3d18009990 tx=0x7f3d18009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.469+0000 7f3d2f3ff700 1 -- 172.21.15.2:0/1373083213 shutdown_connections 2023-11-16T16:47:33.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.469+0000 7f3d2f3ff700 1 --2- 172.21.15.2:0/1373083213 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d28104110 0x7f3d2810a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.469+0000 7f3d2f3ff700 1 --2- 172.21.15.2:0/1373083213 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3d281024d0 0x7f3d281028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.469+0000 7f3d2f3ff700 1 --2- 172.21.15.2:0/1373083213 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d28103760 0x7f3d28103bd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.469+0000 7f3d2f3ff700 1 -- 172.21.15.2:0/1373083213 >> 172.21.15.2:0/1373083213 conn(0x7f3d280fdb30 msgr2=0x7f3d280fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:33.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.469+0000 7f3d2f3ff700 1 -- 172.21.15.2:0/1373083213 shutdown_connections 2023-11-16T16:47:33.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.469+0000 7f3d2f3ff700 1 -- 172.21.15.2:0/1373083213 wait complete. 2023-11-16T16:47:33.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.470+0000 7f3d2f3ff700 1 Processor -- start 2023-11-16T16:47:33.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.471+0000 7f3d2f3ff700 1 -- start start 2023-11-16T16:47:33.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.471+0000 7f3d2f3ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d281024d0 0x7f3d28198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.471+0000 7f3d2f3ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3d28103760 0x7f3d281986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.471+0000 7f3d2f3ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d28104110 0x7f3d2819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.471+0000 7f3d2f3ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2819fcf0 con 0x7f3d281024d0 2023-11-16T16:47:33.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.471+0000 7f3d2f3ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2819fe30 con 0x7f3d28103760 2023-11-16T16:47:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.471+0000 7f3d2f3ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2819ffc0 con 0x7f3d28104110 2023-11-16T16:47:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.471+0000 7f3d2d19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d281024d0 0x7f3d28198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.471+0000 7f3d2d19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d281024d0 0x7f3d28198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50202/0 (socket says 172.21.15.2:50202) 2023-11-16T16:47:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.471+0000 7f3d2d19b700 1 -- 172.21.15.2:0/1678107786 learned_addr learned my addr 172.21.15.2:0/1678107786 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.471+0000 7f3d2c99a700 1 --2- 172.21.15.2:0/1678107786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3d28103760 0x7f3d281986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.471+0000 7f3d2d99c700 1 --2- 172.21.15.2:0/1678107786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d28104110 0x7f3d2819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.472+0000 7f3d2d19b700 1 -- 172.21.15.2:0/1678107786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d28104110 msgr2=0x7f3d2819f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:33.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.472+0000 7f3d2d19b700 1 --2- 172.21.15.2:0/1678107786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d28104110 0x7f3d2819f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.472+0000 7f3d2d19b700 1 -- 172.21.15.2:0/1678107786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3d28103760 msgr2=0x7f3d281986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:33.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.472+0000 7f3d2d19b700 1 --2- 172.21.15.2:0/1678107786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3d28103760 0x7f3d281986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.472+0000 7f3d2d19b700 1 -- 172.21.15.2:0/1678107786 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d180095f0 con 0x7f3d281024d0 2023-11-16T16:47:33.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.472+0000 7f3d2d19b700 1 --2- 172.21.15.2:0/1678107786 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d281024d0 0x7f3d28198170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400d980 tx=0x7f3d2400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:33.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.472+0000 7f3d1e7fc700 1 -- 172.21.15.2:0/1678107786 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24004220 con 0x7f3d281024d0 2023-11-16T16:47:33.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.472+0000 7f3d1e7fc700 1 -- 172.21.15.2:0/1678107786 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d24007640 con 0x7f3d281024d0 2023-11-16T16:47:33.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.472+0000 7f3d2f3ff700 1 -- 172.21.15.2:0/1678107786 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d2810dec0 con 0x7f3d281024d0 2023-11-16T16:47:33.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.472+0000 7f3d1e7fc700 1 -- 172.21.15.2:0/1678107786 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24005690 con 0x7f3d281024d0 2023-11-16T16:47:33.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.472+0000 7f3d2f3ff700 1 -- 172.21.15.2:0/1678107786 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d2810dd30 con 0x7f3d281024d0 2023-11-16T16:47:33.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.473+0000 7f3d2f3ff700 1 -- 172.21.15.2:0/1678107786 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d28109450 con 0x7f3d281024d0 2023-11-16T16:47:33.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.475+0000 7f3d1e7fc700 1 -- 172.21.15.2:0/1678107786 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f3d24004750 con 0x7f3d281024d0 2023-11-16T16:47:33.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.476+0000 7f3d1e7fc700 1 --2- 172.21.15.2:0/1678107786 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3d140a1150 0x7f3d140a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.477+0000 7f3d1e7fc700 1 -- 172.21.15.2:0/1678107786 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3d240cc980 con 0x7f3d281024d0 2023-11-16T16:47:33.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.477+0000 7f3d2c99a700 1 --2- 172.21.15.2:0/1678107786 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3d140a1150 0x7f3d140a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.477+0000 7f3d2c99a700 1 --2- 172.21.15.2:0/1678107786 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3d140a1150 0x7f3d140a3600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800b5c0 tx=0x7f3d18005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:33.482 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.480+0000 7f3d1e7fc700 1 -- 172.21.15.2:0/1678107786 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3d24095e80 con 0x7f3d281024d0 2023-11-16T16:47:33.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.688+0000 7f3d2f3ff700 1 -- 172.21.15.2:0/1678107786 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d28067a60 con 0x7f3d281024d0 2023-11-16T16:47:33.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.689+0000 7f3d1e7fc700 1 -- 172.21.15.2:0/1678107786 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23912 (secure 0 0 0) 0x7f3d24016020 con 0x7f3d281024d0 2023-11-16T16:47:33.692 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:33.692 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":89,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:47:31.327295+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:24.458452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:25.786005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:29.313181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:47:33.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.692+0000 7f3d2f3ff700 1 -- 172.21.15.2:0/1678107786 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3d140a1150 msgr2=0x7f3d140a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:33.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.692+0000 7f3d2f3ff700 1 --2- 172.21.15.2:0/1678107786 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3d140a1150 0x7f3d140a3600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800b5c0 tx=0x7f3d18005fd0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.692+0000 7f3d2f3ff700 1 -- 172.21.15.2:0/1678107786 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d281024d0 msgr2=0x7f3d28198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:33.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.692+0000 7f3d2f3ff700 1 --2- 172.21.15.2:0/1678107786 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d281024d0 0x7f3d28198170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400d980 tx=0x7f3d2400dc90 comp rx=0 tx=0).stop 2023-11-16T16:47:33.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.692+0000 7f3d2f3ff700 1 -- 172.21.15.2:0/1678107786 shutdown_connections 2023-11-16T16:47:33.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.692+0000 7f3d2f3ff700 1 --2- 172.21.15.2:0/1678107786 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3d140a1150 0x7f3d140a3600 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.692+0000 7f3d2f3ff700 1 --2- 172.21.15.2:0/1678107786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d28104110 0x7f3d2819f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.692+0000 7f3d2f3ff700 1 --2- 172.21.15.2:0/1678107786 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d281024d0 0x7f3d28198170 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.692+0000 7f3d2f3ff700 1 --2- 172.21.15.2:0/1678107786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3d28103760 0x7f3d281986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.692+0000 7f3d2f3ff700 1 -- 172.21.15.2:0/1678107786 >> 172.21.15.2:0/1678107786 conn(0x7f3d280fdb30 msgr2=0x7f3d280ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:33.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.692+0000 7f3d2f3ff700 1 -- 172.21.15.2:0/1678107786 shutdown_connections 2023-11-16T16:47:33.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.692+0000 7f3d2f3ff700 1 -- 172.21.15.2:0/1678107786 wait complete. 2023-11-16T16:47:33.714 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-16T16:47:26.349176+0000', 'modified': '2023-11-16T16:47:31.327539+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24590}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24590': {'gid': 24590, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.141:6839/3916358272', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6838', 'nonce': 3916358272}, {'type': 'v1', 'addr': '172.21.15.141:6839', 'nonce': 3916358272}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-11-16T16:47:33.714 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-16T16:47:33.714 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-11-16T16:47:33.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.921+0000 7f54821ff700 1 -- 172.21.15.2:0/1697969899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f547c101db0 msgr2=0x7f547c108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:33.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.921+0000 7f54821ff700 1 --2- 172.21.15.2:0/1697969899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f547c101db0 0x7f547c108670 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5474009f90 tx=0x7f547400b330 comp rx=0 tx=0).stop 2023-11-16T16:47:33.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.921+0000 7f54821ff700 1 -- 172.21.15.2:0/1697969899 shutdown_connections 2023-11-16T16:47:33.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.921+0000 7f54821ff700 1 --2- 172.21.15.2:0/1697969899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f547c101db0 0x7f547c108670 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.921+0000 7f54821ff700 1 --2- 172.21.15.2:0/1697969899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f547c100230 0x7f547c100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.921+0000 7f54821ff700 1 --2- 172.21.15.2:0/1697969899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f547c101420 0x7f547c101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.921+0000 7f54821ff700 1 -- 172.21.15.2:0/1697969899 >> 172.21.15.2:0/1697969899 conn(0x7f547c0fb810 msgr2=0x7f547c0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:33.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.921+0000 7f54821ff700 1 -- 172.21.15.2:0/1697969899 shutdown_connections 2023-11-16T16:47:33.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.921+0000 7f54821ff700 1 -- 172.21.15.2:0/1697969899 wait complete. 2023-11-16T16:47:33.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.922+0000 7f54821ff700 1 Processor -- start 2023-11-16T16:47:33.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.922+0000 7f54821ff700 1 -- start start 2023-11-16T16:47:33.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.922+0000 7f54821ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f547c100230 0x7f547c1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.922+0000 7f54821ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f547c101420 0x7f547c1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.922+0000 7f54821ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f547c101db0 0x7f547c19f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.922+0000 7f54821ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547c19fe30 con 0x7f547c100230 2023-11-16T16:47:33.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.922+0000 7f54821ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547c19ff70 con 0x7f547c101db0 2023-11-16T16:47:33.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.922+0000 7f54821ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547c1a0100 con 0x7f547c101420 2023-11-16T16:47:33.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.923+0000 7f547b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f547c100230 0x7f547c1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.923+0000 7f5473fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f547c101420 0x7f547c1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.929 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.923+0000 7f5473fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f547c101420 0x7f547c1987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:50824/0 (socket says 172.21.15.2:50824) 2023-11-16T16:47:33.929 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.923+0000 7f5473fff700 1 -- 172.21.15.2:0/2742944783 learned_addr learned my addr 172.21.15.2:0/2742944783 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:33.929 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.923+0000 7f547bfff700 1 --2- 172.21.15.2:0/2742944783 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f547c101db0 0x7f547c19f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.929 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.923+0000 7f5473fff700 1 -- 172.21.15.2:0/2742944783 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f547c101db0 msgr2=0x7f547c19f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:33.929 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.923+0000 7f5473fff700 1 --2- 172.21.15.2:0/2742944783 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f547c101db0 0x7f547c19f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.929 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.923+0000 7f5473fff700 1 -- 172.21.15.2:0/2742944783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f547c100230 msgr2=0x7f547c1982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:33.929 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.923+0000 7f5473fff700 1 --2- 172.21.15.2:0/2742944783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f547c100230 0x7f547c1982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.929 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.923+0000 7f5473fff700 1 -- 172.21.15.2:0/2742944783 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5474009c40 con 0x7f547c101420 2023-11-16T16:47:33.930 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.923+0000 7f547b7fe700 1 --2- 172.21.15.2:0/2742944783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f547c100230 0x7f547c1982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:33.930 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.923+0000 7f5473fff700 1 --2- 172.21.15.2:0/2742944783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f547c101420 0x7f547c1987f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f546400e9a0 tx=0x7f546400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:33.930 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.924+0000 7f54797fa700 1 -- 172.21.15.2:0/2742944783 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5464009790 con 0x7f547c101420 2023-11-16T16:47:33.930 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.924+0000 7f54797fa700 1 -- 172.21.15.2:0/2742944783 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f546400ce80 con 0x7f547c101420 2023-11-16T16:47:33.930 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.924+0000 7f54821ff700 1 -- 172.21.15.2:0/2742944783 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f547c10bc00 con 0x7f547c101420 2023-11-16T16:47:33.930 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.924+0000 7f54797fa700 1 -- 172.21.15.2:0/2742944783 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5464018680 con 0x7f547c101420 2023-11-16T16:47:33.930 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.924+0000 7f54821ff700 1 -- 172.21.15.2:0/2742944783 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f547c10ba70 con 0x7f547c101420 2023-11-16T16:47:33.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.927+0000 7f54797fa700 1 -- 172.21.15.2:0/2742944783 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f5464010430 con 0x7f547c101420 2023-11-16T16:47:33.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.928+0000 7f5495ae6b80 1 Processor -- start 2023-11-16T16:47:33.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.928+0000 7f54797fa700 1 --2- 172.21.15.2:0/2742944783 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f54680a11d0 0x7f54680a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.928+0000 7f54797fa700 1 -- 172.21.15.2:0/2742944783 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5464014070 con 0x7f547c101420 2023-11-16T16:47:33.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.929+0000 7f5495ae6b80 1 -- start start 2023-11-16T16:47:33.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.929+0000 7f547bfff700 1 --2- 172.21.15.2:0/2742944783 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f54680a11d0 0x7f54680a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.929+0000 7f5495ae6b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f08bb0 0x55af29f08fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.929+0000 7f5495ae6b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 0x55af29f09ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.929+0000 7f5495ae6b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f0a0c0 0x55af29f0a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.929+0000 7f5495ae6b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af29e0e010 con 0x55af29f0a0c0 2023-11-16T16:47:33.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.929+0000 7f5495ae6b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af29e60850 con 0x55af29f08bb0 2023-11-16T16:47:33.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.929+0000 7f5495ae6b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af29e59ce0 con 0x55af29f09500 2023-11-16T16:47:33.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.929+0000 7f5473fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 0x55af29f09ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.929+0000 7f5473fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 0x55af29f09ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:50826/0 (socket says 172.21.15.2:50826) 2023-11-16T16:47:33.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.929+0000 7f5473fff700 1 -- 172.21.15.2:0/860872914 learned_addr learned my addr 172.21.15.2:0/860872914 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:33.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.929+0000 7f547bfff700 1 --2- 172.21.15.2:0/860872914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f0a0c0 0x55af29f0a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.929+0000 7f547b7fe700 1 --2- 172.21.15.2:0/860872914 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f08bb0 0x55af29f08fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.930+0000 7f547bfff700 1 --2- 172.21.15.2:0/2742944783 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f54680a11d0 0x7f54680a3680 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5474009790 tx=0x7f54740096c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:33.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.930+0000 7f5473fff700 1 -- 172.21.15.2:0/860872914 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f08bb0 msgr2=0x55af29f08fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:47:33.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.930+0000 7f5473fff700 1 --2- 172.21.15.2:0/860872914 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f08bb0 0x55af29f08fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.930+0000 7f5473fff700 1 -- 172.21.15.2:0/860872914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f0a0c0 msgr2=0x55af29f0a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:47:33.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.930+0000 7f5473fff700 1 --2- 172.21.15.2:0/860872914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f0a0c0 0x55af29f0a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.930+0000 7f5473fff700 1 -- 172.21.15.2:0/860872914 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af29cdb5e0 con 0x55af29f09500 2023-11-16T16:47:33.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.930+0000 7f5473fff700 1 --2- 172.21.15.2:0/860872914 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 0x55af29f09ae0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5464099df0 tx=0x7f5464099890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db99f0d040bdbf84 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:33.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.930+0000 7f5471ffb700 1 -- 172.21.15.2:0/860872914 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546409de90 con 0x55af29f09500 2023-11-16T16:47:33.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.930+0000 7f5471ffb700 1 -- 172.21.15.2:0/860872914 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54640a22b0 con 0x55af29f09500 2023-11-16T16:47:33.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.930+0000 7f5471ffb700 1 -- 172.21.15.2:0/860872914 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54640a2080 con 0x55af29f09500 2023-11-16T16:47:33.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.931+0000 7f5495ae6b80 1 -- 172.21.15.2:0/860872914 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 msgr2=0x55af29f09ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:33.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.931+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/860872914 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 0x55af29f09ae0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5464099df0 tx=0x7f5464099890 comp rx=0 tx=0).stop 2023-11-16T16:47:33.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.931+0000 7f5495ae6b80 1 -- 172.21.15.2:0/860872914 shutdown_connections 2023-11-16T16:47:33.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.931+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/860872914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f0a0c0 0x55af29f0a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.931+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/860872914 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f08bb0 0x55af29f08fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.931+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/860872914 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 0x55af29f09ae0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.931+0000 7f5495ae6b80 1 -- 172.21.15.2:0/860872914 >> 172.21.15.2:0/860872914 conn(0x55af29efe310 msgr2=0x55af29f03620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:33.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.931+0000 7f5495ae6b80 1 -- 172.21.15.2:0/860872914 shutdown_connections 2023-11-16T16:47:33.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.931+0000 7f5495ae6b80 1 -- 172.21.15.2:0/860872914 wait complete. 2023-11-16T16:47:33.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.932+0000 7f5495ae6b80 1 -- start start 2023-11-16T16:47:33.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.932+0000 7f5495ae6b80 1 Processor -- start 2023-11-16T16:47:33.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.932+0000 7f5495ae6b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f08bb0 0x55af29f12520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.932+0000 7f5495ae6b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 0x55af29f12a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.932+0000 7f5495ae6b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f0a0c0 0x55af29f19aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.932+0000 7f5495ae6b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af29dfb890 con 0x55af29f0a0c0 2023-11-16T16:47:33.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.932+0000 7f5495ae6b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af29dfb530 con 0x55af29f08bb0 2023-11-16T16:47:33.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.932+0000 7f5495ae6b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af29cdd8d0 con 0x55af29f09500 2023-11-16T16:47:33.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.932+0000 7f5495ae6b80 10 client.0 _ll_register_callbacks cb 0x55af29e19730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:47:33.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.932+0000 7f5473fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 0x55af29f12a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.933+0000 7f547bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f0a0c0 0x55af29f19aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.933+0000 7f5473fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 0x55af29f12a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:50840/0 (socket says 172.21.15.2:50840) 2023-11-16T16:47:33.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.933+0000 7f5473fff700 1 -- 172.21.15.2:0/4238000182 learned_addr learned my addr 172.21.15.2:0/4238000182 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:33.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.933+0000 7f547b7fe700 1 --2- 172.21.15.2:0/4238000182 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f08bb0 0x55af29f12520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.933+0000 7f5473fff700 1 -- 172.21.15.2:0/4238000182 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f08bb0 msgr2=0x55af29f12520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:47:33.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.933+0000 7f5473fff700 1 --2- 172.21.15.2:0/4238000182 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f08bb0 0x55af29f12520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.933+0000 7f5473fff700 1 -- 172.21.15.2:0/4238000182 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f0a0c0 msgr2=0x55af29f19aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:47:33.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.933+0000 7f5473fff700 1 --2- 172.21.15.2:0/4238000182 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f0a0c0 0x55af29f19aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.933+0000 7f5473fff700 1 -- 172.21.15.2:0/4238000182 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f547c10ba70 con 0x55af29f09500 2023-11-16T16:47:33.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.933+0000 7f5473fff700 1 --2- 172.21.15.2:0/4238000182 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 0x55af29f12a60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5464010b50 tx=0x7f5464095b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5670650900322edd server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:33.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.933+0000 7f545affd700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:47:33.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.933+0000 7f545affd700 1 -- 172.21.15.2:0/4238000182 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546409df10 con 0x55af29f09500 2023-11-16T16:47:33.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.933+0000 7f5495ae6b80 1 -- 172.21.15.2:0/4238000182 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55af29efbad0 con 0x55af29f09500 2023-11-16T16:47:33.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.933+0000 7f545affd700 1 -- 172.21.15.2:0/4238000182 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f54640adb60 con 0x55af29f09500 2023-11-16T16:47:33.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.934+0000 7f545affd700 1 -- 172.21.15.2:0/4238000182 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54640ca3b0 con 0x55af29f09500 2023-11-16T16:47:33.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.934+0000 7f5495ae6b80 10 client.24682 fetch_fsmap learned FSMap version 17 2023-11-16T16:47:33.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.934+0000 7f5495ae6b80 1 -- 172.21.15.2:0/4238000182 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x55af29e20140 con 0x55af29f09500 2023-11-16T16:47:33.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.934+0000 7f545affd700 1 -- 172.21.15.2:0/4238000182 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55af29efbad0 con 0x55af29f09500 2023-11-16T16:47:33.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.934+0000 7f545affd700 1 -- 172.21.15.2:0/4238000182 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f54640b2290 con 0x55af29f09500 2023-11-16T16:47:33.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.934+0000 7f5495ae6b80 10 client.24682 fetch_fsmap finished waiting for FSMap version 17 2023-11-16T16:47:33.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.934+0000 7f5495ae6b80 10 client.24682 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T16:47:33.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.934+0000 7f5495ae6b80 20 client.24682 populate_metadata read hostname 'smithi002' 2023-11-16T16:47:33.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.934+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/4238000182 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55af29f1da10 0x55af29f1c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.934+0000 7f5495ae6b80 4 client.24682 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:47:33.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.934+0000 7f5495ae6b80 1 -- 172.21.15.2:0/4238000182 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55af29e1c050 con 0x55af29f1da10 2023-11-16T16:47:33.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.935+0000 7f547b7fe700 1 --2- 172.21.15.2:0/4238000182 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55af29f1da10 0x55af29f1c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.935+0000 7f547b7fe700 1 --2- 172.21.15.2:0/4238000182 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55af29f1da10 0x55af29f1c9d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c6b596665c912ff server_cookie=3908d0d2ff6913cb in_seq=0 out_seq=0 2023-11-16T16:47:33.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.935+0000 7f545affd700 10 client.24682 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:47:33.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.936+0000 7f545affd700 1 -- 172.21.15.2:0/4238000182 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55af29e1c050 con 0x55af29f1da10 2023-11-16T16:47:33.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.936+0000 7f545affd700 10 client.24682 handle_command_reply: tid=0 2023-11-16T16:47:33.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.936+0000 7f5495ae6b80 1 client.24682 shutdown 2023-11-16T16:47:33.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.936+0000 7f5495ae6b80 1 -- 172.21.15.2:0/4238000182 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 msgr2=0x55af29f12a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:33.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.936+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/4238000182 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 0x55af29f12a60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5464010b50 tx=0x7f5464095b90 comp rx=0 tx=0).stop 2023-11-16T16:47:33.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.936+0000 7f5495ae6b80 1 -- 172.21.15.2:0/4238000182 shutdown_connections 2023-11-16T16:47:33.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.936+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/4238000182 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f0a0c0 0x55af29f19aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.936+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/4238000182 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f08bb0 0x55af29f12520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.936+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/4238000182 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55af29f1da10 0x55af29f1c9d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.936+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/4238000182 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 0x55af29f12a60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.949 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.936+0000 7f5495ae6b80 1 -- 172.21.15.2:0/4238000182 >> 172.21.15.2:0/4238000182 conn(0x55af29efe310 msgr2=0x55af29f02d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:33.949 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.936+0000 7f545affd700 0 client.24682 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:47:33.949 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.936+0000 7f5495ae6b80 1 -- 172.21.15.2:0/4238000182 shutdown_connections 2023-11-16T16:47:33.949 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.936+0000 7f5495ae6b80 1 -- 172.21.15.2:0/4238000182 wait complete. 2023-11-16T16:47:33.949 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.937+0000 7f5495ae6b80 20 client.24682 trim_cache size 0 max 16384 2023-11-16T16:47:33.960 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.955+0000 7f5495ae6b80 1 Processor -- start 2023-11-16T16:47:33.960 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.955+0000 7f5495ae6b80 1 -- start start 2023-11-16T16:47:33.960 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.955+0000 7f5495ae6b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f08bb0 0x55af29f111a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.960 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.956+0000 7f5495ae6b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 0x55af29f19a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.960 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.956+0000 7f5495ae6b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f0a0c0 0x55af29f0cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.960 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.956+0000 7f5495ae6b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af29e1c050 con 0x55af29f08bb0 2023-11-16T16:47:33.960 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.956+0000 7f5495ae6b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af29efe120 con 0x55af29f0a0c0 2023-11-16T16:47:33.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.956+0000 7f5495ae6b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af29f13710 con 0x55af29f09500 2023-11-16T16:47:33.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.956+0000 7f547b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f08bb0 0x55af29f111a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.956+0000 7f5473fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 0x55af29f19a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.956+0000 7f547b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f08bb0 0x55af29f111a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50236/0 (socket says 172.21.15.2:50236) 2023-11-16T16:47:33.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.956+0000 7f547b7fe700 1 -- 172.21.15.2:0/1537391048 learned_addr learned my addr 172.21.15.2:0/1537391048 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:33.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.956+0000 7f547bfff700 1 --2- 172.21.15.2:0/1537391048 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f0a0c0 0x55af29f0cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.956+0000 7f547b7fe700 1 -- 172.21.15.2:0/1537391048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 msgr2=0x55af29f19a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:47:33.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.956+0000 7f547b7fe700 1 --2- 172.21.15.2:0/1537391048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 0x55af29f19a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.962 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.956+0000 7f547b7fe700 1 -- 172.21.15.2:0/1537391048 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f0a0c0 msgr2=0x55af29f0cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:47:33.962 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.956+0000 7f547b7fe700 1 --2- 172.21.15.2:0/1537391048 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f0a0c0 0x55af29f0cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.962 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.956+0000 7f547b7fe700 1 -- 172.21.15.2:0/1537391048 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af29e1fe90 con 0x55af29f08bb0 2023-11-16T16:47:33.962 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.956+0000 7f547bfff700 1 --2- 172.21.15.2:0/1537391048 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f0a0c0 0x55af29f0cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:33.962 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.956+0000 7f5473fff700 1 --2- 172.21.15.2:0/1537391048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 0x55af29f19a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:33.962 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.956+0000 7f547b7fe700 1 --2- 172.21.15.2:0/1537391048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f08bb0 0x55af29f111a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x55af29f124c0 tx=0x7f546c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb56941cb509693 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:33.962 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.957+0000 7f54727fc700 1 -- 172.21.15.2:0/1537391048 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c017070 con 0x55af29f08bb0 2023-11-16T16:47:33.962 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.957+0000 7f54727fc700 1 -- 172.21.15.2:0/1537391048 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f546c018040 con 0x55af29f08bb0 2023-11-16T16:47:33.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.957+0000 7f54727fc700 1 -- 172.21.15.2:0/1537391048 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c021480 con 0x55af29f08bb0 2023-11-16T16:47:33.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.957+0000 7f5495ae6b80 1 -- 172.21.15.2:0/1537391048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f08bb0 msgr2=0x55af29f111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:33.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.957+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/1537391048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f08bb0 0x55af29f111a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x55af29f124c0 tx=0x7f546c009310 comp rx=0 tx=0).stop 2023-11-16T16:47:33.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.957+0000 7f5495ae6b80 1 -- 172.21.15.2:0/1537391048 shutdown_connections 2023-11-16T16:47:33.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.957+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/1537391048 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f0a0c0 0x55af29f0cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.957+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/1537391048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f08bb0 0x55af29f111a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.957+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/1537391048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f09500 0x55af29f19a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.957+0000 7f5495ae6b80 1 -- 172.21.15.2:0/1537391048 >> 172.21.15.2:0/1537391048 conn(0x55af29efe310 msgr2=0x55af29f26170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:33.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.957+0000 7f5495ae6b80 1 -- 172.21.15.2:0/1537391048 shutdown_connections 2023-11-16T16:47:33.964 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.958+0000 7f5495ae6b80 1 -- 172.21.15.2:0/1537391048 wait complete. 2023-11-16T16:47:33.964 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.958+0000 7f5495ae6b80 1 -- start start 2023-11-16T16:47:33.964 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.958+0000 7f5495ae6b80 1 Processor -- start 2023-11-16T16:47:33.968 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.959+0000 7f5495ae6b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f08bb0 0x55af29f14cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.968 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.959+0000 7f5495ae6b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f09500 0x55af29f15210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.968 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.959+0000 7f5495ae6b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f0a0c0 0x55af29f712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.968 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.959+0000 7f5495ae6b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af29f158e0 con 0x55af29f09500 2023-11-16T16:47:33.968 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.959+0000 7f5495ae6b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af29f717e0 con 0x55af29f0a0c0 2023-11-16T16:47:33.968 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.959+0000 7f5495ae6b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af29f71970 con 0x55af29f08bb0 2023-11-16T16:47:33.968 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.959+0000 7f5495ae6b80 10 client.0 _ll_register_callbacks cb 0x7f5474009b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:47:33.968 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.959+0000 7f5473fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f09500 0x55af29f15210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.959+0000 7f547b7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f08bb0 0x55af29f14cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.959+0000 7f547bfff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f0a0c0 0x55af29f712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.959+0000 7f5473fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f09500 0x55af29f15210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50238/0 (socket says 172.21.15.2:50238) 2023-11-16T16:47:33.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.959+0000 7f5473fff700 1 -- 172.21.15.2:0/3977443048 learned_addr learned my addr 172.21.15.2:0/3977443048 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:33.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.959+0000 7f547b7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f08bb0 0x55af29f14cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:50864/0 (socket says 172.21.15.2:50864) 2023-11-16T16:47:33.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.959+0000 7f5473fff700 1 -- 172.21.15.2:0/3977443048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f08bb0 msgr2=0x55af29f14cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:47:33.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.959+0000 7f5473fff700 1 --2- 172.21.15.2:0/3977443048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f08bb0 0x55af29f14cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.959+0000 7f5473fff700 1 -- 172.21.15.2:0/3977443048 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f0a0c0 msgr2=0x55af29f712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:47:33.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.959+0000 7f5473fff700 1 --2- 172.21.15.2:0/3977443048 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f0a0c0 0x55af29f712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.959+0000 7f5473fff700 1 -- 172.21.15.2:0/3977443048 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55af29e20140 con 0x55af29f09500 2023-11-16T16:47:33.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.960+0000 7f5473fff700 1 --2- 172.21.15.2:0/3977443048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f09500 0x55af29f15210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f54640968d0 tx=0x7f54640993c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1820340e73f2d217 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:33.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.960+0000 7f545affd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:47:33.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.960+0000 7f545affd700 1 -- 172.21.15.2:0/3977443048 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54640b24a0 con 0x55af29f09500 2023-11-16T16:47:33.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.960+0000 7f545affd700 1 -- 172.21.15.2:0/3977443048 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f54640ca3b0 con 0x55af29f09500 2023-11-16T16:47:33.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.960+0000 7f5495ae6b80 1 -- 172.21.15.2:0/3977443048 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f54640a22b0 con 0x55af29f09500 2023-11-16T16:47:33.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.960+0000 7f545affd700 1 -- 172.21.15.2:0/3977443048 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5464099430 con 0x55af29f09500 2023-11-16T16:47:33.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.960+0000 7f5495ae6b80 10 client.14670 fetch_fsmap learned FSMap version 17 2023-11-16T16:47:33.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.960+0000 7f5495ae6b80 1 -- 172.21.15.2:0/3977443048 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x7f546c004040 con 0x55af29f09500 2023-11-16T16:47:33.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.960+0000 7f545affd700 1 -- 172.21.15.2:0/3977443048 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54640a22b0 con 0x55af29f09500 2023-11-16T16:47:33.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.960+0000 7f545affd700 1 -- 172.21.15.2:0/3977443048 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f546407d070 con 0x55af29f09500 2023-11-16T16:47:33.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.960+0000 7f5495ae6b80 10 client.14670 fetch_fsmap finished waiting for FSMap version 17 2023-11-16T16:47:33.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.960+0000 7f5495ae6b80 10 client.14670 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T16:47:33.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.960+0000 7f5495ae6b80 20 client.14670 populate_metadata read hostname 'smithi002' 2023-11-16T16:47:33.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.960+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/3977443048 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55af29f76250 0x55af29f72bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:33.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.961+0000 7f5495ae6b80 4 client.14670 mds_command: new command op to 24590 tid=0[{"prefix": "status"}] 2023-11-16T16:47:33.971 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.961+0000 7f5495ae6b80 1 -- 172.21.15.2:0/3977443048 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55af29e28610 con 0x55af29f76250 2023-11-16T16:47:33.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.961+0000 7f547b7fe700 1 --2- 172.21.15.2:0/3977443048 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55af29f76250 0x55af29f72bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:33.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.961+0000 7f547b7fe700 1 --2- 172.21.15.2:0/3977443048 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55af29f76250 0x55af29f72bc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a1568987eb7a895 server_cookie=376be7f0bce21ad0 in_seq=0 out_seq=0 2023-11-16T16:47:33.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.961+0000 7f545affd700 10 client.14670 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:47:33.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.961+0000 7f545affd700 1 -- 172.21.15.2:0/3977443048 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55af29e28610 con 0x55af29f76250 2023-11-16T16:47:33.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.961+0000 7f545affd700 10 client.14670 handle_command_reply: tid=0 2023-11-16T16:47:33.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.962+0000 7f5495ae6b80 1 client.14670 shutdown 2023-11-16T16:47:33.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.962+0000 7f5495ae6b80 1 -- 172.21.15.2:0/3977443048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f09500 msgr2=0x55af29f15210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:33.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.962+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/3977443048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f09500 0x55af29f15210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f54640968d0 tx=0x7f54640993c0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.962+0000 7f5495ae6b80 1 -- 172.21.15.2:0/3977443048 shutdown_connections 2023-11-16T16:47:33.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.962+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/3977443048 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55af29f76250 0x55af29f72bc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.962+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/3977443048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55af29f08bb0 0x55af29f14cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.962+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/3977443048 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55af29f0a0c0 0x55af29f712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.962+0000 7f545affd700 0 client.14670 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:47:33.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.962+0000 7f5495ae6b80 1 --2- 172.21.15.2:0/3977443048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55af29f09500 0x55af29f15210 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.962+0000 7f5495ae6b80 1 -- 172.21.15.2:0/3977443048 >> 172.21.15.2:0/3977443048 conn(0x55af29efe310 msgr2=0x55af29f111a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:33.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.962+0000 7f5495ae6b80 1 -- 172.21.15.2:0/3977443048 shutdown_connections 2023-11-16T16:47:33.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.962+0000 7f5495ae6b80 1 -- 172.21.15.2:0/3977443048 wait complete. 2023-11-16T16:47:33.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.962+0000 7f5495ae6b80 20 client.14670 trim_cache size 0 max 16384 2023-11-16T16:47:33.974 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T16:47:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "875fcd24-849e-11ee-8dbe-212e2dc638e7", 2023-11-16T16:47:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2023-11-16T16:47:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24590, 2023-11-16T16:47:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2023-11-16T16:47:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2023-11-16T16:47:33.974 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2023-11-16T16:47:33.975 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 7.6109484749999998, 2023-11-16T16:47:33.975 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 17, 2023-11-16T16:47:33.975 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 89, 2023-11-16T16:47:33.975 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 88, 2023-11-16T16:47:33.975 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 13.65757612 2023-11-16T16:47:33.975 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T16:47:33.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.963+0000 7f54821ff700 1 -- 172.21.15.2:0/2742944783 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f54680a11d0 msgr2=0x7f54680a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:33.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.963+0000 7f54821ff700 1 --2- 172.21.15.2:0/2742944783 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f54680a11d0 0x7f54680a3680 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5474009790 tx=0x7f54740096c0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.963+0000 7f54821ff700 1 -- 172.21.15.2:0/2742944783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f547c101420 msgr2=0x7f547c1987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:33.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.963+0000 7f54821ff700 1 --2- 172.21.15.2:0/2742944783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f547c101420 0x7f547c1987f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f546400e9a0 tx=0x7f546400ecb0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.963+0000 7f54821ff700 1 -- 172.21.15.2:0/2742944783 shutdown_connections 2023-11-16T16:47:33.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.963+0000 7f54821ff700 1 --2- 172.21.15.2:0/2742944783 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f54680a11d0 0x7f54680a3680 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.963+0000 7f54821ff700 1 --2- 172.21.15.2:0/2742944783 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f547c101db0 0x7f547c19f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.963+0000 7f54821ff700 1 --2- 172.21.15.2:0/2742944783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f547c100230 0x7f547c1982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.963+0000 7f54821ff700 1 --2- 172.21.15.2:0/2742944783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f547c101420 0x7f547c1987f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:33.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.963+0000 7f54821ff700 1 -- 172.21.15.2:0/2742944783 >> 172.21.15.2:0/2742944783 conn(0x7f547c0fb810 msgr2=0x7f547c106c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:33.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.963+0000 7f54821ff700 1 -- 172.21.15.2:0/2742944783 shutdown_connections 2023-11-16T16:47:33.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:33.963+0000 7f54821ff700 1 -- 172.21.15.2:0/2742944783 wait complete. 2023-11-16T16:47:33.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:33 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1882276720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:33.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:33 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1678107786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:33.983 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-16T16:47:33.983 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-11-16T16:47:33.983 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph fs ls 2023-11-16T16:47:34.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:33 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1882276720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:34.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:33 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1678107786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:34.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:33 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1882276720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:34.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:33 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1678107786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:34.837 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:47:35.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:34 smithi155 ceph-mon[131607]: pgmap v284: 129 pgs: 16 unknown, 113 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 8 op/s 2023-11-16T16:47:35.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:34 smithi002 ceph-mon[128712]: pgmap v284: 129 pgs: 16 unknown, 113 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 8 op/s 2023-11-16T16:47:35.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:34 smithi141 ceph-mon[131184]: pgmap v284: 129 pgs: 16 unknown, 113 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 8 op/s 2023-11-16T16:47:36.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.367+0000 7f052715f700 1 -- 172.21.15.2:0/2186521699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05201055e0 msgr2=0x7f052010d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:36.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.367+0000 7f052715f700 1 --2- 172.21.15.2:0/2186521699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05201055e0 0x7f052010d160 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f051800b0d0 tx=0x7f051800b3e0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.367+0000 7f052715f700 1 -- 172.21.15.2:0/2186521699 shutdown_connections 2023-11-16T16:47:36.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.367+0000 7f052715f700 1 --2- 172.21.15.2:0/2186521699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05201055e0 0x7f052010d160 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.367+0000 7f052715f700 1 --2- 172.21.15.2:0/2186521699 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0520100540 0x7f0520100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.367+0000 7f052715f700 1 --2- 172.21.15.2:0/2186521699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0520100ee0 0x7f0520104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.367+0000 7f052715f700 1 -- 172.21.15.2:0/2186521699 >> 172.21.15.2:0/2186521699 conn(0x7f05200fbdc0 msgr2=0x7f05200fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:36.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.367+0000 7f052715f700 1 -- 172.21.15.2:0/2186521699 shutdown_connections 2023-11-16T16:47:36.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.367+0000 7f052715f700 1 -- 172.21.15.2:0/2186521699 wait complete. 2023-11-16T16:47:36.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.369+0000 7f052715f700 1 Processor -- start 2023-11-16T16:47:36.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.369+0000 7f052715f700 1 -- start start 2023-11-16T16:47:36.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.370+0000 7f052715f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0520100540 0x7f05201986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:36.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.370+0000 7f052715f700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0520100ee0 0x7f0520198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:36.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.370+0000 7f052715f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05201055e0 0x7f052019bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:36.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.370+0000 7f052715f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052019c5d0 con 0x7f05201055e0 2023-11-16T16:47:36.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.370+0000 7f052715f700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052019c740 con 0x7f0520100ee0 2023-11-16T16:47:36.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.370+0000 7f052715f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052019c900 con 0x7f0520100540 2023-11-16T16:47:36.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.370+0000 7f051ffff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0520100ee0 0x7f0520198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:36.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.370+0000 7f0524efb700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0520100540 0x7f05201986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:36.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.370+0000 7f051ffff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0520100ee0 0x7f0520198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:48478/0 (socket says 172.21.15.2:48478) 2023-11-16T16:47:36.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.370+0000 7f051ffff700 1 -- 172.21.15.2:0/553090852 learned_addr learned my addr 172.21.15.2:0/553090852 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:36.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.370+0000 7f05256fc700 1 --2- 172.21.15.2:0/553090852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05201055e0 0x7f052019bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:36.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.370+0000 7f0524efb700 1 -- 172.21.15.2:0/553090852 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0520100ee0 msgr2=0x7f0520198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:36.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.370+0000 7f0524efb700 1 --2- 172.21.15.2:0/553090852 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0520100ee0 0x7f0520198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.370+0000 7f0524efb700 1 -- 172.21.15.2:0/553090852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05201055e0 msgr2=0x7f052019bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:36.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.370+0000 7f0524efb700 1 --2- 172.21.15.2:0/553090852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05201055e0 0x7f052019bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.371+0000 7f0524efb700 1 -- 172.21.15.2:0/553090852 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0518009d00 con 0x7f0520100540 2023-11-16T16:47:36.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.371+0000 7f051ffff700 1 --2- 172.21.15.2:0/553090852 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0520100ee0 0x7f0520198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:36.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.371+0000 7f05256fc700 1 --2- 172.21.15.2:0/553090852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05201055e0 0x7f052019bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:36.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.371+0000 7f0524efb700 1 --2- 172.21.15.2:0/553090852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0520100540 0x7f05201986d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f051400e9f0 tx=0x7f051400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:36.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.371+0000 7f051dffb700 1 -- 172.21.15.2:0/553090852 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051400c9c0 con 0x7f0520100540 2023-11-16T16:47:36.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.371+0000 7f051dffb700 1 -- 172.21.15.2:0/553090852 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0514010430 con 0x7f0520100540 2023-11-16T16:47:36.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.371+0000 7f052715f700 1 -- 172.21.15.2:0/553090852 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0520110370 con 0x7f0520100540 2023-11-16T16:47:36.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.371+0000 7f051dffb700 1 -- 172.21.15.2:0/553090852 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0514018680 con 0x7f0520100540 2023-11-16T16:47:36.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.371+0000 7f052715f700 1 -- 172.21.15.2:0/553090852 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05201101e0 con 0x7f0520100540 2023-11-16T16:47:36.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.372+0000 7f052715f700 1 -- 172.21.15.2:0/553090852 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0520110500 con 0x7f0520100540 2023-11-16T16:47:36.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.374+0000 7f051dffb700 1 -- 172.21.15.2:0/553090852 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f0514010860 con 0x7f0520100540 2023-11-16T16:47:36.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.376+0000 7f051dffb700 1 --2- 172.21.15.2:0/553090852 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f05100a1150 0x7f05100a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:36.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.376+0000 7f051dffb700 1 -- 172.21.15.2:0/553090852 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0514014070 con 0x7f0520100540 2023-11-16T16:47:36.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.376+0000 7f051ffff700 1 --2- 172.21.15.2:0/553090852 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f05100a1150 0x7f05100a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:36.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.376+0000 7f051ffff700 1 --2- 172.21.15.2:0/553090852 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f05100a1150 0x7f05100a3600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0520199cf0 tx=0x7f050c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:36.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.378+0000 7f051dffb700 1 -- 172.21.15.2:0/553090852 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0514096340 con 0x7f0520100540 2023-11-16T16:47:36.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.629+0000 7f052715f700 1 -- 172.21.15.2:0/553090852 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0520066ea0 con 0x7f0520100540 2023-11-16T16:47:36.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.630+0000 7f051dffb700 1 -- 172.21.15.2:0/553090852 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7f0514026070 con 0x7f0520100540 2023-11-16T16:47:36.632 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-11-16T16:47:36.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.632+0000 7f052715f700 1 -- 172.21.15.2:0/553090852 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f05100a1150 msgr2=0x7f05100a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:36.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.632+0000 7f052715f700 1 --2- 172.21.15.2:0/553090852 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f05100a1150 0x7f05100a3600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0520199cf0 tx=0x7f050c009260 comp rx=0 tx=0).stop 2023-11-16T16:47:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.632+0000 7f052715f700 1 -- 172.21.15.2:0/553090852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0520100540 msgr2=0x7f05201986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.632+0000 7f052715f700 1 --2- 172.21.15.2:0/553090852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0520100540 0x7f05201986d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f051400e9f0 tx=0x7f051400ed00 comp rx=0 tx=0).stop 2023-11-16T16:47:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.632+0000 7f052715f700 1 -- 172.21.15.2:0/553090852 shutdown_connections 2023-11-16T16:47:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.632+0000 7f052715f700 1 --2- 172.21.15.2:0/553090852 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f05100a1150 0x7f05100a3600 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.632+0000 7f052715f700 1 --2- 172.21.15.2:0/553090852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0520100540 0x7f05201986d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.632+0000 7f052715f700 1 --2- 172.21.15.2:0/553090852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05201055e0 0x7f052019bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.632+0000 7f052715f700 1 --2- 172.21.15.2:0/553090852 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0520100ee0 0x7f0520198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.632+0000 7f052715f700 1 -- 172.21.15.2:0/553090852 >> 172.21.15.2:0/553090852 conn(0x7f05200fbdc0 msgr2=0x7f052010b620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.632+0000 7f052715f700 1 -- 172.21.15.2:0/553090852 shutdown_connections 2023-11-16T16:47:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:36.633+0000 7f052715f700 1 -- 172.21.15.2:0/553090852 wait complete. 2023-11-16T16:47:37.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:36 smithi155 ceph-mon[131607]: pgmap v285: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 3 op/s 2023-11-16T16:47:37.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:36 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/553090852' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T16:47:37.125 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:36 smithi002 ceph-mon[128712]: pgmap v285: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 3 op/s 2023-11-16T16:47:37.125 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:36 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/553090852' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T16:47:37.129 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-16T16:47:37.129 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-16T16:47:37.130 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi002.front.sepia.ceph.com 2023-11-16T16:47:37.130 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-11-16T16:47:37.130 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-11-16T16:47:37.130 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-16T16:47:37.130 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-16T16:47:37.130 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-16T16:47:37.130 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-11-16T16:47:37.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:37.130 DEBUG:teuthology.orchestra.run.smithi002:> ip addr 2023-11-16T16:47:37.177 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-16T16:47:37.177 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-16T16:47:37.178 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2023-11-16T16:47:37.178 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2023-11-16T16:47:37.178 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2023-11-16T16:47:37.178 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2023-11-16T16:47:37.178 INFO:teuthology.orchestra.run.smithi002.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T16:47:37.178 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:02 brd ff:ff:ff:ff:ff:ff 2023-11-16T16:47:37.178 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f0 2023-11-16T16:47:37.178 INFO:teuthology.orchestra.run.smithi002.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T16:47:37.178 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:03 brd ff:ff:ff:ff:ff:ff 2023-11-16T16:47:37.178 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f1 2023-11-16T16:47:37.179 INFO:teuthology.orchestra.run.smithi002.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-16T16:47:37.179 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:20 brd ff:ff:ff:ff:ff:ff 2023-11-16T16:47:37.179 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f0 2023-11-16T16:47:37.179 INFO:teuthology.orchestra.run.smithi002.stdout: inet 172.21.15.2/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-16T16:47:37.179 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft 41554sec preferred_lft 41554sec 2023-11-16T16:47:37.179 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::ec4:7aff:febd:1920/64 scope link 2023-11-16T16:47:37.179 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2023-11-16T16:47:37.179 INFO:teuthology.orchestra.run.smithi002.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T16:47:37.179 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:21 brd ff:ff:ff:ff:ff:ff 2023-11-16T16:47:37.179 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f1 2023-11-16T16:47:37.180 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-16T16:47:37.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:37.180 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T16:47:37.180 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T16:47:37.180 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add name ceph-brx type bridge 2023-11-16T16:47:37.181 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr flush dev ceph-brx 2023-11-16T16:47:37.181 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set ceph-brx up 2023-11-16T16:47:37.181 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-16T16:47:37.181 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T16:47:37.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:36 smithi141 ceph-mon[131184]: pgmap v285: 129 pgs: 129 active+clean; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 3 op/s 2023-11-16T16:47:37.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:36 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/553090852' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T16:47:37.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:37.661 DEBUG:teuthology.orchestra.run.smithi002:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-16T16:47:37.734 INFO:teuthology.orchestra.run.smithi002.stdout:1 2023-11-16T16:47:37.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:37.739 DEBUG:teuthology.orchestra.run.smithi002:> route 2023-11-16T16:47:37.800 INFO:teuthology.orchestra.run.smithi002.stdout:Kernel IP routing table 2023-11-16T16:47:37.800 INFO:teuthology.orchestra.run.smithi002.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-16T16:47:37.800 INFO:teuthology.orchestra.run.smithi002.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-16T16:47:37.800 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-16T16:47:37.800 INFO:teuthology.orchestra.run.smithi002.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-16T16:47:37.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:37.802 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T16:47:37.802 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T16:47:37.802 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-16T16:47:37.802 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-16T16:47:37.802 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-16T16:47:37.802 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T16:47:38.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:38.512 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2023-11-16T16:47:38.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:38.557 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list-id 2023-11-16T16:47:38.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:38.643 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T16:47:38.643 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T16:47:38.643 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-11-16T16:47:38.643 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-11-16T16:47:38.643 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T16:47:38.835 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-11-16T16:47:38.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:38.836 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T16:47:38.836 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T16:47:38.836 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-11-16T16:47:38.836 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-16T16:47:38.836 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-11-16T16:47:38.836 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-11-16T16:47:38.836 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-11-16T16:47:38.836 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T16:47:39.039 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:38 smithi002 ceph-mon[128712]: pgmap v286: 129 pgs: 129 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 128 B/s wr, 3 op/s 2023-11-16T16:47:39.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:39.117 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T16:47:39.117 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T16:47:39.117 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set brx.0 up 2023-11-16T16:47:39.117 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set dev brx.0 master ceph-brx 2023-11-16T16:47:39.118 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T16:47:39.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:38 smithi155 ceph-mon[131607]: pgmap v286: 129 pgs: 129 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 128 B/s wr, 3 op/s 2023-11-16T16:47:39.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:38 smithi141 ceph-mon[131184]: pgmap v286: 129 pgs: 129 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 128 B/s wr, 3 op/s 2023-11-16T16:47:39.295 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-11-16T16:47:39.295 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-16T16:47:39.295 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-11-16T16:47:39.340 INFO:teuthology.orchestra.run.smithi002.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-11-16T16:47:39.340 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-16T16:47:39.340 DEBUG:teuthology.orchestra.run.smithi002:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-11-16T16:47:39.424 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2023-11-16T16:47:39.538 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-16T16:47:39.538 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-16T16:47:39.608 INFO:teuthology.orchestra.run.smithi002.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-16T16:47:39.611 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-16T16:47:39.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:39.612 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2023-11-16T16:47:39.657 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-16T16:47:39.657 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-11-16T16:47:39.702 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2023-11-16T16:47:39.774 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-16T16:47:39.774 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-16T16:47:39.845 INFO:teuthology.orchestra.run.smithi002.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-16T16:47:39.848 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-16T16:47:39.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:39.849 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2023-11-16T16:47:39.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi002.stderr:2023-11-16T16:47:39.906+0000 7fc14f0d15c0 -1 init, newargv = 0x55709c8784e0 newargc=15 2023-11-16T16:47:39.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi002.stderr:ceph-fuse[165445]: starting ceph client 2023-11-16T16:47:39.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi002.stderr:ceph-fuse[165445]: starting fuse 2023-11-16T16:47:40.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:40 smithi141 ceph-mon[131184]: pgmap v287: 129 pgs: 129 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 113 B/s wr, 2 op/s 2023-11-16T16:47:40.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:40 smithi002 ceph-mon[128712]: pgmap v287: 129 pgs: 129 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 113 B/s wr, 2 op/s 2023-11-16T16:47:40.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:40 smithi155 ceph-mon[131607]: pgmap v287: 129 pgs: 129 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 113 B/s wr, 2 op/s 2023-11-16T16:47:40.995 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2023-11-16T16:47:41.075 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-11-16T16:47:41.075 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-11-16T16:47:41.146 INFO:teuthology.orchestra.run.smithi002.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-11-16T16:47:41.149 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-11-16T16:47:41.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:41.150 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2023-11-16T16:47:41.195 INFO:teuthology.orchestra.run.smithi002.stdout:121 2023-11-16T16:47:41.195 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [121] 2023-11-16T16:47:41.196 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-11-16T16:47:41.196 DEBUG:teuthology.orchestra.run.smithi002:> import glob 2023-11-16T16:47:41.196 DEBUG:teuthology.orchestra.run.smithi002:> import re 2023-11-16T16:47:41.196 DEBUG:teuthology.orchestra.run.smithi002:> import os 2023-11-16T16:47:41.196 DEBUG:teuthology.orchestra.run.smithi002:> import subprocess 2023-11-16T16:47:41.196 DEBUG:teuthology.orchestra.run.smithi002:> 2023-11-16T16:47:41.196 DEBUG:teuthology.orchestra.run.smithi002:> def _find_admin_socket(client_name): 2023-11-16T16:47:41.197 DEBUG:teuthology.orchestra.run.smithi002:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-11-16T16:47:41.197 DEBUG:teuthology.orchestra.run.smithi002:> files = glob.glob(asok_path) 2023-11-16T16:47:41.197 DEBUG:teuthology.orchestra.run.smithi002:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-11-16T16:47:41.197 DEBUG:teuthology.orchestra.run.smithi002:> 2023-11-16T16:47:41.197 DEBUG:teuthology.orchestra.run.smithi002:> # Given a non-glob path, it better be there 2023-11-16T16:47:41.197 DEBUG:teuthology.orchestra.run.smithi002:> if "*" not in asok_path: 2023-11-16T16:47:41.197 DEBUG:teuthology.orchestra.run.smithi002:> assert(len(files) == 1) 2023-11-16T16:47:41.198 DEBUG:teuthology.orchestra.run.smithi002:> return files[0] 2023-11-16T16:47:41.198 DEBUG:teuthology.orchestra.run.smithi002:> 2023-11-16T16:47:41.198 DEBUG:teuthology.orchestra.run.smithi002:> for f in files: 2023-11-16T16:47:41.198 DEBUG:teuthology.orchestra.run.smithi002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-16T16:47:41.198 DEBUG:teuthology.orchestra.run.smithi002:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-16T16:47:41.198 DEBUG:teuthology.orchestra.run.smithi002:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-16T16:47:41.198 DEBUG:teuthology.orchestra.run.smithi002:> contents = proc_f.read() 2023-11-16T16:47:41.199 DEBUG:teuthology.orchestra.run.smithi002:> if mountpoint in contents: 2023-11-16T16:47:41.199 DEBUG:teuthology.orchestra.run.smithi002:> return f 2023-11-16T16:47:41.199 DEBUG:teuthology.orchestra.run.smithi002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-16T16:47:41.199 DEBUG:teuthology.orchestra.run.smithi002:> 2023-11-16T16:47:41.199 DEBUG:teuthology.orchestra.run.smithi002:> print(_find_admin_socket("client.admin")) 2023-11-16T16:47:41.199 DEBUG:teuthology.orchestra.run.smithi002:> ' 2023-11-16T16:47:41.383 INFO:teuthology.orchestra.run.smithi002.stdout:/var/run/ceph/ceph-client.admin.165445.asok 2023-11-16T16:47:41.559 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.165445.asok 2023-11-16T16:47:41.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:41.560 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.165445.asok status 2023-11-16T16:47:41.792 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T16:47:41.792 INFO:teuthology.orchestra.run.smithi002.stdout: "metadata": { 2023-11-16T16:47:41.792 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_sha1": "5e4a743ad7fd1422ebd6c4dc950a21a303279207", 2023-11-16T16:47:41.792 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev)", 2023-11-16T16:47:41.792 INFO:teuthology.orchestra.run.smithi002.stdout: "entity_id": "admin", 2023-11-16T16:47:41.792 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": "smithi002", 2023-11-16T16:47:41.792 INFO:teuthology.orchestra.run.smithi002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-11-16T16:47:41.792 INFO:teuthology.orchestra.run.smithi002.stdout: "pid": "165445", 2023-11-16T16:47:41.792 INFO:teuthology.orchestra.run.smithi002.stdout: "root": "/" 2023-11-16T16:47:41.792 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T16:47:41.793 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_count": 0, 2023-11-16T16:47:41.793 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_pinned_count": 0, 2023-11-16T16:47:41.793 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 14682, 2023-11-16T16:47:41.793 INFO:teuthology.orchestra.run.smithi002.stdout: "inst": { 2023-11-16T16:47:41.793 INFO:teuthology.orchestra.run.smithi002.stdout: "name": { 2023-11-16T16:47:41.793 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "client", 2023-11-16T16:47:41.793 INFO:teuthology.orchestra.run.smithi002.stdout: "num": 14682 2023-11-16T16:47:41.793 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T16:47:41.793 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2023-11-16T16:47:41.793 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-16T16:47:41.793 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2023-11-16T16:47:41.793 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1239981235 2023-11-16T16:47:41.794 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T16:47:41.794 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T16:47:41.794 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2023-11-16T16:47:41.794 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-16T16:47:41.794 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2023-11-16T16:47:41.794 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1239981235 2023-11-16T16:47:41.794 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T16:47:41.794 INFO:teuthology.orchestra.run.smithi002.stdout: "inst_str": "client.14682 172.21.15.2:0/1239981235", 2023-11-16T16:47:41.794 INFO:teuthology.orchestra.run.smithi002.stdout: "addr_str": "172.21.15.2:0/1239981235", 2023-11-16T16:47:41.794 INFO:teuthology.orchestra.run.smithi002.stdout: "inode_count": 1, 2023-11-16T16:47:41.794 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_epoch": 17, 2023-11-16T16:47:41.794 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch": 89, 2023-11-16T16:47:41.795 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch_barrier": 0, 2023-11-16T16:47:41.795 INFO:teuthology.orchestra.run.smithi002.stdout: "blocklisted": false, 2023-11-16T16:47:41.795 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs" 2023-11-16T16:47:41.795 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T16:47:41.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:41.807 DEBUG:teuthology.orchestra.run.smithi002:> stat --file-system '--printf=%T 2023-11-16T16:47:41.807 DEBUG:teuthology.orchestra.run.smithi002:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-11-16T16:47:41.853 INFO:teuthology.orchestra.run.smithi002.stdout:fuseblk 2023-11-16T16:47:41.854 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-11-16T16:47:41.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:41.854 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-11-16T16:47:41.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:41.966 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-11-16T16:47:42.099 INFO:teuthology.orchestra.run.smithi002.stdout:# file: . 2023-11-16T16:47:42.099 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-11-16T16:47:42.099 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:42.100 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T16:47:42.101 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-16T16:47:42.147 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:47:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.390+0000 7f7927dff700 1 -- 172.21.15.2:0/670381405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7920100290 msgr2=0x7f79201006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:42.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.390+0000 7f7927dff700 1 --2- 172.21.15.2:0/670381405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7920100290 0x7f79201006a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f791c009990 tx=0x7f791c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:42.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.390+0000 7f7927dff700 1 -- 172.21.15.2:0/670381405 shutdown_connections 2023-11-16T16:47:42.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.390+0000 7f7927dff700 1 --2- 172.21.15.2:0/670381405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7920100290 0x7f79201006a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:42.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.390+0000 7f7927dff700 1 --2- 172.21.15.2:0/670381405 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7920101e10 0x7f79201086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:42.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.390+0000 7f7927dff700 1 --2- 172.21.15.2:0/670381405 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7920101480 0x7f79201018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:42.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.390+0000 7f7927dff700 1 -- 172.21.15.2:0/670381405 >> 172.21.15.2:0/670381405 conn(0x7f79200fb870 msgr2=0x7f79200fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:42.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.390+0000 7f7927dff700 1 -- 172.21.15.2:0/670381405 shutdown_connections 2023-11-16T16:47:42.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.390+0000 7f7927dff700 1 -- 172.21.15.2:0/670381405 wait complete. 2023-11-16T16:47:42.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.391+0000 7f7927dff700 1 Processor -- start 2023-11-16T16:47:42.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.391+0000 7f7927dff700 1 -- start start 2023-11-16T16:47:42.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.392+0000 7f7927dff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7920100290 0x7f79200748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:42.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.392+0000 7f7927dff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7920101480 0x7f7920072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:42.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.392+0000 7f7927dff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7920101e10 0x7f7920073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:42.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.392+0000 7f7927dff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7920073bc0 con 0x7f7920100290 2023-11-16T16:47:42.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.392+0000 7f7927dff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7920073d30 con 0x7f7920101e10 2023-11-16T16:47:42.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.392+0000 7f7927dff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792019ffa0 con 0x7f7920101480 2023-11-16T16:47:42.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.392+0000 7f792539a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7920101480 0x7f7920072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:42.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.392+0000 7f7925b9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7920100290 0x7f79200748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:42.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.392+0000 7f792539a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7920101480 0x7f7920072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:41178/0 (socket says 172.21.15.2:41178) 2023-11-16T16:47:42.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.392+0000 7f792539a700 1 -- 172.21.15.2:0/15834297 learned_addr learned my addr 172.21.15.2:0/15834297 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:42.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.392+0000 7f792639c700 1 --2- 172.21.15.2:0/15834297 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7920101e10 0x7f7920073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:42.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.392+0000 7f792539a700 1 -- 172.21.15.2:0/15834297 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7920101e10 msgr2=0x7f7920073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:42.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.392+0000 7f792539a700 1 --2- 172.21.15.2:0/15834297 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7920101e10 0x7f7920073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:42.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.392+0000 7f792539a700 1 -- 172.21.15.2:0/15834297 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7920100290 msgr2=0x7f79200748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:42.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.392+0000 7f792539a700 1 --2- 172.21.15.2:0/15834297 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7920100290 0x7f79200748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:42.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.392+0000 7f792539a700 1 -- 172.21.15.2:0/15834297 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791c0095f0 con 0x7f7920101480 2023-11-16T16:47:42.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.392+0000 7f792639c700 1 --2- 172.21.15.2:0/15834297 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7920101e10 0x7f7920073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:42.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.393+0000 7f792539a700 1 --2- 172.21.15.2:0/15834297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7920101480 0x7f7920072ee0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f791000e9a0 tx=0x7f791000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:42.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.393+0000 7f7916ffd700 1 -- 172.21.15.2:0/15834297 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7910009790 con 0x7f7920101480 2023-11-16T16:47:42.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.393+0000 7f7916ffd700 1 -- 172.21.15.2:0/15834297 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f791000cbd0 con 0x7f7920101480 2023-11-16T16:47:42.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.393+0000 7f7927dff700 1 -- 172.21.15.2:0/15834297 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f792010bbe0 con 0x7f7920101480 2023-11-16T16:47:42.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.393+0000 7f7916ffd700 1 -- 172.21.15.2:0/15834297 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7910018680 con 0x7f7920101480 2023-11-16T16:47:42.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.393+0000 7f7927dff700 1 -- 172.21.15.2:0/15834297 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f792010ba50 con 0x7f7920101480 2023-11-16T16:47:42.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.394+0000 7f7927dff700 1 -- 172.21.15.2:0/15834297 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7920107820 con 0x7f7920101480 2023-11-16T16:47:42.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.396+0000 7f7916ffd700 1 -- 172.21.15.2:0/15834297 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f791000cd40 con 0x7f7920101480 2023-11-16T16:47:42.401 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.397+0000 7f7916ffd700 1 --2- 172.21.15.2:0/15834297 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f790c0a1150 0x7f790c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:42.401 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.398+0000 7f7916ffd700 1 -- 172.21.15.2:0/15834297 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7910014070 con 0x7f7920101480 2023-11-16T16:47:42.401 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.398+0000 7f7925b9b700 1 --2- 172.21.15.2:0/15834297 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f790c0a1150 0x7f790c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:42.401 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.398+0000 7f7925b9b700 1 --2- 172.21.15.2:0/15834297 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f790c0a1150 0x7f790c0a3600 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f791c00b2e0 tx=0x7f791c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:42.402 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.400+0000 7f7916ffd700 1 -- 172.21.15.2:0/15834297 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f79100963b0 con 0x7f7920101480 2023-11-16T16:47:42.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.607+0000 7f7927dff700 1 -- 172.21.15.2:0/15834297 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f792002cfe0 con 0x7f7920101480 2023-11-16T16:47:42.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.610+0000 7f7916ffd700 1 -- 172.21.15.2:0/15834297 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23912 (secure 0 0 0) 0x7f79100999d0 con 0x7f7920101480 2023-11-16T16:47:42.612 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:42.612 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":89,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:47:31.327295+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:24.458452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:25.786005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:29.313181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:47:42.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.612+0000 7f7927dff700 1 -- 172.21.15.2:0/15834297 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f790c0a1150 msgr2=0x7f790c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:42.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.612+0000 7f7927dff700 1 --2- 172.21.15.2:0/15834297 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f790c0a1150 0x7f790c0a3600 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f791c00b2e0 tx=0x7f791c005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:42.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.612+0000 7f7927dff700 1 -- 172.21.15.2:0/15834297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7920101480 msgr2=0x7f7920072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:42.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.612+0000 7f7927dff700 1 --2- 172.21.15.2:0/15834297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7920101480 0x7f7920072ee0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f791000e9a0 tx=0x7f791000ecb0 comp rx=0 tx=0).stop 2023-11-16T16:47:42.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.612+0000 7f7927dff700 1 -- 172.21.15.2:0/15834297 shutdown_connections 2023-11-16T16:47:42.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.612+0000 7f7927dff700 1 --2- 172.21.15.2:0/15834297 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f790c0a1150 0x7f790c0a3600 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:42.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.612+0000 7f7927dff700 1 --2- 172.21.15.2:0/15834297 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7920101e10 0x7f7920073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:42.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.612+0000 7f7927dff700 1 --2- 172.21.15.2:0/15834297 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7920100290 0x7f79200748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:42.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.612+0000 7f7927dff700 1 --2- 172.21.15.2:0/15834297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7920101480 0x7f7920072ee0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:42.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.612+0000 7f7927dff700 1 -- 172.21.15.2:0/15834297 >> 172.21.15.2:0/15834297 conn(0x7f79200fb870 msgr2=0x7f7920104960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:42.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.612+0000 7f7927dff700 1 -- 172.21.15.2:0/15834297 shutdown_connections 2023-11-16T16:47:42.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.612+0000 7f7927dff700 1 -- 172.21.15.2:0/15834297 wait complete. 2023-11-16T16:47:42.633 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:04.306584+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T16:47:24.458452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T16:47:25.786005+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T16:47:29.313181+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T16:47:42.633 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:47:42.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.839+0000 7f820c1ff700 1 -- 172.21.15.2:0/2820624573 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82041014c0 msgr2=0x7f8204101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:42.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.839+0000 7f820c1ff700 1 --2- 172.21.15.2:0/2820624573 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82041014c0 0x7f8204101910 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f81f4009990 tx=0x7f81f4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:42.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.839+0000 7f820c1ff700 1 -- 172.21.15.2:0/2820624573 shutdown_connections 2023-11-16T16:47:42.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.839+0000 7f820c1ff700 1 --2- 172.21.15.2:0/2820624573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8204101e50 0x7f8204108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:42.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.839+0000 7f820c1ff700 1 --2- 172.21.15.2:0/2820624573 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f82041002d0 0x7f82041006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:42.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.839+0000 7f820c1ff700 1 --2- 172.21.15.2:0/2820624573 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82041014c0 0x7f8204101910 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:42.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.839+0000 7f820c1ff700 1 -- 172.21.15.2:0/2820624573 >> 172.21.15.2:0/2820624573 conn(0x7f82040fb8b0 msgr2=0x7f82040fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:42.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.839+0000 7f820c1ff700 1 -- 172.21.15.2:0/2820624573 shutdown_connections 2023-11-16T16:47:42.843 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.839+0000 7f820c1ff700 1 -- 172.21.15.2:0/2820624573 wait complete. 2023-11-16T16:47:42.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.840+0000 7f820c1ff700 1 Processor -- start 2023-11-16T16:47:42.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.841+0000 7f820c1ff700 1 -- start start 2023-11-16T16:47:42.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.841+0000 7f820c1ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f82041002d0 0x7f82040748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:42.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.841+0000 7f820c1ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82041014c0 0x7f8204072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:42.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.841+0000 7f820c1ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8204101e50 0x7f8204073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:42.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.841+0000 7f820c1ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8204073bc0 con 0x7f8204101e50 2023-11-16T16:47:42.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.841+0000 7f820c1ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8204073d30 con 0x7f82041002d0 2023-11-16T16:47:42.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.841+0000 7f820c1ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82041a00e0 con 0x7f82041014c0 2023-11-16T16:47:42.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.841+0000 7f820a79c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8204101e50 0x7f8204073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:42.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.841+0000 7f820979a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82041014c0 0x7f8204072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:42.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.841+0000 7f820979a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82041014c0 0x7f8204072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:41188/0 (socket says 172.21.15.2:41188) 2023-11-16T16:47:42.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.841+0000 7f820a79c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8204101e50 0x7f8204073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48432/0 (socket says 172.21.15.2:48432) 2023-11-16T16:47:42.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.841+0000 7f820979a700 1 -- 172.21.15.2:0/4211718978 learned_addr learned my addr 172.21.15.2:0/4211718978 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:42.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.841+0000 7f8209f9b700 1 --2- 172.21.15.2:0/4211718978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f82041002d0 0x7f82040748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:42.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.842+0000 7f820979a700 1 -- 172.21.15.2:0/4211718978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f82041002d0 msgr2=0x7f82040748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:42.847 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.842+0000 7f820979a700 1 --2- 172.21.15.2:0/4211718978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f82041002d0 0x7f82040748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:42.847 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.842+0000 7f820979a700 1 -- 172.21.15.2:0/4211718978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8204101e50 msgr2=0x7f8204073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:42.847 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.842+0000 7f820979a700 1 --2- 172.21.15.2:0/4211718978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8204101e50 0x7f8204073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:42.847 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.842+0000 7f820979a700 1 -- 172.21.15.2:0/4211718978 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f40095f0 con 0x7f82041014c0 2023-11-16T16:47:42.847 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.842+0000 7f820a79c700 1 --2- 172.21.15.2:0/4211718978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8204101e50 0x7f8204073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:42.847 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.842+0000 7f820979a700 1 --2- 172.21.15.2:0/4211718978 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82041014c0 0x7f8204072ee0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f81f400b620 tx=0x7f81f4005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:42.848 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.846+0000 7f81faffd700 1 -- 172.21.15.2:0/4211718978 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f401d070 con 0x7f82041014c0 2023-11-16T16:47:42.848 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.846+0000 7f81faffd700 1 -- 172.21.15.2:0/4211718978 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f81f4022470 con 0x7f82041014c0 2023-11-16T16:47:42.848 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.846+0000 7f81faffd700 1 -- 172.21.15.2:0/4211718978 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f4017650 con 0x7f82041014c0 2023-11-16T16:47:42.849 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.846+0000 7f820c1ff700 1 -- 172.21.15.2:0/4211718978 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f820410bc80 con 0x7f82041014c0 2023-11-16T16:47:42.849 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.847+0000 7f820c1ff700 1 -- 172.21.15.2:0/4211718978 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820410baf0 con 0x7f82041014c0 2023-11-16T16:47:42.849 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.848+0000 7f820c1ff700 1 -- 172.21.15.2:0/4211718978 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82041078b0 con 0x7f82041014c0 2023-11-16T16:47:42.851 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.849+0000 7f81faffd700 1 -- 172.21.15.2:0/4211718978 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f81f40225e0 con 0x7f82041014c0 2023-11-16T16:47:42.852 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.852+0000 7f81faffd700 1 --2- 172.21.15.2:0/4211718978 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f81f00a11d0 0x7f81f00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:42.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.852+0000 7f81faffd700 1 -- 172.21.15.2:0/4211718978 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f81f40ce070 con 0x7f82041014c0 2023-11-16T16:47:42.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.852+0000 7f8209f9b700 1 --2- 172.21.15.2:0/4211718978 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f81f00a11d0 0x7f81f00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:42.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.852+0000 7f8209f9b700 1 --2- 172.21.15.2:0/4211718978 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f81f00a11d0 0x7f81f00a3680 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8204101320 tx=0x7f8200009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:42.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:42.855+0000 7f81faffd700 1 -- 172.21.15.2:0/4211718978 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f81f4097600 con 0x7f82041014c0 2023-11-16T16:47:42.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:42 smithi002 ceph-mon[128712]: pgmap v288: 129 pgs: 129 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T16:47:42.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:42 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/15834297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:43.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.065+0000 7f820c1ff700 1 -- 172.21.15.2:0/4211718978 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8204061700 con 0x7f82041014c0 2023-11-16T16:47:43.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.066+0000 7f81faffd700 1 -- 172.21.15.2:0/4211718978 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f81f4011110 con 0x7f82041014c0 2023-11-16T16:47:43.068 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T16:47:43.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.069+0000 7f820c1ff700 1 -- 172.21.15.2:0/4211718978 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f81f00a11d0 msgr2=0x7f81f00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:43.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.069+0000 7f820c1ff700 1 --2- 172.21.15.2:0/4211718978 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f81f00a11d0 0x7f81f00a3680 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8204101320 tx=0x7f8200009260 comp rx=0 tx=0).stop 2023-11-16T16:47:43.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.069+0000 7f820c1ff700 1 -- 172.21.15.2:0/4211718978 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82041014c0 msgr2=0x7f8204072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:43.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.069+0000 7f820c1ff700 1 --2- 172.21.15.2:0/4211718978 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82041014c0 0x7f8204072ee0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f81f400b620 tx=0x7f81f4005740 comp rx=0 tx=0).stop 2023-11-16T16:47:43.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.069+0000 7f820c1ff700 1 -- 172.21.15.2:0/4211718978 shutdown_connections 2023-11-16T16:47:43.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.069+0000 7f820c1ff700 1 --2- 172.21.15.2:0/4211718978 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f81f00a11d0 0x7f81f00a3680 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.069+0000 7f820c1ff700 1 --2- 172.21.15.2:0/4211718978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8204101e50 0x7f8204073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.069+0000 7f820c1ff700 1 --2- 172.21.15.2:0/4211718978 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f82041002d0 0x7f82040748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.069+0000 7f820c1ff700 1 --2- 172.21.15.2:0/4211718978 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82041014c0 0x7f8204072ee0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.069+0000 7f820c1ff700 1 -- 172.21.15.2:0/4211718978 >> 172.21.15.2:0/4211718978 conn(0x7f82040fb8b0 msgr2=0x7f82041049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:43.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.069+0000 7f820c1ff700 1 -- 172.21.15.2:0/4211718978 shutdown_connections 2023-11-16T16:47:43.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.069+0000 7f820c1ff700 1 -- 172.21.15.2:0/4211718978 wait complete. 2023-11-16T16:47:43.090 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T16:47:43.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:42 smithi155 ceph-mon[131607]: pgmap v288: 129 pgs: 129 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T16:47:43.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:42 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/15834297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:43.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:42 smithi141 ceph-mon[131184]: pgmap v288: 129 pgs: 129 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T16:47:43.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:42 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/15834297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:43.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.294+0000 7ff0cd5ff700 1 -- 172.21.15.2:0/2244775911 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0c81002c0 msgr2=0x7ff0c81006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:43.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.294+0000 7ff0cd5ff700 1 --2- 172.21.15.2:0/2244775911 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0c81002c0 0x7ff0c81006d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8009990 tx=0x7ff0b8009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.294+0000 7ff0cd5ff700 1 -- 172.21.15.2:0/2244775911 shutdown_connections 2023-11-16T16:47:43.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.294+0000 7ff0cd5ff700 1 --2- 172.21.15.2:0/2244775911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0c8101e40 0x7ff0c8108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.294+0000 7ff0cd5ff700 1 --2- 172.21.15.2:0/2244775911 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0c81002c0 0x7ff0c81006d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.294+0000 7ff0cd5ff700 1 --2- 172.21.15.2:0/2244775911 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0c81014b0 0x7ff0c8101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.294+0000 7ff0cd5ff700 1 -- 172.21.15.2:0/2244775911 >> 172.21.15.2:0/2244775911 conn(0x7ff0c80fb8a0 msgr2=0x7ff0c80fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:43.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.294+0000 7ff0cd5ff700 1 -- 172.21.15.2:0/2244775911 shutdown_connections 2023-11-16T16:47:43.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.294+0000 7ff0cd5ff700 1 -- 172.21.15.2:0/2244775911 wait complete. 2023-11-16T16:47:43.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.295+0000 7ff0cd5ff700 1 Processor -- start 2023-11-16T16:47:43.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.295+0000 7ff0cd5ff700 1 -- start start 2023-11-16T16:47:43.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.296+0000 7ff0cd5ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0c81002c0 0x7ff0c8196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:43.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.296+0000 7ff0cd5ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0c81014b0 0x7ff0c81965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:43.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.296+0000 7ff0cd5ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0c8101e40 0x7ff0c819d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:43.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.296+0000 7ff0cd5ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c819dc10 con 0x7ff0c81002c0 2023-11-16T16:47:43.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.296+0000 7ff0cd5ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c819dd50 con 0x7ff0c8101e40 2023-11-16T16:47:43.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.296+0000 7ff0cd5ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c819dee0 con 0x7ff0c81014b0 2023-11-16T16:47:43.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.296+0000 7ff0c6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0c81002c0 0x7ff0c8196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:43.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.296+0000 7ff0c6ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0c81002c0 0x7ff0c8196090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48456/0 (socket says 172.21.15.2:48456) 2023-11-16T16:47:43.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.296+0000 7ff0c6ffd700 1 -- 172.21.15.2:0/424271866 learned_addr learned my addr 172.21.15.2:0/424271866 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:43.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.296+0000 7ff0c77fe700 1 --2- 172.21.15.2:0/424271866 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0c8101e40 0x7ff0c819d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:43.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.296+0000 7ff0c67fc700 1 --2- 172.21.15.2:0/424271866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0c81014b0 0x7ff0c81965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:43.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.296+0000 7ff0c6ffd700 1 -- 172.21.15.2:0/424271866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0c81014b0 msgr2=0x7ff0c81965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:43.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.296+0000 7ff0c6ffd700 1 --2- 172.21.15.2:0/424271866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0c81014b0 0x7ff0c81965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.296+0000 7ff0c6ffd700 1 -- 172.21.15.2:0/424271866 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0c8101e40 msgr2=0x7ff0c819d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:43.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.296+0000 7ff0c6ffd700 1 --2- 172.21.15.2:0/424271866 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0c8101e40 0x7ff0c819d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.296+0000 7ff0c6ffd700 1 -- 172.21.15.2:0/424271866 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b80095f0 con 0x7ff0c81002c0 2023-11-16T16:47:43.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.297+0000 7ff0c6ffd700 1 --2- 172.21.15.2:0/424271866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0c81002c0 0x7ff0c8196090 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff0b80098e0 tx=0x7ff0b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:43.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.297+0000 7ff0affff700 1 -- 172.21.15.2:0/424271866 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b801d070 con 0x7ff0c81002c0 2023-11-16T16:47:43.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.297+0000 7ff0cd5ff700 1 -- 172.21.15.2:0/424271866 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c810bc90 con 0x7ff0c81002c0 2023-11-16T16:47:43.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.297+0000 7ff0affff700 1 -- 172.21.15.2:0/424271866 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff0b8005030 con 0x7ff0c81002c0 2023-11-16T16:47:43.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.297+0000 7ff0affff700 1 -- 172.21.15.2:0/424271866 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8004b50 con 0x7ff0c81002c0 2023-11-16T16:47:43.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.297+0000 7ff0cd5ff700 1 -- 172.21.15.2:0/424271866 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c810bb00 con 0x7ff0c81002c0 2023-11-16T16:47:43.302 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.298+0000 7ff0cd5ff700 1 -- 172.21.15.2:0/424271866 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c8107850 con 0x7ff0c81002c0 2023-11-16T16:47:43.302 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.300+0000 7ff0affff700 1 -- 172.21.15.2:0/424271866 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7ff0b8004d10 con 0x7ff0c81002c0 2023-11-16T16:47:43.303 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.301+0000 7ff0affff700 1 --2- 172.21.15.2:0/424271866 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff0b00a1150 0x7ff0b00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:43.303 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.302+0000 7ff0affff700 1 -- 172.21.15.2:0/424271866 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff0b80ce230 con 0x7ff0c81002c0 2023-11-16T16:47:43.303 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.302+0000 7ff0c67fc700 1 --2- 172.21.15.2:0/424271866 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff0b00a1150 0x7ff0b00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:43.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.302+0000 7ff0c67fc700 1 --2- 172.21.15.2:0/424271866 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff0b00a1150 0x7ff0b00a3600 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc005950 tx=0x7ff0bc00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:43.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.305+0000 7ff0affff700 1 -- 172.21.15.2:0/424271866 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff0b8097820 con 0x7ff0c81002c0 2023-11-16T16:47:43.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.514+0000 7ff0cd5ff700 1 -- 172.21.15.2:0/424271866 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff0c8067b00 con 0x7ff0c81002c0 2023-11-16T16:47:43.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.515+0000 7ff0affff700 1 -- 172.21.15.2:0/424271866 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff0b809ae40 con 0x7ff0c81002c0 2023-11-16T16:47:43.517 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T16:47:43.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.517+0000 7ff0cd5ff700 1 -- 172.21.15.2:0/424271866 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff0b00a1150 msgr2=0x7ff0b00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:43.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.517+0000 7ff0cd5ff700 1 --2- 172.21.15.2:0/424271866 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff0b00a1150 0x7ff0b00a3600 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc005950 tx=0x7ff0bc00b410 comp rx=0 tx=0).stop 2023-11-16T16:47:43.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.517+0000 7ff0cd5ff700 1 -- 172.21.15.2:0/424271866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0c81002c0 msgr2=0x7ff0c8196090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:43.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.517+0000 7ff0cd5ff700 1 --2- 172.21.15.2:0/424271866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0c81002c0 0x7ff0c8196090 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff0b80098e0 tx=0x7ff0b8004730 comp rx=0 tx=0).stop 2023-11-16T16:47:43.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.517+0000 7ff0cd5ff700 1 -- 172.21.15.2:0/424271866 shutdown_connections 2023-11-16T16:47:43.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.517+0000 7ff0cd5ff700 1 --2- 172.21.15.2:0/424271866 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff0b00a1150 0x7ff0b00a3600 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.517+0000 7ff0cd5ff700 1 --2- 172.21.15.2:0/424271866 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0c8101e40 0x7ff0c819d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.517+0000 7ff0cd5ff700 1 --2- 172.21.15.2:0/424271866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0c81002c0 0x7ff0c8196090 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.517+0000 7ff0cd5ff700 1 --2- 172.21.15.2:0/424271866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0c81014b0 0x7ff0c81965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.517+0000 7ff0cd5ff700 1 -- 172.21.15.2:0/424271866 >> 172.21.15.2:0/424271866 conn(0x7ff0c80fb8a0 msgr2=0x7ff0c8104990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:43.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.517+0000 7ff0cd5ff700 1 -- 172.21.15.2:0/424271866 shutdown_connections 2023-11-16T16:47:43.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.517+0000 7ff0cd5ff700 1 -- 172.21.15.2:0/424271866 wait complete. 2023-11-16T16:47:43.538 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T16:47:43.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.746+0000 7f766a9ff700 1 -- 172.21.15.2:0/117800067 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76641002c0 msgr2=0x7f76641006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:43.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.746+0000 7f766a9ff700 1 --2- 172.21.15.2:0/117800067 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76641002c0 0x7f76641006d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7654009990 tx=0x7f7654009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.747+0000 7f766a9ff700 1 -- 172.21.15.2:0/117800067 shutdown_connections 2023-11-16T16:47:43.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.747+0000 7f766a9ff700 1 --2- 172.21.15.2:0/117800067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7664101e40 0x7f7664108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.747+0000 7f766a9ff700 1 --2- 172.21.15.2:0/117800067 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76641002c0 0x7f76641006d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.747+0000 7f766a9ff700 1 --2- 172.21.15.2:0/117800067 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76641014b0 0x7f7664101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.747+0000 7f766a9ff700 1 -- 172.21.15.2:0/117800067 >> 172.21.15.2:0/117800067 conn(0x7f76640fb8a0 msgr2=0x7f76640fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:43.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.747+0000 7f766a9ff700 1 -- 172.21.15.2:0/117800067 shutdown_connections 2023-11-16T16:47:43.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.747+0000 7f766a9ff700 1 -- 172.21.15.2:0/117800067 wait complete. 2023-11-16T16:47:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.748+0000 7f766a9ff700 1 Processor -- start 2023-11-16T16:47:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.748+0000 7f766a9ff700 1 -- start start 2023-11-16T16:47:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.748+0000 7f766a9ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76641002c0 0x7f76640748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.748+0000 7f766a9ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76641014b0 0x7f7664072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.748+0000 7f766a9ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7664101e40 0x7f7664073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.748+0000 7f766a9ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7664073bc0 con 0x7f7664101e40 2023-11-16T16:47:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.748+0000 7f766a9ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7664073d30 con 0x7f76641014b0 2023-11-16T16:47:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.748+0000 7f766a9ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76641a00f0 con 0x7f76641002c0 2023-11-16T16:47:43.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.749+0000 7f7668f9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7664101e40 0x7f7664073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:43.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.749+0000 7f7663fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76641002c0 0x7f76640748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:43.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.749+0000 7f76637fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76641014b0 0x7f7664072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:43.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.749+0000 7f7668f9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7664101e40 0x7f7664073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48486/0 (socket says 172.21.15.2:48486) 2023-11-16T16:47:43.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.749+0000 7f7663fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76641002c0 0x7f76640748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:41218/0 (socket says 172.21.15.2:41218) 2023-11-16T16:47:43.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.749+0000 7f7668f9c700 1 -- 172.21.15.2:0/2151042014 learned_addr learned my addr 172.21.15.2:0/2151042014 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:43.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.749+0000 7f7663fff700 1 -- 172.21.15.2:0/2151042014 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76641014b0 msgr2=0x7f7664072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:43.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.749+0000 7f7663fff700 1 --2- 172.21.15.2:0/2151042014 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76641014b0 0x7f7664072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.749+0000 7f7663fff700 1 -- 172.21.15.2:0/2151042014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7664101e40 msgr2=0x7f7664073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:43.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.749+0000 7f7663fff700 1 --2- 172.21.15.2:0/2151042014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7664101e40 0x7f7664073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.749+0000 7f7663fff700 1 -- 172.21.15.2:0/2151042014 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7658009520 con 0x7f76641002c0 2023-11-16T16:47:43.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.749+0000 7f7668f9c700 1 --2- 172.21.15.2:0/2151042014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7664101e40 0x7f7664073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:43.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.749+0000 7f7663fff700 1 --2- 172.21.15.2:0/2151042014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76641002c0 0x7f76640748a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f76540098e0 tx=0x7f7654005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:43.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.750+0000 7f76617fa700 1 -- 172.21.15.2:0/2151042014 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765401d070 con 0x7f76641002c0 2023-11-16T16:47:43.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.750+0000 7f766a9ff700 1 -- 172.21.15.2:0/2151042014 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76540095f0 con 0x7f76641002c0 2023-11-16T16:47:43.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.750+0000 7f76617fa700 1 -- 172.21.15.2:0/2151042014 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7654022470 con 0x7f76641002c0 2023-11-16T16:47:43.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.750+0000 7f76617fa700 1 -- 172.21.15.2:0/2151042014 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654022ea0 con 0x7f76641002c0 2023-11-16T16:47:43.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.750+0000 7f766a9ff700 1 -- 172.21.15.2:0/2151042014 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f766410bc90 con 0x7f76641002c0 2023-11-16T16:47:43.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.751+0000 7f766a9ff700 1 -- 172.21.15.2:0/2151042014 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766410bb00 con 0x7f76641002c0 2023-11-16T16:47:43.754 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.753+0000 7f76617fa700 1 -- 172.21.15.2:0/2151042014 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f7654022890 con 0x7f76641002c0 2023-11-16T16:47:43.755 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.754+0000 7f76617fa700 1 --2- 172.21.15.2:0/2151042014 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f764c0a1150 0x7f764c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:43.755 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.755+0000 7f76617fa700 1 -- 172.21.15.2:0/2151042014 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f76540ce230 con 0x7f76641002c0 2023-11-16T16:47:43.755 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.755+0000 7f76637fe700 1 --2- 172.21.15.2:0/2151042014 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f764c0a1150 0x7f764c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:43.756 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.755+0000 7f76637fe700 1 --2- 172.21.15.2:0/2151042014 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f764c0a1150 0x7f764c0a3600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7658009df0 tx=0x7f7658009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:43.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.756+0000 7f76617fa700 1 -- 172.21.15.2:0/2151042014 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f76540977c0 con 0x7f76641002c0 2023-11-16T16:47:43.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.976+0000 7f766a9ff700 1 -- 172.21.15.2:0/2151042014 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f766410be20 con 0x7f76641002c0 2023-11-16T16:47:43.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.977+0000 7f76617fa700 1 -- 172.21.15.2:0/2151042014 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f765409ade0 con 0x7f76641002c0 2023-11-16T16:47:43.978 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2023-11-16T16:47:43.980 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.979+0000 7f766a9ff700 1 -- 172.21.15.2:0/2151042014 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f764c0a1150 msgr2=0x7f764c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:43.980 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.979+0000 7f766a9ff700 1 --2- 172.21.15.2:0/2151042014 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f764c0a1150 0x7f764c0a3600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7658009df0 tx=0x7f7658009260 comp rx=0 tx=0).stop 2023-11-16T16:47:43.980 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.979+0000 7f766a9ff700 1 -- 172.21.15.2:0/2151042014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76641002c0 msgr2=0x7f76640748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:43.980 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.979+0000 7f766a9ff700 1 --2- 172.21.15.2:0/2151042014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76641002c0 0x7f76640748a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f76540098e0 tx=0x7f7654005740 comp rx=0 tx=0).stop 2023-11-16T16:47:43.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.979+0000 7f766a9ff700 1 -- 172.21.15.2:0/2151042014 shutdown_connections 2023-11-16T16:47:43.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.979+0000 7f766a9ff700 1 --2- 172.21.15.2:0/2151042014 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f764c0a1150 0x7f764c0a3600 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.979+0000 7f766a9ff700 1 --2- 172.21.15.2:0/2151042014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76641002c0 0x7f76640748a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.979+0000 7f766a9ff700 1 --2- 172.21.15.2:0/2151042014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7664101e40 0x7f7664073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.979+0000 7f766a9ff700 1 --2- 172.21.15.2:0/2151042014 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f76641014b0 0x7f7664072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:43.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.979+0000 7f766a9ff700 1 -- 172.21.15.2:0/2151042014 >> 172.21.15.2:0/2151042014 conn(0x7f76640fb8a0 msgr2=0x7f7664104990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:43.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.979+0000 7f766a9ff700 1 -- 172.21.15.2:0/2151042014 shutdown_connections 2023-11-16T16:47:43.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:43.980+0000 7f766a9ff700 1 -- 172.21.15.2:0/2151042014 wait complete. 2023-11-16T16:47:43.999 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T16:47:44.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:43 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/4211718978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:47:44.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:43 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/424271866' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:47:44.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2023-11-16T16:47:44.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 8]}]: dispatch 2023-11-16T16:47:44.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2023-11-16T16:47:44.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2023-11-16T16:47:44.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2023-11-16T16:47:44.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2023-11-16T16:47:44.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2023-11-16T16:47:44.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2023-11-16T16:47:44.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2023-11-16T16:47:44.120 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:43 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2023-11-16T16:47:44.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:43 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/4211718978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:47:44.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:43 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/424271866' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:47:44.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2023-11-16T16:47:44.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 8]}]: dispatch 2023-11-16T16:47:44.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2023-11-16T16:47:44.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2023-11-16T16:47:44.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2023-11-16T16:47:44.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2023-11-16T16:47:44.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2023-11-16T16:47:44.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2023-11-16T16:47:44.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2023-11-16T16:47:44.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:43 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2023-11-16T16:47:44.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.205+0000 7fc32a9ff700 1 -- 172.21.15.2:0/2152323229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc324074ad0 msgr2=0x7fc324072f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:44.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.205+0000 7fc32a9ff700 1 --2- 172.21.15.2:0/2152323229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc324074ad0 0x7fc324072f20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc314009990 tx=0x7fc314009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.205+0000 7fc32a9ff700 1 -- 172.21.15.2:0/2152323229 shutdown_connections 2023-11-16T16:47:44.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.205+0000 7fc32a9ff700 1 --2- 172.21.15.2:0/2152323229 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc324108160 0x7fc32410a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.205+0000 7fc32a9ff700 1 --2- 172.21.15.2:0/2152323229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc324074ad0 0x7fc324072f20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.205+0000 7fc32a9ff700 1 --2- 172.21.15.2:0/2152323229 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3240734f0 0x7fc324073960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.205+0000 7fc32a9ff700 1 -- 172.21.15.2:0/2152323229 >> 172.21.15.2:0/2152323229 conn(0x7fc3240fb8b0 msgr2=0x7fc3240fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:44.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.205+0000 7fc32a9ff700 1 -- 172.21.15.2:0/2152323229 shutdown_connections 2023-11-16T16:47:44.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.205+0000 7fc32a9ff700 1 -- 172.21.15.2:0/2152323229 wait complete. 2023-11-16T16:47:44.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.206+0000 7fc32a9ff700 1 Processor -- start 2023-11-16T16:47:44.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.206+0000 7fc32a9ff700 1 -- start start 2023-11-16T16:47:44.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.207+0000 7fc32a9ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3240734f0 0x7fc324198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:44.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.207+0000 7fc32a9ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc324074ad0 0x7fc3241987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:44.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.207+0000 7fc32a9ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc324108160 0x7fc32419f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:44.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.207+0000 7fc32a9ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32419fdf0 con 0x7fc324074ad0 2023-11-16T16:47:44.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.207+0000 7fc32a9ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32419ff30 con 0x7fc324108160 2023-11-16T16:47:44.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.207+0000 7fc32a9ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3241a00c0 con 0x7fc3240734f0 2023-11-16T16:47:44.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.207+0000 7fc3237fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc324074ad0 0x7fc3241987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:44.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.207+0000 7fc3237fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc324074ad0 0x7fc3241987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48498/0 (socket says 172.21.15.2:48498) 2023-11-16T16:47:44.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.207+0000 7fc3237fe700 1 -- 172.21.15.2:0/3542453205 learned_addr learned my addr 172.21.15.2:0/3542453205 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:44.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.207+0000 7fc323fff700 1 --2- 172.21.15.2:0/3542453205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3240734f0 0x7fc324198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:44.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.207+0000 7fc328f9c700 1 --2- 172.21.15.2:0/3542453205 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc324108160 0x7fc32419f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:44.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.207+0000 7fc3237fe700 1 -- 172.21.15.2:0/3542453205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3240734f0 msgr2=0x7fc324198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:44.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.207+0000 7fc3237fe700 1 --2- 172.21.15.2:0/3542453205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3240734f0 0x7fc324198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.207+0000 7fc3237fe700 1 -- 172.21.15.2:0/3542453205 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc324108160 msgr2=0x7fc32419f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:44.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.207+0000 7fc3237fe700 1 --2- 172.21.15.2:0/3542453205 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc324108160 0x7fc32419f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.207+0000 7fc3237fe700 1 -- 172.21.15.2:0/3542453205 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3140095f0 con 0x7fc324074ad0 2023-11-16T16:47:44.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.208+0000 7fc3237fe700 1 --2- 172.21.15.2:0/3542453205 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc324074ad0 0x7fc3241987b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc31800e9a0 tx=0x7fc31800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:44.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.208+0000 7fc3217fa700 1 -- 172.21.15.2:0/3542453205 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318009790 con 0x7fc324074ad0 2023-11-16T16:47:44.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.208+0000 7fc32a9ff700 1 -- 172.21.15.2:0/3542453205 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc32410de60 con 0x7fc324074ad0 2023-11-16T16:47:44.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.208+0000 7fc3217fa700 1 -- 172.21.15.2:0/3542453205 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc318010430 con 0x7fc324074ad0 2023-11-16T16:47:44.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.208+0000 7fc32a9ff700 1 -- 172.21.15.2:0/3542453205 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc32410dcd0 con 0x7fc324074ad0 2023-11-16T16:47:44.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.208+0000 7fc3217fa700 1 -- 172.21.15.2:0/3542453205 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318018620 con 0x7fc324074ad0 2023-11-16T16:47:44.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.209+0000 7fc32a9ff700 1 -- 172.21.15.2:0/3542453205 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3241068a0 con 0x7fc324074ad0 2023-11-16T16:47:44.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.211+0000 7fc3217fa700 1 -- 172.21.15.2:0/3542453205 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fc3180105a0 con 0x7fc324074ad0 2023-11-16T16:47:44.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.212+0000 7fc3217fa700 1 --2- 172.21.15.2:0/3542453205 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc30c0a1150 0x7fc30c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:44.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.212+0000 7fc3217fa700 1 -- 172.21.15.2:0/3542453205 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc318014070 con 0x7fc324074ad0 2023-11-16T16:47:44.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.213+0000 7fc323fff700 1 --2- 172.21.15.2:0/3542453205 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc30c0a1150 0x7fc30c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:44.214 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.213+0000 7fc323fff700 1 --2- 172.21.15.2:0/3542453205 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc30c0a1150 0x7fc30c0a3600 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc31400b2e0 tx=0x7fc314005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:44.217 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.216+0000 7fc3217fa700 1 -- 172.21.15.2:0/3542453205 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc318051730 con 0x7fc324074ad0 2023-11-16T16:47:44.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:43 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/4211718978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:47:44.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:43 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/424271866' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:47:44.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2023-11-16T16:47:44.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 8]}]: dispatch 2023-11-16T16:47:44.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2023-11-16T16:47:44.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2023-11-16T16:47:44.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2023-11-16T16:47:44.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2023-11-16T16:47:44.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2023-11-16T16:47:44.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2023-11-16T16:47:44.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2023-11-16T16:47:44.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:43 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2023-11-16T16:47:44.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.423+0000 7fc32a9ff700 1 -- 172.21.15.2:0/3542453205 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc324061700 con 0x7fc324074ad0 2023-11-16T16:47:44.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.424+0000 7fc3217fa700 1 -- 172.21.15.2:0/3542453205 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc3180961b0 con 0x7fc324074ad0 2023-11-16T16:47:44.425 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T16:47:44.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.426+0000 7fc32a9ff700 1 -- 172.21.15.2:0/3542453205 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc30c0a1150 msgr2=0x7fc30c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:44.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.426+0000 7fc32a9ff700 1 --2- 172.21.15.2:0/3542453205 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc30c0a1150 0x7fc30c0a3600 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc31400b2e0 tx=0x7fc314005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.426+0000 7fc32a9ff700 1 -- 172.21.15.2:0/3542453205 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc324074ad0 msgr2=0x7fc3241987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:44.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.426+0000 7fc32a9ff700 1 --2- 172.21.15.2:0/3542453205 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc324074ad0 0x7fc3241987b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc31800e9a0 tx=0x7fc31800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.426+0000 7fc32a9ff700 1 -- 172.21.15.2:0/3542453205 shutdown_connections 2023-11-16T16:47:44.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.426+0000 7fc32a9ff700 1 --2- 172.21.15.2:0/3542453205 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc30c0a1150 0x7fc30c0a3600 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.426+0000 7fc32a9ff700 1 --2- 172.21.15.2:0/3542453205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3240734f0 0x7fc324198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.426+0000 7fc32a9ff700 1 --2- 172.21.15.2:0/3542453205 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc324108160 0x7fc32419f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.426+0000 7fc32a9ff700 1 --2- 172.21.15.2:0/3542453205 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc324074ad0 0x7fc3241987b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.426+0000 7fc32a9ff700 1 -- 172.21.15.2:0/3542453205 >> 172.21.15.2:0/3542453205 conn(0x7fc3240fb8b0 msgr2=0x7fc3240fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:44.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.426+0000 7fc32a9ff700 1 -- 172.21.15.2:0/3542453205 shutdown_connections 2023-11-16T16:47:44.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.426+0000 7fc32a9ff700 1 -- 172.21.15.2:0/3542453205 wait complete. 2023-11-16T16:47:44.446 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-16T16:47:44.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.651+0000 7f02779ff700 1 -- 172.21.15.2:0/3068394886 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0270101420 msgr2=0x7f0270101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:44.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.651+0000 7f02779ff700 1 --2- 172.21.15.2:0/3068394886 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0270101420 0x7f0270101870 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0260009990 tx=0x7f0260009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.651+0000 7f02779ff700 1 -- 172.21.15.2:0/3068394886 shutdown_connections 2023-11-16T16:47:44.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.651+0000 7f02779ff700 1 --2- 172.21.15.2:0/3068394886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0270100230 0x7f0270100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.651+0000 7f02779ff700 1 --2- 172.21.15.2:0/3068394886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0270101db0 0x7f0270108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.651+0000 7f02779ff700 1 --2- 172.21.15.2:0/3068394886 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0270101420 0x7f0270101870 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.651+0000 7f02779ff700 1 -- 172.21.15.2:0/3068394886 >> 172.21.15.2:0/3068394886 conn(0x7f02700fb810 msgr2=0x7f02700fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:44.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.651+0000 7f02779ff700 1 -- 172.21.15.2:0/3068394886 shutdown_connections 2023-11-16T16:47:44.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.651+0000 7f02779ff700 1 -- 172.21.15.2:0/3068394886 wait complete. 2023-11-16T16:47:44.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.652+0000 7f02779ff700 1 Processor -- start 2023-11-16T16:47:44.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.652+0000 7f02779ff700 1 -- start start 2023-11-16T16:47:44.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.653+0000 7f02779ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0270100230 0x7f02700748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:44.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.653+0000 7f02779ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0270101420 0x7f0270072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:44.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.653+0000 7f02779ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0270101db0 0x7f0270073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:44.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.653+0000 7f02779ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0270073bc0 con 0x7f0270101420 2023-11-16T16:47:44.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.653+0000 7f02779ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0270073d30 con 0x7f0270100230 2023-11-16T16:47:44.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.653+0000 7f02779ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027019ffc0 con 0x7f0270101db0 2023-11-16T16:47:44.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.653+0000 7f0274f9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0270101420 0x7f0270072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:44.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.653+0000 7f0274f9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0270101420 0x7f0270072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33052/0 (socket says 172.21.15.2:33052) 2023-11-16T16:47:44.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.653+0000 7f0274f9a700 1 -- 172.21.15.2:0/1230033459 learned_addr learned my addr 172.21.15.2:0/1230033459 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:44.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.653+0000 7f027579b700 1 --2- 172.21.15.2:0/1230033459 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0270100230 0x7f02700748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:44.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.653+0000 7f0275f9c700 1 --2- 172.21.15.2:0/1230033459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0270101db0 0x7f0270073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:44.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.653+0000 7f0274f9a700 1 -- 172.21.15.2:0/1230033459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0270101db0 msgr2=0x7f0270073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:44.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.653+0000 7f0274f9a700 1 --2- 172.21.15.2:0/1230033459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0270101db0 0x7f0270073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.653+0000 7f0274f9a700 1 -- 172.21.15.2:0/1230033459 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0270100230 msgr2=0x7f02700748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:44.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.653+0000 7f0274f9a700 1 --2- 172.21.15.2:0/1230033459 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0270100230 0x7f02700748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.653+0000 7f0274f9a700 1 -- 172.21.15.2:0/1230033459 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02600095f0 con 0x7f0270101420 2023-11-16T16:47:44.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.654+0000 7f0274f9a700 1 --2- 172.21.15.2:0/1230033459 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0270101420 0x7f0270072ee0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0260005740 tx=0x7f0260005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:44.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.654+0000 7f02667fc700 1 -- 172.21.15.2:0/1230033459 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026001d070 con 0x7f0270101420 2023-11-16T16:47:44.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.654+0000 7f02667fc700 1 -- 172.21.15.2:0/1230033459 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0260021410 con 0x7f0270101420 2023-11-16T16:47:44.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.654+0000 7f02667fc700 1 -- 172.21.15.2:0/1230033459 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0260017650 con 0x7f0270101420 2023-11-16T16:47:44.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.654+0000 7f02779ff700 1 -- 172.21.15.2:0/1230033459 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027010bb90 con 0x7f0270101420 2023-11-16T16:47:44.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.654+0000 7f02779ff700 1 -- 172.21.15.2:0/1230033459 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027010ba00 con 0x7f0270101420 2023-11-16T16:47:44.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.655+0000 7f02779ff700 1 -- 172.21.15.2:0/1230033459 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02701a05a0 con 0x7f0270101420 2023-11-16T16:47:44.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.657+0000 7f02667fc700 1 -- 172.21.15.2:0/1230033459 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f0260021580 con 0x7f0270101420 2023-11-16T16:47:44.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.658+0000 7f02667fc700 1 --2- 172.21.15.2:0/1230033459 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f025c0a1150 0x7f025c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:44.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.658+0000 7f02667fc700 1 -- 172.21.15.2:0/1230033459 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f02600ce600 con 0x7f0270101420 2023-11-16T16:47:44.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.659+0000 7f027579b700 1 --2- 172.21.15.2:0/1230033459 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f025c0a1150 0x7f025c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:44.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.659+0000 7f027579b700 1 --2- 172.21.15.2:0/1230033459 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f025c0a1150 0x7f025c0a3600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0270101280 tx=0x7f026c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:44.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.661+0000 7f02667fc700 1 -- 172.21.15.2:0/1230033459 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0260097a70 con 0x7f0270101420 2023-11-16T16:47:44.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:44 smithi002 ceph-mon[128712]: pgmap v289: 129 pgs: 129 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T16:47:44.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2023-11-16T16:47:44.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 8]}]': finished 2023-11-16T16:47:44.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2023-11-16T16:47:44.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2023-11-16T16:47:44.862 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2023-11-16T16:47:44.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2023-11-16T16:47:44.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2023-11-16T16:47:44.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2023-11-16T16:47:44.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2023-11-16T16:47:44.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2023-11-16T16:47:44.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:44 smithi002 ceph-mon[128712]: osdmap e90: 12 total, 12 up, 12 in 2023-11-16T16:47:44.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:44 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2151042014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:47:44.863 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:44 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3542453205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:47:44.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.891+0000 7f02779ff700 1 -- 172.21.15.2:0/1230033459 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f02701070e0 con 0x7f0270101420 2023-11-16T16:47:44.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.892+0000 7f02667fc700 1 -- 172.21.15.2:0/1230033459 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) v1 ==== 81+0+578 (secure 0 0 0) 0x7f026009b090 con 0x7f0270101420 2023-11-16T16:47:44.893 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/2137541200 2023-11-17T16:40:43.705118+0000 2023-11-16T16:47:44.893 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/2525877070 2023-11-17T16:40:43.705118+0000 2023-11-16T16:47:44.893 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/30571981 2023-11-17T16:40:16.538166+0000 2023-11-16T16:47:44.893 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/3359672074 2023-11-17T16:40:43.705118+0000 2023-11-16T16:47:44.893 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/500255408 2023-11-17T16:40:16.538166+0000 2023-11-16T16:47:44.893 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/981081063 2023-11-17T16:40:43.705118+0000 2023-11-16T16:47:44.893 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/3453602765 2023-11-17T16:40:16.538166+0000 2023-11-16T16:47:44.893 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/3359672074 2023-11-17T16:40:43.705118+0000 2023-11-16T16:47:44.893 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/3453602765 2023-11-17T16:40:16.538166+0000 2023-11-16T16:47:44.893 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3818551797 2023-11-17T16:40:16.538166+0000 2023-11-16T16:47:44.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.894+0000 7f02779ff700 1 -- 172.21.15.2:0/1230033459 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f025c0a1150 msgr2=0x7f025c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:44.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.894+0000 7f02779ff700 1 --2- 172.21.15.2:0/1230033459 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f025c0a1150 0x7f025c0a3600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0270101280 tx=0x7f026c009260 comp rx=0 tx=0).stop 2023-11-16T16:47:44.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.894+0000 7f02779ff700 1 -- 172.21.15.2:0/1230033459 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0270101420 msgr2=0x7f0270072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:44.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.894+0000 7f02779ff700 1 --2- 172.21.15.2:0/1230033459 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0270101420 0x7f0270072ee0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0260005740 tx=0x7f0260005770 comp rx=0 tx=0).stop 2023-11-16T16:47:44.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.894+0000 7f02779ff700 1 -- 172.21.15.2:0/1230033459 shutdown_connections 2023-11-16T16:47:44.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.894+0000 7f02779ff700 1 --2- 172.21.15.2:0/1230033459 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f025c0a1150 0x7f025c0a3600 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.894+0000 7f02779ff700 1 --2- 172.21.15.2:0/1230033459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0270101db0 0x7f0270073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.894+0000 7f02779ff700 1 --2- 172.21.15.2:0/1230033459 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0270100230 0x7f02700748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.894+0000 7f02779ff700 1 --2- 172.21.15.2:0/1230033459 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0270101420 0x7f0270072ee0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:44.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.894+0000 7f02779ff700 1 -- 172.21.15.2:0/1230033459 >> 172.21.15.2:0/1230033459 conn(0x7f02700fb810 msgr2=0x7f02700fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:44.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.894+0000 7f02779ff700 1 -- 172.21.15.2:0/1230033459 shutdown_connections 2023-11-16T16:47:44.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:44.894+0000 7f02779ff700 1 -- 172.21.15.2:0/1230033459 wait complete. 2023-11-16T16:47:44.896 INFO:teuthology.orchestra.run.smithi002.stderr:listed 10 entries 2023-11-16T16:47:44.914 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T16:47:44.914 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-16T16:47:44.958 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:47:45.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:44 smithi155 ceph-mon[131607]: pgmap v289: 129 pgs: 129 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T16:47:45.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2023-11-16T16:47:45.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 8]}]': finished 2023-11-16T16:47:45.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2023-11-16T16:47:45.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2023-11-16T16:47:45.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2023-11-16T16:47:45.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2023-11-16T16:47:45.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2023-11-16T16:47:45.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2023-11-16T16:47:45.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2023-11-16T16:47:45.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2023-11-16T16:47:45.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:44 smithi155 ceph-mon[131607]: osdmap e90: 12 total, 12 up, 12 in 2023-11-16T16:47:45.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:44 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2151042014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:47:45.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:44 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3542453205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:47:45.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.201+0000 7f8ec51ff700 1 -- 172.21.15.2:0/2190854684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ec0101480 msgr2=0x7f8ec01018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:45.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.201+0000 7f8ec51ff700 1 --2- 172.21.15.2:0/2190854684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ec0101480 0x7f8ec01018d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8009990 tx=0x7f8ea8009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.201+0000 7f8ec51ff700 1 -- 172.21.15.2:0/2190854684 shutdown_connections 2023-11-16T16:47:45.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.201+0000 7f8ec51ff700 1 --2- 172.21.15.2:0/2190854684 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8ec0101e10 0x7f8ec01086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.201+0000 7f8ec51ff700 1 --2- 172.21.15.2:0/2190854684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ec0100290 0x7f8ec01006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.201+0000 7f8ec51ff700 1 --2- 172.21.15.2:0/2190854684 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ec0101480 0x7f8ec01018d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.201+0000 7f8ec51ff700 1 -- 172.21.15.2:0/2190854684 >> 172.21.15.2:0/2190854684 conn(0x7f8ec00fb870 msgr2=0x7f8ec00fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:45.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.201+0000 7f8ec51ff700 1 -- 172.21.15.2:0/2190854684 shutdown_connections 2023-11-16T16:47:45.203 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.201+0000 7f8ec51ff700 1 -- 172.21.15.2:0/2190854684 wait complete. 2023-11-16T16:47:45.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.202+0000 7f8ec51ff700 1 Processor -- start 2023-11-16T16:47:45.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.202+0000 7f8ec51ff700 1 -- start start 2023-11-16T16:47:45.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.203+0000 7f8ec51ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8ec0100290 0x7f8ec00748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:45.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.203+0000 7f8ec51ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ec0101480 0x7f8ec0072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:45.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.203+0000 7f8ec51ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ec0101e10 0x7f8ec0073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:45.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.203+0000 7f8ec51ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec0073bc0 con 0x7f8ec0101e10 2023-11-16T16:47:45.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.203+0000 7f8ec51ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec0073d30 con 0x7f8ec0100290 2023-11-16T16:47:45.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.203+0000 7f8ec51ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec019ff70 con 0x7f8ec0101480 2023-11-16T16:47:45.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.203+0000 7f8ebed9d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8ec0100290 0x7f8ec00748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:45.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.203+0000 7f8ebf59e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ec0101e10 0x7f8ec0073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:45.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.203+0000 7f8eb7fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ec0101480 0x7f8ec0072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:45.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.203+0000 7f8ebed9d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8ec0100290 0x7f8ec00748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:42596/0 (socket says 172.21.15.2:42596) 2023-11-16T16:47:45.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.203+0000 7f8ebed9d700 1 -- 172.21.15.2:0/112179987 learned_addr learned my addr 172.21.15.2:0/112179987 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:45.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.203+0000 7f8eb7fff700 1 -- 172.21.15.2:0/112179987 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8ec0100290 msgr2=0x7f8ec00748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:45.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.203+0000 7f8eb7fff700 1 --2- 172.21.15.2:0/112179987 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8ec0100290 0x7f8ec00748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.203+0000 7f8eb7fff700 1 -- 172.21.15.2:0/112179987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ec0101e10 msgr2=0x7f8ec0073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:45.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.203+0000 7f8eb7fff700 1 --2- 172.21.15.2:0/112179987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ec0101e10 0x7f8ec0073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.203+0000 7f8eb7fff700 1 -- 172.21.15.2:0/112179987 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb0009520 con 0x7f8ec0101480 2023-11-16T16:47:45.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.203+0000 7f8ebed9d700 1 --2- 172.21.15.2:0/112179987 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8ec0100290 0x7f8ec00748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:45.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.204+0000 7f8eb7fff700 1 --2- 172.21.15.2:0/112179987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ec0101480 0x7f8ec0072ee0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8ea800b680 tx=0x7f8ea8009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:45.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.204+0000 7f8ebcd99700 1 -- 172.21.15.2:0/112179987 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea801d070 con 0x7f8ec0101480 2023-11-16T16:47:45.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.204+0000 7f8ebcd99700 1 -- 172.21.15.2:0/112179987 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ea8021410 con 0x7f8ec0101480 2023-11-16T16:47:45.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.204+0000 7f8ec51ff700 1 -- 172.21.15.2:0/112179987 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea80095f0 con 0x7f8ec0101480 2023-11-16T16:47:45.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.204+0000 7f8ebcd99700 1 -- 172.21.15.2:0/112179987 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea8017650 con 0x7f8ec0101480 2023-11-16T16:47:45.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.204+0000 7f8ec51ff700 1 -- 172.21.15.2:0/112179987 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec010bbf0 con 0x7f8ec0101480 2023-11-16T16:47:45.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.205+0000 7f8ec51ff700 1 -- 172.21.15.2:0/112179987 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec010bd80 con 0x7f8ec0101480 2023-11-16T16:47:45.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.207+0000 7f8ebcd99700 1 -- 172.21.15.2:0/112179987 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f8ea8021810 con 0x7f8ec0101480 2023-11-16T16:47:45.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.208+0000 7f8ebcd99700 1 --2- 172.21.15.2:0/112179987 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8eac0a1150 0x7f8eac0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:45.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.209+0000 7f8ebcd99700 1 -- 172.21.15.2:0/112179987 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8ea80ce320 con 0x7f8ec0101480 2023-11-16T16:47:45.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.209+0000 7f8ebed9d700 1 --2- 172.21.15.2:0/112179987 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8eac0a1150 0x7f8eac0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:45.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.209+0000 7f8ebed9d700 1 --2- 172.21.15.2:0/112179987 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8eac0a1150 0x7f8eac0a3600 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0009fd0 tx=0x7f8eb0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:45.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.210+0000 7f8ebcd99700 1 -- 172.21.15.2:0/112179987 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8ea8097790 con 0x7f8ec0101480 2023-11-16T16:47:45.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:44 smithi141 ceph-mon[131184]: pgmap v289: 129 pgs: 129 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-11-16T16:47:45.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2023-11-16T16:47:45.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 8]}]': finished 2023-11-16T16:47:45.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2023-11-16T16:47:45.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2023-11-16T16:47:45.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2023-11-16T16:47:45.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2023-11-16T16:47:45.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2023-11-16T16:47:45.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2023-11-16T16:47:45.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2023-11-16T16:47:45.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2023-11-16T16:47:45.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:44 smithi141 ceph-mon[131184]: osdmap e90: 12 total, 12 up, 12 in 2023-11-16T16:47:45.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:44 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2151042014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:47:45.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:44 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3542453205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:47:45.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.420+0000 7f8ec51ff700 1 -- 172.21.15.2:0/112179987 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ec002cfe0 con 0x7f8ec0101480 2023-11-16T16:47:45.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.422+0000 7f8ebcd99700 1 -- 172.21.15.2:0/112179987 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24379 (secure 0 0 0) 0x7f8ea809adb0 con 0x7f8ec0101480 2023-11-16T16:47:45.423 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:45.423 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":91,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:47:44.849645+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:24.458452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:25.786005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:29.313181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:47:45.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.424+0000 7f8ec51ff700 1 -- 172.21.15.2:0/112179987 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8eac0a1150 msgr2=0x7f8eac0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:45.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.424+0000 7f8ec51ff700 1 --2- 172.21.15.2:0/112179987 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8eac0a1150 0x7f8eac0a3600 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0009fd0 tx=0x7f8eb0009260 comp rx=0 tx=0).stop 2023-11-16T16:47:45.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.424+0000 7f8ec51ff700 1 -- 172.21.15.2:0/112179987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ec0101480 msgr2=0x7f8ec0072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:45.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.424+0000 7f8ec51ff700 1 --2- 172.21.15.2:0/112179987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ec0101480 0x7f8ec0072ee0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8ea800b680 tx=0x7f8ea8009580 comp rx=0 tx=0).stop 2023-11-16T16:47:45.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.424+0000 7f8ec51ff700 1 -- 172.21.15.2:0/112179987 shutdown_connections 2023-11-16T16:47:45.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.425+0000 7f8ec51ff700 1 --2- 172.21.15.2:0/112179987 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8eac0a1150 0x7f8eac0a3600 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.425+0000 7f8ec51ff700 1 --2- 172.21.15.2:0/112179987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ec0101e10 0x7f8ec0073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.425+0000 7f8ec51ff700 1 --2- 172.21.15.2:0/112179987 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8ec0100290 0x7f8ec00748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.425+0000 7f8ec51ff700 1 --2- 172.21.15.2:0/112179987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ec0101480 0x7f8ec0072ee0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.425+0000 7f8ec51ff700 1 -- 172.21.15.2:0/112179987 >> 172.21.15.2:0/112179987 conn(0x7f8ec00fb870 msgr2=0x7f8ec0104990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:45.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.425+0000 7f8ec51ff700 1 -- 172.21.15.2:0/112179987 shutdown_connections 2023-11-16T16:47:45.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.425+0000 7f8ec51ff700 1 -- 172.21.15.2:0/112179987 wait complete. 2023-11-16T16:47:45.444 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:04.306584+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T16:47:24.458452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T16:47:25.786005+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T16:47:29.313181+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T16:47:45.445 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:47:45.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.654+0000 7f982cbff700 1 -- 172.21.15.2:0/1152265976 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98281002c0 msgr2=0x7f98281006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:45.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.654+0000 7f982cbff700 1 --2- 172.21.15.2:0/1152265976 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98281002c0 0x7f98281006d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9818009990 tx=0x7f9818009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.654+0000 7f982cbff700 1 -- 172.21.15.2:0/1152265976 shutdown_connections 2023-11-16T16:47:45.655 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.654+0000 7f982cbff700 1 --2- 172.21.15.2:0/1152265976 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9828101e40 0x7f9828108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.654+0000 7f982cbff700 1 --2- 172.21.15.2:0/1152265976 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98281002c0 0x7f98281006d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.654+0000 7f982cbff700 1 --2- 172.21.15.2:0/1152265976 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98281014b0 0x7f9828101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.654+0000 7f982cbff700 1 -- 172.21.15.2:0/1152265976 >> 172.21.15.2:0/1152265976 conn(0x7f98280fb8a0 msgr2=0x7f98280fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:45.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.654+0000 7f982cbff700 1 -- 172.21.15.2:0/1152265976 shutdown_connections 2023-11-16T16:47:45.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.654+0000 7f982cbff700 1 -- 172.21.15.2:0/1152265976 wait complete. 2023-11-16T16:47:45.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.655+0000 7f982cbff700 1 Processor -- start 2023-11-16T16:47:45.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.656+0000 7f982cbff700 1 -- start start 2023-11-16T16:47:45.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.656+0000 7f982cbff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98281002c0 0x7f9828198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:45.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.656+0000 7f982cbff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98281014b0 0x7f9828198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:45.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.656+0000 7f982cbff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9828101e40 0x7f982819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:45.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.656+0000 7f982cbff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982819fdb0 con 0x7f98281014b0 2023-11-16T16:47:45.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.656+0000 7f982cbff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982819fef0 con 0x7f98281002c0 2023-11-16T16:47:45.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.656+0000 7f982cbff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98281a0080 con 0x7f9828101e40 2023-11-16T16:47:45.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.656+0000 7f981ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98281014b0 0x7f9828198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:45.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.656+0000 7f982659c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98281002c0 0x7f9828198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:45.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.656+0000 7f981ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98281014b0 0x7f9828198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33084/0 (socket says 172.21.15.2:33084) 2023-11-16T16:47:45.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.656+0000 7f9826d9d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9828101e40 0x7f982819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:45.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.656+0000 7f981ffff700 1 -- 172.21.15.2:0/3304212358 learned_addr learned my addr 172.21.15.2:0/3304212358 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:45.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.657+0000 7f981ffff700 1 -- 172.21.15.2:0/3304212358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9828101e40 msgr2=0x7f982819f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:45.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.657+0000 7f981ffff700 1 --2- 172.21.15.2:0/3304212358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9828101e40 0x7f982819f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.657+0000 7f981ffff700 1 -- 172.21.15.2:0/3304212358 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98281002c0 msgr2=0x7f9828198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:45.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.657+0000 7f981ffff700 1 --2- 172.21.15.2:0/3304212358 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98281002c0 0x7f9828198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.657+0000 7f981ffff700 1 -- 172.21.15.2:0/3304212358 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9810009520 con 0x7f98281014b0 2023-11-16T16:47:45.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.657+0000 7f981ffff700 1 --2- 172.21.15.2:0/3304212358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98281014b0 0x7f9828198770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f981000eb30 tx=0x7f981000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:45.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.657+0000 7f981f7fe700 1 -- 172.21.15.2:0/3304212358 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810009880 con 0x7f98281014b0 2023-11-16T16:47:45.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.657+0000 7f982cbff700 1 -- 172.21.15.2:0/3304212358 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98180095f0 con 0x7f98281014b0 2023-11-16T16:47:45.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.657+0000 7f981f7fe700 1 -- 172.21.15.2:0/3304212358 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98100044e0 con 0x7f98281014b0 2023-11-16T16:47:45.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.657+0000 7f981f7fe700 1 -- 172.21.15.2:0/3304212358 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810003d80 con 0x7f98281014b0 2023-11-16T16:47:45.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.657+0000 7f982cbff700 1 -- 172.21.15.2:0/3304212358 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982810bc20 con 0x7f98281014b0 2023-11-16T16:47:45.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.659+0000 7f982cbff700 1 -- 172.21.15.2:0/3304212358 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9828192980 con 0x7f98281014b0 2023-11-16T16:47:45.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.661+0000 7f981f7fe700 1 -- 172.21.15.2:0/3304212358 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f9810003f20 con 0x7f98281014b0 2023-11-16T16:47:45.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.662+0000 7f981f7fe700 1 --2- 172.21.15.2:0/3304212358 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f98140a1150 0x7f98140a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:45.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.662+0000 7f981f7fe700 1 -- 172.21.15.2:0/3304212358 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9810014070 con 0x7f98281014b0 2023-11-16T16:47:45.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.662+0000 7f982659c700 1 --2- 172.21.15.2:0/3304212358 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f98140a1150 0x7f98140a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:45.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.663+0000 7f982659c700 1 --2- 172.21.15.2:0/3304212358 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f98140a1150 0x7f98140a3600 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9818006010 tx=0x7f9818005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:45.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.666+0000 7f981f7fe700 1 -- 172.21.15.2:0/3304212358 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9810096590 con 0x7f98281014b0 2023-11-16T16:47:45.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.881+0000 7f982cbff700 1 -- 172.21.15.2:0/3304212358 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9828067b00 con 0x7f98281014b0 2023-11-16T16:47:45.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.881+0000 7f981f7fe700 1 -- 172.21.15.2:0/3304212358 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9810094e60 con 0x7f98281014b0 2023-11-16T16:47:45.882 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T16:47:45.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.883+0000 7f982cbff700 1 -- 172.21.15.2:0/3304212358 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f98140a1150 msgr2=0x7f98140a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:45.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.883+0000 7f982cbff700 1 --2- 172.21.15.2:0/3304212358 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f98140a1150 0x7f98140a3600 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9818006010 tx=0x7f9818005960 comp rx=0 tx=0).stop 2023-11-16T16:47:45.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.883+0000 7f982cbff700 1 -- 172.21.15.2:0/3304212358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98281014b0 msgr2=0x7f9828198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:45.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.883+0000 7f982cbff700 1 --2- 172.21.15.2:0/3304212358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98281014b0 0x7f9828198770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f981000eb30 tx=0x7f981000ee40 comp rx=0 tx=0).stop 2023-11-16T16:47:45.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.883+0000 7f982cbff700 1 -- 172.21.15.2:0/3304212358 shutdown_connections 2023-11-16T16:47:45.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.883+0000 7f982cbff700 1 --2- 172.21.15.2:0/3304212358 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f98140a1150 0x7f98140a3600 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.883+0000 7f982cbff700 1 --2- 172.21.15.2:0/3304212358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9828101e40 0x7f982819f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.883+0000 7f982cbff700 1 --2- 172.21.15.2:0/3304212358 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98281002c0 0x7f9828198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.883+0000 7f982cbff700 1 --2- 172.21.15.2:0/3304212358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98281014b0 0x7f9828198770 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:45.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.883+0000 7f982cbff700 1 -- 172.21.15.2:0/3304212358 >> 172.21.15.2:0/3304212358 conn(0x7f98280fb8a0 msgr2=0x7f9828106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:45.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.884+0000 7f982cbff700 1 -- 172.21.15.2:0/3304212358 shutdown_connections 2023-11-16T16:47:45.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:45.884+0000 7f982cbff700 1 -- 172.21.15.2:0/3304212358 wait complete. 2023-11-16T16:47:45.902 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T16:47:46.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.108+0000 7fc28e1ff700 1 -- 172.21.15.2:0/57765425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2881014b0 msgr2=0x7fc288101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.108+0000 7fc28e1ff700 1 --2- 172.21.15.2:0/57765425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2881014b0 0x7fc288101900 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc27c009990 tx=0x7fc27c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.108+0000 7fc28e1ff700 1 -- 172.21.15.2:0/57765425 shutdown_connections 2023-11-16T16:47:46.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.108+0000 7fc28e1ff700 1 --2- 172.21.15.2:0/57765425 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2881002c0 0x7fc2881006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.108+0000 7fc28e1ff700 1 --2- 172.21.15.2:0/57765425 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc288101e40 0x7fc288108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.108+0000 7fc28e1ff700 1 --2- 172.21.15.2:0/57765425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2881014b0 0x7fc288101900 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.108+0000 7fc28e1ff700 1 -- 172.21.15.2:0/57765425 >> 172.21.15.2:0/57765425 conn(0x7fc2880fb8a0 msgr2=0x7fc2880fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:46.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.108+0000 7fc28e1ff700 1 -- 172.21.15.2:0/57765425 shutdown_connections 2023-11-16T16:47:46.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.108+0000 7fc28e1ff700 1 -- 172.21.15.2:0/57765425 wait complete. 2023-11-16T16:47:46.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.109+0000 7fc28e1ff700 1 Processor -- start 2023-11-16T16:47:46.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.109+0000 7fc28e1ff700 1 -- start start 2023-11-16T16:47:46.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.110+0000 7fc28e1ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2881002c0 0x7fc288198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:46.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.110+0000 7fc28e1ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2881014b0 0x7fc288198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:46.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.110+0000 7fc28e1ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc288101e40 0x7fc28819f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:46.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.110+0000 7fc28e1ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28819fdd0 con 0x7fc288101e40 2023-11-16T16:47:46.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.110+0000 7fc28e1ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28819ff10 con 0x7fc2881002c0 2023-11-16T16:47:46.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.110+0000 7fc28e1ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2881a00a0 con 0x7fc2881014b0 2023-11-16T16:47:46.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.110+0000 7fc287fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc288101e40 0x7fc28819f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:46.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.110+0000 7fc2877fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2881002c0 0x7fc288198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:46.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.110+0000 7fc287fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc288101e40 0x7fc28819f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33102/0 (socket says 172.21.15.2:33102) 2023-11-16T16:47:46.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.110+0000 7fc287fff700 1 -- 172.21.15.2:0/1746446887 learned_addr learned my addr 172.21.15.2:0/1746446887 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:46.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.110+0000 7fc286ffd700 1 --2- 172.21.15.2:0/1746446887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2881014b0 0x7fc288198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:46.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.110+0000 7fc287fff700 1 -- 172.21.15.2:0/1746446887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2881014b0 msgr2=0x7fc288198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.110+0000 7fc287fff700 1 --2- 172.21.15.2:0/1746446887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2881014b0 0x7fc288198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.110+0000 7fc287fff700 1 -- 172.21.15.2:0/1746446887 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2881002c0 msgr2=0x7fc288198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.110+0000 7fc287fff700 1 --2- 172.21.15.2:0/1746446887 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2881002c0 0x7fc288198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.110+0000 7fc287fff700 1 -- 172.21.15.2:0/1746446887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc27c0095f0 con 0x7fc288101e40 2023-11-16T16:47:46.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.111+0000 7fc286ffd700 1 --2- 172.21.15.2:0/1746446887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2881014b0 0x7fc288198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:46.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.111+0000 7fc2877fe700 1 --2- 172.21.15.2:0/1746446887 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2881002c0 0x7fc288198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:46.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.111+0000 7fc287fff700 1 --2- 172.21.15.2:0/1746446887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc288101e40 0x7fc28819f800 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc28000ef30 tx=0x7fc28000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:46.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.111+0000 7fc284ff9700 1 -- 172.21.15.2:0/1746446887 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc280018070 con 0x7fc288101e40 2023-11-16T16:47:46.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.111+0000 7fc284ff9700 1 -- 172.21.15.2:0/1746446887 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc28000f040 con 0x7fc288101e40 2023-11-16T16:47:46.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.111+0000 7fc28e1ff700 1 -- 172.21.15.2:0/1746446887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28810bc70 con 0x7fc288101e40 2023-11-16T16:47:46.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.111+0000 7fc284ff9700 1 -- 172.21.15.2:0/1746446887 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28000cdb0 con 0x7fc288101e40 2023-11-16T16:47:46.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.111+0000 7fc28e1ff700 1 -- 172.21.15.2:0/1746446887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28810bae0 con 0x7fc288101e40 2023-11-16T16:47:46.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.112+0000 7fc28e1ff700 1 -- 172.21.15.2:0/1746446887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2881929d0 con 0x7fc288101e40 2023-11-16T16:47:46.116 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.114+0000 7fc284ff9700 1 -- 172.21.15.2:0/1746446887 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fc280003a00 con 0x7fc288101e40 2023-11-16T16:47:46.117 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.116+0000 7fc284ff9700 1 --2- 172.21.15.2:0/1746446887 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2700a11d0 0x7fc2700a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:46.117 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.117+0000 7fc284ff9700 1 -- 172.21.15.2:0/1746446887 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc2800cc7b0 con 0x7fc288101e40 2023-11-16T16:47:46.117 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.117+0000 7fc2877fe700 1 --2- 172.21.15.2:0/1746446887 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2700a11d0 0x7fc2700a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:46.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:45 smithi155 ceph-mon[131607]: osdmap e91: 12 total, 12 up, 12 in 2023-11-16T16:47:46.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:45 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1230033459' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T16:47:46.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:45 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/112179987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:46.118 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.117+0000 7fc2877fe700 1 --2- 172.21.15.2:0/1746446887 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2700a11d0 0x7fc2700a3680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc288101310 tx=0x7fc27800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:46.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.120+0000 7fc284ff9700 1 -- 172.21.15.2:0/1746446887 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc280095c20 con 0x7fc288101e40 2023-11-16T16:47:46.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:45 smithi002 ceph-mon[128712]: osdmap e91: 12 total, 12 up, 12 in 2023-11-16T16:47:46.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:45 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1230033459' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T16:47:46.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:45 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/112179987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:46.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:45 smithi141 ceph-mon[131184]: osdmap e91: 12 total, 12 up, 12 in 2023-11-16T16:47:46.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:45 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1230033459' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T16:47:46.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:45 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/112179987' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:46.333 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.333+0000 7fc28e1ff700 1 -- 172.21.15.2:0/1746446887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc288061700 con 0x7fc288101e40 2023-11-16T16:47:46.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.333+0000 7fc284ff9700 1 -- 172.21.15.2:0/1746446887 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc280099240 con 0x7fc288101e40 2023-11-16T16:47:46.334 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T16:47:46.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.335+0000 7fc28e1ff700 1 -- 172.21.15.2:0/1746446887 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2700a11d0 msgr2=0x7fc2700a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.335+0000 7fc28e1ff700 1 --2- 172.21.15.2:0/1746446887 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2700a11d0 0x7fc2700a3680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc288101310 tx=0x7fc27800b410 comp rx=0 tx=0).stop 2023-11-16T16:47:46.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.335+0000 7fc28e1ff700 1 -- 172.21.15.2:0/1746446887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc288101e40 msgr2=0x7fc28819f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.335+0000 7fc28e1ff700 1 --2- 172.21.15.2:0/1746446887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc288101e40 0x7fc28819f800 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc28000ef30 tx=0x7fc28000c3a0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.336+0000 7fc28e1ff700 1 -- 172.21.15.2:0/1746446887 shutdown_connections 2023-11-16T16:47:46.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.336+0000 7fc28e1ff700 1 --2- 172.21.15.2:0/1746446887 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc2700a11d0 0x7fc2700a3680 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.336+0000 7fc28e1ff700 1 --2- 172.21.15.2:0/1746446887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc288101e40 0x7fc28819f800 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.336+0000 7fc28e1ff700 1 --2- 172.21.15.2:0/1746446887 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc2881002c0 0x7fc288198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.336+0000 7fc28e1ff700 1 --2- 172.21.15.2:0/1746446887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2881014b0 0x7fc288198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.336+0000 7fc28e1ff700 1 -- 172.21.15.2:0/1746446887 >> 172.21.15.2:0/1746446887 conn(0x7fc2880fb8a0 msgr2=0x7fc288106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:46.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.336+0000 7fc28e1ff700 1 -- 172.21.15.2:0/1746446887 shutdown_connections 2023-11-16T16:47:46.337 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.336+0000 7fc28e1ff700 1 -- 172.21.15.2:0/1746446887 wait complete. 2023-11-16T16:47:46.355 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T16:47:46.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.560+0000 7f6f41dff700 1 -- 172.21.15.2:0/3666149140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c1014b0 msgr2=0x7f6f3c101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.560+0000 7f6f41dff700 1 --2- 172.21.15.2:0/3666149140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c1014b0 0x7f6f3c101900 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6f30009990 tx=0x7f6f30009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.560+0000 7f6f41dff700 1 -- 172.21.15.2:0/3666149140 shutdown_connections 2023-11-16T16:47:46.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.560+0000 7f6f41dff700 1 --2- 172.21.15.2:0/3666149140 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f3c1002c0 0x7f6f3c1006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.560+0000 7f6f41dff700 1 --2- 172.21.15.2:0/3666149140 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f3c101e40 0x7f6f3c108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.560+0000 7f6f41dff700 1 --2- 172.21.15.2:0/3666149140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c1014b0 0x7f6f3c101900 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.560+0000 7f6f41dff700 1 -- 172.21.15.2:0/3666149140 >> 172.21.15.2:0/3666149140 conn(0x7f6f3c0fb8a0 msgr2=0x7f6f3c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:46.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.560+0000 7f6f41dff700 1 -- 172.21.15.2:0/3666149140 shutdown_connections 2023-11-16T16:47:46.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.561+0000 7f6f41dff700 1 -- 172.21.15.2:0/3666149140 wait complete. 2023-11-16T16:47:46.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.561+0000 7f6f41dff700 1 Processor -- start 2023-11-16T16:47:46.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.562+0000 7f6f41dff700 1 -- start start 2023-11-16T16:47:46.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.562+0000 7f6f41dff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f3c1002c0 0x7f6f3c0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:46.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.562+0000 7f6f41dff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f3c1014b0 0x7f6f3c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:46.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.562+0000 7f6f41dff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c101e40 0x7f6f3c073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:46.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.562+0000 7f6f41dff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3c073bc0 con 0x7f6f3c101e40 2023-11-16T16:47:46.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.562+0000 7f6f41dff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3c073d30 con 0x7f6f3c1002c0 2023-11-16T16:47:46.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.562+0000 7f6f41dff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3c1a00e0 con 0x7f6f3c1014b0 2023-11-16T16:47:46.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.562+0000 7f6f3bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c101e40 0x7f6f3c073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:46.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.562+0000 7f6f3bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c101e40 0x7f6f3c073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33112/0 (socket says 172.21.15.2:33112) 2023-11-16T16:47:46.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.562+0000 7f6f3bfff700 1 -- 172.21.15.2:0/3888371657 learned_addr learned my addr 172.21.15.2:0/3888371657 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:46.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.562+0000 7f6f3b7fe700 1 --2- 172.21.15.2:0/3888371657 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f3c1002c0 0x7f6f3c0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:46.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.562+0000 7f6f3affd700 1 --2- 172.21.15.2:0/3888371657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f3c1014b0 0x7f6f3c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:46.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.563+0000 7f6f3bfff700 1 -- 172.21.15.2:0/3888371657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f3c1014b0 msgr2=0x7f6f3c072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.563+0000 7f6f3bfff700 1 --2- 172.21.15.2:0/3888371657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f3c1014b0 0x7f6f3c072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.563+0000 7f6f3bfff700 1 -- 172.21.15.2:0/3888371657 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f3c1002c0 msgr2=0x7f6f3c0748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.563+0000 7f6f3bfff700 1 --2- 172.21.15.2:0/3888371657 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f3c1002c0 0x7f6f3c0748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.563+0000 7f6f3bfff700 1 -- 172.21.15.2:0/3888371657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f300095f0 con 0x7f6f3c101e40 2023-11-16T16:47:46.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.563+0000 7f6f3bfff700 1 --2- 172.21.15.2:0/3888371657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c101e40 0x7f6f3c073420 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400ef30 tx=0x7f6f3400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:46.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.563+0000 7f6f38ff9700 1 -- 172.21.15.2:0/3888371657 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f34018070 con 0x7f6f3c101e40 2023-11-16T16:47:46.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.563+0000 7f6f38ff9700 1 -- 172.21.15.2:0/3888371657 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f3400f040 con 0x7f6f3c101e40 2023-11-16T16:47:46.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.563+0000 7f6f38ff9700 1 -- 172.21.15.2:0/3888371657 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3400cd00 con 0x7f6f3c101e40 2023-11-16T16:47:46.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.563+0000 7f6f41dff700 1 -- 172.21.15.2:0/3888371657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f3c10bc80 con 0x7f6f3c101e40 2023-11-16T16:47:46.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.563+0000 7f6f41dff700 1 -- 172.21.15.2:0/3888371657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f3c10baf0 con 0x7f6f3c101e40 2023-11-16T16:47:46.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.564+0000 7f6f41dff700 1 -- 172.21.15.2:0/3888371657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f3c107850 con 0x7f6f3c101e40 2023-11-16T16:47:46.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.566+0000 7f6f38ff9700 1 -- 172.21.15.2:0/3888371657 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f6f34003a00 con 0x7f6f3c101e40 2023-11-16T16:47:46.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.568+0000 7f6f38ff9700 1 --2- 172.21.15.2:0/3888371657 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6f240a1150 0x7f6f240a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:46.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.568+0000 7f6f38ff9700 1 -- 172.21.15.2:0/3888371657 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6f340cc7a0 con 0x7f6f3c101e40 2023-11-16T16:47:46.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.568+0000 7f6f3b7fe700 1 --2- 172.21.15.2:0/3888371657 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6f240a1150 0x7f6f240a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:46.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.569+0000 7f6f3b7fe700 1 --2- 172.21.15.2:0/3888371657 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6f240a1150 0x7f6f240a3600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c101310 tx=0x7f6f2c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:46.572 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.571+0000 7f6f38ff9700 1 -- 172.21.15.2:0/3888371657 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6f34095c10 con 0x7f6f3c101e40 2023-11-16T16:47:46.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.779+0000 7f6f41dff700 1 -- 172.21.15.2:0/3888371657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6f3c061700 con 0x7f6f3c101e40 2023-11-16T16:47:46.780 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.779+0000 7f6f38ff9700 1 -- 172.21.15.2:0/3888371657 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6f34099230 con 0x7f6f3c101e40 2023-11-16T16:47:46.780 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2023-11-16T16:47:46.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.782+0000 7f6f41dff700 1 -- 172.21.15.2:0/3888371657 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6f240a1150 msgr2=0x7f6f240a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.782+0000 7f6f41dff700 1 --2- 172.21.15.2:0/3888371657 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6f240a1150 0x7f6f240a3600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c101310 tx=0x7f6f2c009260 comp rx=0 tx=0).stop 2023-11-16T16:47:46.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.782+0000 7f6f41dff700 1 -- 172.21.15.2:0/3888371657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c101e40 msgr2=0x7f6f3c073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:46.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.782+0000 7f6f41dff700 1 --2- 172.21.15.2:0/3888371657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c101e40 0x7f6f3c073420 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400ef30 tx=0x7f6f3400c3a0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.782+0000 7f6f41dff700 1 -- 172.21.15.2:0/3888371657 shutdown_connections 2023-11-16T16:47:46.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.782+0000 7f6f41dff700 1 --2- 172.21.15.2:0/3888371657 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6f240a1150 0x7f6f240a3600 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.782+0000 7f6f41dff700 1 --2- 172.21.15.2:0/3888371657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c101e40 0x7f6f3c073420 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.782+0000 7f6f41dff700 1 --2- 172.21.15.2:0/3888371657 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f3c1002c0 0x7f6f3c0748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.782+0000 7f6f41dff700 1 --2- 172.21.15.2:0/3888371657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f3c1014b0 0x7f6f3c072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:46.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.782+0000 7f6f41dff700 1 -- 172.21.15.2:0/3888371657 >> 172.21.15.2:0/3888371657 conn(0x7f6f3c0fb8a0 msgr2=0x7f6f3c104990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:46.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.782+0000 7f6f41dff700 1 -- 172.21.15.2:0/3888371657 shutdown_connections 2023-11-16T16:47:46.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:46.782+0000 7f6f41dff700 1 -- 172.21.15.2:0/3888371657 wait complete. 2023-11-16T16:47:46.801 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T16:47:47.011 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.010+0000 7f6a2e3ff700 1 -- 172.21.15.2:0/3917398692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a28100d40 msgr2=0x7f6a28103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:47.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.010+0000 7f6a2e3ff700 1 --2- 172.21.15.2:0/3917398692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a28100d40 0x7f6a28103120 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6a18009990 tx=0x7f6a18009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.011+0000 7f6a2e3ff700 1 -- 172.21.15.2:0/3917398692 shutdown_connections 2023-11-16T16:47:47.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.011+0000 7f6a2e3ff700 1 --2- 172.21.15.2:0/3917398692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a28105f80 0x7f6a281083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.011+0000 7f6a2e3ff700 1 --2- 172.21.15.2:0/3917398692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a28100d40 0x7f6a28103120 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.011+0000 7f6a2e3ff700 1 --2- 172.21.15.2:0/3917398692 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6a28103660 0x7f6a28105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.011+0000 7f6a2e3ff700 1 -- 172.21.15.2:0/3917398692 >> 172.21.15.2:0/3917398692 conn(0x7f6a280fa7c0 msgr2=0x7f6a280fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:47.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.011+0000 7f6a2e3ff700 1 -- 172.21.15.2:0/3917398692 shutdown_connections 2023-11-16T16:47:47.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.011+0000 7f6a2e3ff700 1 -- 172.21.15.2:0/3917398692 wait complete. 2023-11-16T16:47:47.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.012+0000 7f6a2e3ff700 1 Processor -- start 2023-11-16T16:47:47.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.012+0000 7f6a2e3ff700 1 -- start start 2023-11-16T16:47:47.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.013+0000 7f6a2e3ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a28100d40 0x7f6a281960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:47.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.013+0000 7f6a2e3ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6a28103660 0x7f6a281965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:47.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.013+0000 7f6a2e3ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a28105f80 0x7f6a2819d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:47.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.013+0000 7f6a2e3ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2819dc30 con 0x7f6a28100d40 2023-11-16T16:47:47.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.013+0000 7f6a2e3ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2819dd70 con 0x7f6a28105f80 2023-11-16T16:47:47.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.013+0000 7f6a2e3ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2819df00 con 0x7f6a28103660 2023-11-16T16:47:47.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.013+0000 7f6a27fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a28100d40 0x7f6a281960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:47.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.013+0000 7f6a27fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a28100d40 0x7f6a281960b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33132/0 (socket says 172.21.15.2:33132) 2023-11-16T16:47:47.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.013+0000 7f6a27fff700 1 -- 172.21.15.2:0/1961405683 learned_addr learned my addr 172.21.15.2:0/1961405683 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:47.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.013+0000 7f6a277fe700 1 --2- 172.21.15.2:0/1961405683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6a28103660 0x7f6a281965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:47.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.013+0000 7f6a2c99c700 1 --2- 172.21.15.2:0/1961405683 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a28105f80 0x7f6a2819d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:47.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.014+0000 7f6a27fff700 1 -- 172.21.15.2:0/1961405683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6a28103660 msgr2=0x7f6a281965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:47.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.014+0000 7f6a27fff700 1 --2- 172.21.15.2:0/1961405683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6a28103660 0x7f6a281965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.014+0000 7f6a27fff700 1 -- 172.21.15.2:0/1961405683 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a28105f80 msgr2=0x7f6a2819d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:47.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.014+0000 7f6a27fff700 1 --2- 172.21.15.2:0/1961405683 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a28105f80 0x7f6a2819d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.014+0000 7f6a27fff700 1 -- 172.21.15.2:0/1961405683 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a180095f0 con 0x7f6a28100d40 2023-11-16T16:47:47.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.014+0000 7f6a27fff700 1 --2- 172.21.15.2:0/1961405683 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a28100d40 0x7f6a281960b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6a180036c0 tx=0x7f6a1800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:47.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.014+0000 7f6a257fa700 1 -- 172.21.15.2:0/1961405683 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1801d070 con 0x7f6a28100d40 2023-11-16T16:47:47.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.014+0000 7f6a257fa700 1 -- 172.21.15.2:0/1961405683 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a18022470 con 0x7f6a28100d40 2023-11-16T16:47:47.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.014+0000 7f6a2e3ff700 1 -- 172.21.15.2:0/1961405683 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2810bc70 con 0x7f6a28100d40 2023-11-16T16:47:47.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.014+0000 7f6a257fa700 1 -- 172.21.15.2:0/1961405683 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a18017650 con 0x7f6a28100d40 2023-11-16T16:47:47.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.014+0000 7f6a2e3ff700 1 -- 172.21.15.2:0/1961405683 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a2810bae0 con 0x7f6a28100d40 2023-11-16T16:47:47.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.015+0000 7f6a2e3ff700 1 -- 172.21.15.2:0/1961405683 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a280fc330 con 0x7f6a28100d40 2023-11-16T16:47:47.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.017+0000 7f6a257fa700 1 -- 172.21.15.2:0/1961405683 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f6a18022890 con 0x7f6a28100d40 2023-11-16T16:47:47.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.019+0000 7f6a257fa700 1 --2- 172.21.15.2:0/1961405683 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6a100a11d0 0x7f6a100a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:47.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.019+0000 7f6a257fa700 1 -- 172.21.15.2:0/1961405683 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6a180cdeb0 con 0x7f6a28100d40 2023-11-16T16:47:47.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.019+0000 7f6a277fe700 1 --2- 172.21.15.2:0/1961405683 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6a100a11d0 0x7f6a100a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:47.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.019+0000 7f6a277fe700 1 --2- 172.21.15.2:0/1961405683 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6a100a11d0 0x7f6a100a3680 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c007c60 tx=0x7f6a1c0073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:47.023 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.022+0000 7f6a257fa700 1 -- 172.21.15.2:0/1961405683 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6a18097320 con 0x7f6a28100d40 2023-11-16T16:47:47.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:46 smithi002 ceph-mon[128712]: pgmap v292: 129 pgs: 1 peering, 128 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2023-11-16T16:47:47.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:46 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3304212358' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:47:47.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:46 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1746446887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:47:47.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:46 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3888371657' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:47:47.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.232+0000 7f6a2e3ff700 1 -- 172.21.15.2:0/1961405683 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f6a28061700 con 0x7f6a28100d40 2023-11-16T16:47:47.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.232+0000 7f6a257fa700 1 -- 172.21.15.2:0/1961405683 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6a18027020 con 0x7f6a28100d40 2023-11-16T16:47:47.233 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T16:47:47.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.235+0000 7f6a2e3ff700 1 -- 172.21.15.2:0/1961405683 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6a100a11d0 msgr2=0x7f6a100a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:47.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.235+0000 7f6a2e3ff700 1 --2- 172.21.15.2:0/1961405683 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6a100a11d0 0x7f6a100a3680 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c007c60 tx=0x7f6a1c0073d0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.235+0000 7f6a2e3ff700 1 -- 172.21.15.2:0/1961405683 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a28100d40 msgr2=0x7f6a281960b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:47.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.235+0000 7f6a2e3ff700 1 --2- 172.21.15.2:0/1961405683 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a28100d40 0x7f6a281960b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6a180036c0 tx=0x7f6a1800f690 comp rx=0 tx=0).stop 2023-11-16T16:47:47.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.235+0000 7f6a2e3ff700 1 -- 172.21.15.2:0/1961405683 shutdown_connections 2023-11-16T16:47:47.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.235+0000 7f6a2e3ff700 1 --2- 172.21.15.2:0/1961405683 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6a100a11d0 0x7f6a100a3680 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.235+0000 7f6a2e3ff700 1 --2- 172.21.15.2:0/1961405683 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6a28105f80 0x7f6a2819d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.235+0000 7f6a2e3ff700 1 --2- 172.21.15.2:0/1961405683 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a28100d40 0x7f6a281960b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.235+0000 7f6a2e3ff700 1 --2- 172.21.15.2:0/1961405683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6a28103660 0x7f6a281965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.235+0000 7f6a2e3ff700 1 -- 172.21.15.2:0/1961405683 >> 172.21.15.2:0/1961405683 conn(0x7f6a280fa7c0 msgr2=0x7f6a281019a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:47.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.235+0000 7f6a2e3ff700 1 -- 172.21.15.2:0/1961405683 shutdown_connections 2023-11-16T16:47:47.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.235+0000 7f6a2e3ff700 1 -- 172.21.15.2:0/1961405683 wait complete. 2023-11-16T16:47:47.254 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-16T16:47:47.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:46 smithi141 ceph-mon[131184]: pgmap v292: 129 pgs: 1 peering, 128 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2023-11-16T16:47:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:46 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3304212358' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:47:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:46 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1746446887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:47:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:46 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3888371657' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:47:47.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:46 smithi155 ceph-mon[131607]: pgmap v292: 129 pgs: 1 peering, 128 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2023-11-16T16:47:47.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:46 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3304212358' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:47:47.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:46 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1746446887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:47:47.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:46 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3888371657' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:47:47.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.458+0000 7f067ffff700 1 -- 172.21.15.2:0/1172446245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0678100230 msgr2=0x7f0678100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:47.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.458+0000 7f067ffff700 1 --2- 172.21.15.2:0/1172446245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0678100230 0x7f0678100640 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0674009990 tx=0x7f0674009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.458+0000 7f067ffff700 1 -- 172.21.15.2:0/1172446245 shutdown_connections 2023-11-16T16:47:47.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.458+0000 7f067ffff700 1 --2- 172.21.15.2:0/1172446245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0678101db0 0x7f0678108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.458+0000 7f067ffff700 1 --2- 172.21.15.2:0/1172446245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0678100230 0x7f0678100640 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.458+0000 7f067ffff700 1 --2- 172.21.15.2:0/1172446245 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0678101420 0x7f0678101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.458+0000 7f067ffff700 1 -- 172.21.15.2:0/1172446245 >> 172.21.15.2:0/1172446245 conn(0x7f06780fb810 msgr2=0x7f06780fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:47.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.458+0000 7f067ffff700 1 -- 172.21.15.2:0/1172446245 shutdown_connections 2023-11-16T16:47:47.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.458+0000 7f067ffff700 1 -- 172.21.15.2:0/1172446245 wait complete. 2023-11-16T16:47:47.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.459+0000 7f067ffff700 1 Processor -- start 2023-11-16T16:47:47.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.459+0000 7f067ffff700 1 -- start start 2023-11-16T16:47:47.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.459+0000 7f067ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0678100230 0x7f0678195fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:47.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.459+0000 7f067ffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0678101420 0x7f0678196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:47.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.459+0000 7f067ffff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0678101db0 0x7f067819d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:47.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.459+0000 7f067ffff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067819db40 con 0x7f0678100230 2023-11-16T16:47:47.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.459+0000 7f067ffff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067819dc80 con 0x7f0678101db0 2023-11-16T16:47:47.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.459+0000 7f067ffff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067819de10 con 0x7f0678101420 2023-11-16T16:47:47.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.460+0000 7f067dd9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0678100230 0x7f0678195fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:47.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.460+0000 7f067dd9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0678100230 0x7f0678195fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33150/0 (socket says 172.21.15.2:33150) 2023-11-16T16:47:47.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.460+0000 7f067dd9b700 1 -- 172.21.15.2:0/2789764090 learned_addr learned my addr 172.21.15.2:0/2789764090 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:47.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.460+0000 7f067e59c700 1 --2- 172.21.15.2:0/2789764090 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0678101db0 0x7f067819d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:47.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.460+0000 7f067d59a700 1 --2- 172.21.15.2:0/2789764090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0678101420 0x7f0678196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:47.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.460+0000 7f067dd9b700 1 -- 172.21.15.2:0/2789764090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0678101420 msgr2=0x7f0678196500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:47.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.460+0000 7f067dd9b700 1 --2- 172.21.15.2:0/2789764090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0678101420 0x7f0678196500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.460+0000 7f067dd9b700 1 -- 172.21.15.2:0/2789764090 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0678101db0 msgr2=0x7f067819d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:47.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.460+0000 7f067dd9b700 1 --2- 172.21.15.2:0/2789764090 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0678101db0 0x7f067819d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.460+0000 7f067dd9b700 1 -- 172.21.15.2:0/2789764090 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06740095f0 con 0x7f0678100230 2023-11-16T16:47:47.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.460+0000 7f067dd9b700 1 --2- 172.21.15.2:0/2789764090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0678100230 0x7f0678195fc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0674004760 tx=0x7f0674009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:47.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.461+0000 7f066effd700 1 -- 172.21.15.2:0/2789764090 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067401d070 con 0x7f0678100230 2023-11-16T16:47:47.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.461+0000 7f066effd700 1 -- 172.21.15.2:0/2789764090 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0674004cd0 con 0x7f0678100230 2023-11-16T16:47:47.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.461+0000 7f067ffff700 1 -- 172.21.15.2:0/2789764090 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f067810bbe0 con 0x7f0678100230 2023-11-16T16:47:47.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.461+0000 7f066effd700 1 -- 172.21.15.2:0/2789764090 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0674005290 con 0x7f0678100230 2023-11-16T16:47:47.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.461+0000 7f067ffff700 1 -- 172.21.15.2:0/2789764090 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f067810ba50 con 0x7f0678100230 2023-11-16T16:47:47.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.462+0000 7f067ffff700 1 -- 172.21.15.2:0/2789764090 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0678107160 con 0x7f0678100230 2023-11-16T16:47:47.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.464+0000 7f066effd700 1 -- 172.21.15.2:0/2789764090 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f0674003b60 con 0x7f0678100230 2023-11-16T16:47:47.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.465+0000 7f066effd700 1 --2- 172.21.15.2:0/2789764090 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f06640a11d0 0x7f06640a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:47.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.465+0000 7f066effd700 1 -- 172.21.15.2:0/2789764090 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f06740cdfb0 con 0x7f0678100230 2023-11-16T16:47:47.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.465+0000 7f067d59a700 1 --2- 172.21.15.2:0/2789764090 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f06640a11d0 0x7f06640a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:47.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.466+0000 7f067d59a700 1 --2- 172.21.15.2:0/2789764090 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f06640a11d0 0x7f06640a3680 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f066800bbd0 tx=0x7f066800b5b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:47.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.467+0000 7f066effd700 1 -- 172.21.15.2:0/2789764090 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0674097420 con 0x7f0678100230 2023-11-16T16:47:47.693 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.693+0000 7f067ffff700 1 -- 172.21.15.2:0/2789764090 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f067802cfe0 con 0x7f0678100230 2023-11-16T16:47:47.694 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.693+0000 7f066effd700 1 -- 172.21.15.2:0/2789764090 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) v1 ==== 81+0+578 (secure 0 0 0) 0x7f0674026080 con 0x7f0678100230 2023-11-16T16:47:47.694 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/2137541200 2023-11-17T16:40:43.705118+0000 2023-11-16T16:47:47.694 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/2525877070 2023-11-17T16:40:43.705118+0000 2023-11-16T16:47:47.694 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/30571981 2023-11-17T16:40:16.538166+0000 2023-11-16T16:47:47.694 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/3359672074 2023-11-17T16:40:43.705118+0000 2023-11-16T16:47:47.695 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/500255408 2023-11-17T16:40:16.538166+0000 2023-11-16T16:47:47.695 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/981081063 2023-11-17T16:40:43.705118+0000 2023-11-16T16:47:47.695 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/3453602765 2023-11-17T16:40:16.538166+0000 2023-11-16T16:47:47.695 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/3359672074 2023-11-17T16:40:43.705118+0000 2023-11-16T16:47:47.695 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/3453602765 2023-11-17T16:40:16.538166+0000 2023-11-16T16:47:47.695 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3818551797 2023-11-17T16:40:16.538166+0000 2023-11-16T16:47:47.696 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.696+0000 7f067ffff700 1 -- 172.21.15.2:0/2789764090 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f06640a11d0 msgr2=0x7f06640a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:47.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.696+0000 7f067ffff700 1 --2- 172.21.15.2:0/2789764090 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f06640a11d0 0x7f06640a3680 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f066800bbd0 tx=0x7f066800b5b0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.696+0000 7f067ffff700 1 -- 172.21.15.2:0/2789764090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0678100230 msgr2=0x7f0678195fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:47.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.696+0000 7f067ffff700 1 --2- 172.21.15.2:0/2789764090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0678100230 0x7f0678195fc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0674004760 tx=0x7f0674009580 comp rx=0 tx=0).stop 2023-11-16T16:47:47.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.696+0000 7f067ffff700 1 -- 172.21.15.2:0/2789764090 shutdown_connections 2023-11-16T16:47:47.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.696+0000 7f067ffff700 1 --2- 172.21.15.2:0/2789764090 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f06640a11d0 0x7f06640a3680 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.696+0000 7f067ffff700 1 --2- 172.21.15.2:0/2789764090 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0678101db0 0x7f067819d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.696+0000 7f067ffff700 1 --2- 172.21.15.2:0/2789764090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0678100230 0x7f0678195fc0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.696+0000 7f067ffff700 1 --2- 172.21.15.2:0/2789764090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0678101420 0x7f0678196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:47.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.696+0000 7f067ffff700 1 -- 172.21.15.2:0/2789764090 >> 172.21.15.2:0/2789764090 conn(0x7f06780fb810 msgr2=0x7f06780fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:47.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.696+0000 7f067ffff700 1 -- 172.21.15.2:0/2789764090 shutdown_connections 2023-11-16T16:47:47.697 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:47.696+0000 7f067ffff700 1 -- 172.21.15.2:0/2789764090 wait complete. 2023-11-16T16:47:47.698 INFO:teuthology.orchestra.run.smithi002.stderr:listed 10 entries 2023-11-16T16:47:47.715 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi002.front.sepia.ceph.com... 2023-11-16T16:47:47.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:47.715 DEBUG:teuthology.orchestra.run.smithi002:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-11-16T16:47:47.782 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi002.stderr:ceph-fuse[165445]: fuse finished with error 0 and tester_r 0 2023-11-16T16:47:47.787 INFO:teuthology.orchestra.run:waiting for 300 2023-11-16T16:47:47.885 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T16:47:47.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:47:47.885 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-11-16T16:47:47.934 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:47:47.994 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:47 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1961405683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:47:47.994 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:47 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2789764090' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T16:47:48.138 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.137+0000 7f0a5d5ff700 1 -- 172.21.15.2:0/501391938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a58100290 msgr2=0x7f0a581006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:48.138 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.137+0000 7f0a5d5ff700 1 --2- 172.21.15.2:0/501391938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a58100290 0x7f0a581006a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0a48009990 tx=0x7f0a48009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.138 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.137+0000 7f0a5d5ff700 1 -- 172.21.15.2:0/501391938 shutdown_connections 2023-11-16T16:47:48.138 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.137+0000 7f0a5d5ff700 1 --2- 172.21.15.2:0/501391938 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a58101e10 0x7f0a581086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.138 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.137+0000 7f0a5d5ff700 1 --2- 172.21.15.2:0/501391938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a58100290 0x7f0a581006a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.138 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.137+0000 7f0a5d5ff700 1 --2- 172.21.15.2:0/501391938 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a58101480 0x7f0a581018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.138 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.137+0000 7f0a5d5ff700 1 -- 172.21.15.2:0/501391938 >> 172.21.15.2:0/501391938 conn(0x7f0a580fb870 msgr2=0x7f0a580fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:48.138 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.137+0000 7f0a5d5ff700 1 -- 172.21.15.2:0/501391938 shutdown_connections 2023-11-16T16:47:48.139 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.137+0000 7f0a5d5ff700 1 -- 172.21.15.2:0/501391938 wait complete. 2023-11-16T16:47:48.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.138+0000 7f0a5d5ff700 1 Processor -- start 2023-11-16T16:47:48.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.138+0000 7f0a5d5ff700 1 -- start start 2023-11-16T16:47:48.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.139+0000 7f0a5d5ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a58100290 0x7f0a581981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:48.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.139+0000 7f0a5d5ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a58101480 0x7f0a58198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:48.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.139+0000 7f0a5d5ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a58101e10 0x7f0a5819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:48.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.139+0000 7f0a5d5ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5819fd40 con 0x7f0a58101e10 2023-11-16T16:47:48.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.139+0000 7f0a5d5ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5819fe80 con 0x7f0a58100290 2023-11-16T16:47:48.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.139+0000 7f0a5d5ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a581a0010 con 0x7f0a58101480 2023-11-16T16:47:48.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.139+0000 7f0a577fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a58101e10 0x7f0a5819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:48.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.139+0000 7f0a577fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a58101e10 0x7f0a5819f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33160/0 (socket says 172.21.15.2:33160) 2023-11-16T16:47:48.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.139+0000 7f0a577fe700 1 -- 172.21.15.2:0/76478206 learned_addr learned my addr 172.21.15.2:0/76478206 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:48.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.139+0000 7f0a56ffd700 1 --2- 172.21.15.2:0/76478206 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a58100290 0x7f0a581981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:48.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.139+0000 7f0a567fc700 1 --2- 172.21.15.2:0/76478206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a58101480 0x7f0a58198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:48.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.139+0000 7f0a577fe700 1 -- 172.21.15.2:0/76478206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a58101480 msgr2=0x7f0a58198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:48.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.139+0000 7f0a577fe700 1 --2- 172.21.15.2:0/76478206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a58101480 0x7f0a58198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.139+0000 7f0a577fe700 1 -- 172.21.15.2:0/76478206 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a58100290 msgr2=0x7f0a581981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:48.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.139+0000 7f0a577fe700 1 --2- 172.21.15.2:0/76478206 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a58100290 0x7f0a581981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.139+0000 7f0a577fe700 1 -- 172.21.15.2:0/76478206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a480095f0 con 0x7f0a58101e10 2023-11-16T16:47:48.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.139+0000 7f0a56ffd700 1 --2- 172.21.15.2:0/76478206 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a58100290 0x7f0a581981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:48.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.140+0000 7f0a577fe700 1 --2- 172.21.15.2:0/76478206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a58101e10 0x7f0a5819f770 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0a50007ee0 tx=0x7f0a5000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:48.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.140+0000 7f0a3ffff700 1 -- 172.21.15.2:0/76478206 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a50017070 con 0x7f0a58101e10 2023-11-16T16:47:48.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.140+0000 7f0a3ffff700 1 -- 172.21.15.2:0/76478206 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a5000f040 con 0x7f0a58101e10 2023-11-16T16:47:48.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.140+0000 7f0a3ffff700 1 -- 172.21.15.2:0/76478206 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a50005580 con 0x7f0a58101e10 2023-11-16T16:47:48.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.140+0000 7f0a5d5ff700 1 -- 172.21.15.2:0/76478206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a5810bc40 con 0x7f0a58101e10 2023-11-16T16:47:48.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.140+0000 7f0a5d5ff700 1 -- 172.21.15.2:0/76478206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a5810bab0 con 0x7f0a58101e10 2023-11-16T16:47:48.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.141+0000 7f0a5d5ff700 1 -- 172.21.15.2:0/76478206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a581928d0 con 0x7f0a58101e10 2023-11-16T16:47:48.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.143+0000 7f0a3ffff700 1 -- 172.21.15.2:0/76478206 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f0a5000db20 con 0x7f0a58101e10 2023-11-16T16:47:48.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.144+0000 7f0a3ffff700 1 --2- 172.21.15.2:0/76478206 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0a400a11d0 0x7f0a400a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:48.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.144+0000 7f0a3ffff700 1 -- 172.21.15.2:0/76478206 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0a500cc320 con 0x7f0a58101e10 2023-11-16T16:47:48.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.145+0000 7f0a56ffd700 1 --2- 172.21.15.2:0/76478206 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0a400a11d0 0x7f0a400a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:48.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.145+0000 7f0a56ffd700 1 --2- 172.21.15.2:0/76478206 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0a400a11d0 0x7f0a400a3680 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800b2e0 tx=0x7f0a48005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:48.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.148+0000 7f0a3ffff700 1 -- 172.21.15.2:0/76478206 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0a50095700 con 0x7f0a58101e10 2023-11-16T16:47:48.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:47 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1961405683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:47:48.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:47 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2789764090' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T16:47:48.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.359+0000 7f0a5d5ff700 1 -- 172.21.15.2:0/76478206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a5802cfe0 con 0x7f0a58101e10 2023-11-16T16:47:48.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.361+0000 7f0a3ffff700 1 -- 172.21.15.2:0/76478206 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24379 (secure 0 0 0) 0x7f0a50098d20 con 0x7f0a58101e10 2023-11-16T16:47:48.363 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:48.363 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":91,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:47:44.849645+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:24.458452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:25.786005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:29.313181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:47:48.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.363+0000 7f0a5d5ff700 1 -- 172.21.15.2:0/76478206 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0a400a11d0 msgr2=0x7f0a400a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:48.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.363+0000 7f0a5d5ff700 1 --2- 172.21.15.2:0/76478206 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0a400a11d0 0x7f0a400a3680 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800b2e0 tx=0x7f0a48005fb0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.363+0000 7f0a5d5ff700 1 -- 172.21.15.2:0/76478206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a58101e10 msgr2=0x7f0a5819f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:48.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.363+0000 7f0a5d5ff700 1 --2- 172.21.15.2:0/76478206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a58101e10 0x7f0a5819f770 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0a50007ee0 tx=0x7f0a5000d3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.363+0000 7f0a5d5ff700 1 -- 172.21.15.2:0/76478206 shutdown_connections 2023-11-16T16:47:48.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.363+0000 7f0a5d5ff700 1 --2- 172.21.15.2:0/76478206 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0a400a11d0 0x7f0a400a3680 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.363+0000 7f0a5d5ff700 1 --2- 172.21.15.2:0/76478206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a58101e10 0x7f0a5819f770 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.363+0000 7f0a5d5ff700 1 --2- 172.21.15.2:0/76478206 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0a58100290 0x7f0a581981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.363+0000 7f0a5d5ff700 1 --2- 172.21.15.2:0/76478206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a58101480 0x7f0a58198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.363+0000 7f0a5d5ff700 1 -- 172.21.15.2:0/76478206 >> 172.21.15.2:0/76478206 conn(0x7f0a580fb870 msgr2=0x7f0a58106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:48.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.364+0000 7f0a5d5ff700 1 -- 172.21.15.2:0/76478206 shutdown_connections 2023-11-16T16:47:48.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.364+0000 7f0a5d5ff700 1 -- 172.21.15.2:0/76478206 wait complete. 2023-11-16T16:47:48.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:47 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1961405683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:47:48.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:47 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2789764090' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T16:47:48.384 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:47:48.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.586+0000 7fa90f1ff700 1 -- 172.21.15.2:0/303701975 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa908101e10 msgr2=0x7fa9081086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:48.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.586+0000 7fa90f1ff700 1 --2- 172.21.15.2:0/303701975 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa908101e10 0x7fa9081086d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa90000b0a0 tx=0x7fa90000b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.586+0000 7fa90f1ff700 1 -- 172.21.15.2:0/303701975 shutdown_connections 2023-11-16T16:47:48.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.587+0000 7fa90f1ff700 1 --2- 172.21.15.2:0/303701975 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa908100290 0x7fa9081006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.587+0000 7fa90f1ff700 1 --2- 172.21.15.2:0/303701975 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa908101e10 0x7fa9081086d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.587+0000 7fa90f1ff700 1 --2- 172.21.15.2:0/303701975 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa908101480 0x7fa9081018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.587+0000 7fa90f1ff700 1 -- 172.21.15.2:0/303701975 >> 172.21.15.2:0/303701975 conn(0x7fa9080fb870 msgr2=0x7fa9080fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:48.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.587+0000 7fa90f1ff700 1 -- 172.21.15.2:0/303701975 shutdown_connections 2023-11-16T16:47:48.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.587+0000 7fa90f1ff700 1 -- 172.21.15.2:0/303701975 wait complete. 2023-11-16T16:47:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.588+0000 7fa90f1ff700 1 Processor -- start 2023-11-16T16:47:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.588+0000 7fa90f1ff700 1 -- start start 2023-11-16T16:47:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.588+0000 7fa90f1ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa908100290 0x7fa908196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.588+0000 7fa90f1ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa908101480 0x7fa9081965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.588+0000 7fa90f1ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa908101e10 0x7fa90819d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.588+0000 7fa90f1ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90819dbe0 con 0x7fa908101e10 2023-11-16T16:47:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.588+0000 7fa90f1ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90819dd20 con 0x7fa908101480 2023-11-16T16:47:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.588+0000 7fa90f1ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90819deb0 con 0x7fa908100290 2023-11-16T16:47:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.588+0000 7fa907fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa908101480 0x7fa9081965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.588+0000 7fa90d79c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa908101e10 0x7fa90819d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.588+0000 7fa90cf9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa908100290 0x7fa908196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.588+0000 7fa907fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa908101480 0x7fa9081965a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:42712/0 (socket says 172.21.15.2:42712) 2023-11-16T16:47:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.588+0000 7fa90d79c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa908101e10 0x7fa90819d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33168/0 (socket says 172.21.15.2:33168) 2023-11-16T16:47:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.588+0000 7fa907fff700 1 -- 172.21.15.2:0/2394636908 learned_addr learned my addr 172.21.15.2:0/2394636908 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.589+0000 7fa90d79c700 1 -- 172.21.15.2:0/2394636908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa908100290 msgr2=0x7fa908196060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.589+0000 7fa90d79c700 1 --2- 172.21.15.2:0/2394636908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa908100290 0x7fa908196060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.589+0000 7fa90d79c700 1 -- 172.21.15.2:0/2394636908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa908101480 msgr2=0x7fa9081965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.589+0000 7fa90d79c700 1 --2- 172.21.15.2:0/2394636908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa908101480 0x7fa9081965a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.589+0000 7fa90d79c700 1 -- 172.21.15.2:0/2394636908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8fc009520 con 0x7fa908101e10 2023-11-16T16:47:48.592 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.589+0000 7fa907fff700 1 --2- 172.21.15.2:0/2394636908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa908101480 0x7fa9081965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:48.592 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.589+0000 7fa90cf9b700 1 --2- 172.21.15.2:0/2394636908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa908100290 0x7fa908196060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:48.592 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.589+0000 7fa90d79c700 1 --2- 172.21.15.2:0/2394636908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa908101e10 0x7fa90819d610 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa90000b070 tx=0x7fa90000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:48.592 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.589+0000 7fa905ffb700 1 -- 172.21.15.2:0/2394636908 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90000e040 con 0x7fa908101e10 2023-11-16T16:47:48.592 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.589+0000 7fa905ffb700 1 -- 172.21.15.2:0/2394636908 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa900009480 con 0x7fa908101e10 2023-11-16T16:47:48.592 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.589+0000 7fa905ffb700 1 -- 172.21.15.2:0/2394636908 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9000052a0 con 0x7fa908101e10 2023-11-16T16:47:48.592 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.589+0000 7fa90f1ff700 1 -- 172.21.15.2:0/2394636908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f4009520 con 0x7fa908101e10 2023-11-16T16:47:48.592 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.590+0000 7fa90f1ff700 1 -- 172.21.15.2:0/2394636908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa900009d00 con 0x7fa908101e10 2023-11-16T16:47:48.592 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.591+0000 7fa90f1ff700 1 -- 172.21.15.2:0/2394636908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9081077e0 con 0x7fa908101e10 2023-11-16T16:47:48.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.593+0000 7fa905ffb700 1 -- 172.21.15.2:0/2394636908 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fa90001d480 con 0x7fa908101e10 2023-11-16T16:47:48.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.594+0000 7fa905ffb700 1 --2- 172.21.15.2:0/2394636908 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa8f80a1150 0x7fa8f80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:48.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.594+0000 7fa905ffb700 1 -- 172.21.15.2:0/2394636908 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa9000ce100 con 0x7fa908101e10 2023-11-16T16:47:48.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.594+0000 7fa90cf9b700 1 --2- 172.21.15.2:0/2394636908 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa8f80a1150 0x7fa8f80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:48.595 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.595+0000 7fa90cf9b700 1 --2- 172.21.15.2:0/2394636908 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa8f80a1150 0x7fa8f80a3600 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc009fd0 tx=0x7fa8fc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:48.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.596+0000 7fa905ffb700 1 -- 172.21.15.2:0/2394636908 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa900097570 con 0x7fa908101e10 2023-11-16T16:47:48.806 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.806+0000 7fa90f1ff700 1 -- 172.21.15.2:0/2394636908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa90802cdd0 con 0x7fa908101e10 2023-11-16T16:47:48.808 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.807+0000 7fa905ffb700 1 -- 172.21.15.2:0/2394636908 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24379 (secure 0 0 0) 0x7fa900004030 con 0x7fa908101e10 2023-11-16T16:47:48.808 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:48.808 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":91,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:47:44.849645+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:24.458452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:25.786005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:29.313181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:47:48.811 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.810+0000 7fa90f1ff700 1 -- 172.21.15.2:0/2394636908 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa8f80a1150 msgr2=0x7fa8f80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:48.811 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.810+0000 7fa90f1ff700 1 --2- 172.21.15.2:0/2394636908 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa8f80a1150 0x7fa8f80a3600 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc009fd0 tx=0x7fa8fc009260 comp rx=0 tx=0).stop 2023-11-16T16:47:48.811 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.810+0000 7fa90f1ff700 1 -- 172.21.15.2:0/2394636908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa908101e10 msgr2=0x7fa90819d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:48.811 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.810+0000 7fa90f1ff700 1 --2- 172.21.15.2:0/2394636908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa908101e10 0x7fa90819d610 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa90000b070 tx=0x7fa90000bc80 comp rx=0 tx=0).stop 2023-11-16T16:47:48.811 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.810+0000 7fa90f1ff700 1 -- 172.21.15.2:0/2394636908 shutdown_connections 2023-11-16T16:47:48.811 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.810+0000 7fa90f1ff700 1 --2- 172.21.15.2:0/2394636908 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa8f80a1150 0x7fa8f80a3600 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.811 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.810+0000 7fa90f1ff700 1 --2- 172.21.15.2:0/2394636908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa908100290 0x7fa908196060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.810+0000 7fa90f1ff700 1 --2- 172.21.15.2:0/2394636908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa908101e10 0x7fa90819d610 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.810+0000 7fa90f1ff700 1 --2- 172.21.15.2:0/2394636908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa908101480 0x7fa9081965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:48.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.810+0000 7fa90f1ff700 1 -- 172.21.15.2:0/2394636908 >> 172.21.15.2:0/2394636908 conn(0x7fa9080fb870 msgr2=0x7fa908104920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:48.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.810+0000 7fa90f1ff700 1 -- 172.21.15.2:0/2394636908 shutdown_connections 2023-11-16T16:47:48.812 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:48.810+0000 7fa90f1ff700 1 -- 172.21.15.2:0/2394636908 wait complete. 2023-11-16T16:47:48.830 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-16T16:47:49.039 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.038+0000 7fc573fff700 1 -- 172.21.15.2:0/2442214140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc56c101e40 msgr2=0x7fc56c108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:49.039 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.038+0000 7fc573fff700 1 --2- 172.21.15.2:0/2442214140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc56c101e40 0x7fc56c108700 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc56400b0a0 tx=0x7fc56400b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.039 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.038+0000 7fc573fff700 1 -- 172.21.15.2:0/2442214140 shutdown_connections 2023-11-16T16:47:49.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.038+0000 7fc573fff700 1 --2- 172.21.15.2:0/2442214140 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc56c1002c0 0x7fc56c1006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.038+0000 7fc573fff700 1 --2- 172.21.15.2:0/2442214140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc56c101e40 0x7fc56c108700 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.038+0000 7fc573fff700 1 --2- 172.21.15.2:0/2442214140 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc56c1014b0 0x7fc56c101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.038+0000 7fc573fff700 1 -- 172.21.15.2:0/2442214140 >> 172.21.15.2:0/2442214140 conn(0x7fc56c0fb8a0 msgr2=0x7fc56c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:49.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.038+0000 7fc573fff700 1 -- 172.21.15.2:0/2442214140 shutdown_connections 2023-11-16T16:47:49.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.038+0000 7fc573fff700 1 -- 172.21.15.2:0/2442214140 wait complete. 2023-11-16T16:47:49.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.040+0000 7fc573fff700 1 Processor -- start 2023-11-16T16:47:49.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.040+0000 7fc573fff700 1 -- start start 2023-11-16T16:47:49.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.040+0000 7fc573fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc56c1002c0 0x7fc56c0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:49.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.040+0000 7fc573fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc56c1014b0 0x7fc56c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:49.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.040+0000 7fc573fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc56c101e40 0x7fc56c073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:49.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.040+0000 7fc573fff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c073bc0 con 0x7fc56c1002c0 2023-11-16T16:47:49.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.040+0000 7fc573fff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c073d30 con 0x7fc56c1014b0 2023-11-16T16:47:49.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.041+0000 7fc573fff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c1a01a0 con 0x7fc56c101e40 2023-11-16T16:47:49.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.041+0000 7fc571d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc56c1002c0 0x7fc56c0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:49.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.041+0000 7fc571d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc56c1002c0 0x7fc56c0748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33184/0 (socket says 172.21.15.2:33184) 2023-11-16T16:47:49.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.041+0000 7fc571d9b700 1 -- 172.21.15.2:0/3318917676 learned_addr learned my addr 172.21.15.2:0/3318917676 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:49.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.041+0000 7fc57159a700 1 --2- 172.21.15.2:0/3318917676 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc56c1014b0 0x7fc56c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:49.043 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.041+0000 7fc57259c700 1 --2- 172.21.15.2:0/3318917676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc56c101e40 0x7fc56c073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:49.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.041+0000 7fc571d9b700 1 -- 172.21.15.2:0/3318917676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc56c101e40 msgr2=0x7fc56c073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:49.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.041+0000 7fc571d9b700 1 --2- 172.21.15.2:0/3318917676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc56c101e40 0x7fc56c073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.041+0000 7fc571d9b700 1 -- 172.21.15.2:0/3318917676 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc56c1014b0 msgr2=0x7fc56c072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:49.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.041+0000 7fc571d9b700 1 --2- 172.21.15.2:0/3318917676 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc56c1014b0 0x7fc56c072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.041+0000 7fc571d9b700 1 -- 172.21.15.2:0/3318917676 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc564009d00 con 0x7fc56c1002c0 2023-11-16T16:47:49.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.041+0000 7fc57259c700 1 --2- 172.21.15.2:0/3318917676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc56c101e40 0x7fc56c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:49.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.041+0000 7fc57159a700 1 --2- 172.21.15.2:0/3318917676 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc56c1014b0 0x7fc56c072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:49.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.041+0000 7fc571d9b700 1 --2- 172.21.15.2:0/3318917676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc56c1002c0 0x7fc56c0748a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc56800e970 tx=0x7fc56800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:49.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.042+0000 7fc562ffd700 1 -- 172.21.15.2:0/3318917676 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc568009790 con 0x7fc56c1002c0 2023-11-16T16:47:49.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.042+0000 7fc562ffd700 1 -- 172.21.15.2:0/3318917676 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5680044e0 con 0x7fc56c1002c0 2023-11-16T16:47:49.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.042+0000 7fc573fff700 1 -- 172.21.15.2:0/3318917676 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc56c10bc90 con 0x7fc56c1002c0 2023-11-16T16:47:49.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.042+0000 7fc562ffd700 1 -- 172.21.15.2:0/3318917676 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc568003e30 con 0x7fc56c1002c0 2023-11-16T16:47:49.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.042+0000 7fc573fff700 1 -- 172.21.15.2:0/3318917676 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc56c10bb00 con 0x7fc56c1002c0 2023-11-16T16:47:49.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.043+0000 7fc573fff700 1 -- 172.21.15.2:0/3318917676 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc56c1a0940 con 0x7fc56c1002c0 2023-11-16T16:47:49.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.045+0000 7fc562ffd700 1 -- 172.21.15.2:0/3318917676 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fc568003f90 con 0x7fc56c1002c0 2023-11-16T16:47:49.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.046+0000 7fc562ffd700 1 --2- 172.21.15.2:0/3318917676 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc5580a11d0 0x7fc5580a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:49.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.046+0000 7fc562ffd700 1 -- 172.21.15.2:0/3318917676 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc568014070 con 0x7fc56c1002c0 2023-11-16T16:47:49.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.046+0000 7fc57159a700 1 --2- 172.21.15.2:0/3318917676 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc5580a11d0 0x7fc5580a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:49.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.047+0000 7fc57159a700 1 --2- 172.21.15.2:0/3318917676 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc5580a11d0 0x7fc5580a3680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc55c009cb0 tx=0x7fc55c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:49.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.050+0000 7fc562ffd700 1 -- 172.21.15.2:0/3318917676 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc568096300 con 0x7fc56c1002c0 2023-11-16T16:47:49.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:48 smithi002 ceph-mon[128712]: pgmap v293: 129 pgs: 1 peering, 128 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2023-11-16T16:47:49.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:48 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/76478206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:49.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:48 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2394636908' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:49.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:48 smithi141 ceph-mon[131184]: pgmap v293: 129 pgs: 1 peering, 128 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2023-11-16T16:47:49.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:48 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/76478206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:49.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:48 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2394636908' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:49.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.303+0000 7fc573fff700 1 -- 172.21.15.2:0/3318917676 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc56c1070d0 con 0x7fc56c1002c0 2023-11-16T16:47:49.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:48 smithi155 ceph-mon[131607]: pgmap v293: 129 pgs: 1 peering, 128 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2023-11-16T16:47:49.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:48 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/76478206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:49.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:48 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2394636908' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:49.882 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.881+0000 7fc562ffd700 1 -- 172.21.15.2:0/3318917676 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc568016330 con 0x7fc56c1002c0 2023-11-16T16:47:49.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.884+0000 7fc573fff700 1 -- 172.21.15.2:0/3318917676 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc5580a11d0 msgr2=0x7fc5580a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:49.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.884+0000 7fc573fff700 1 --2- 172.21.15.2:0/3318917676 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc5580a11d0 0x7fc5580a3680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc55c009cb0 tx=0x7fc55c009260 comp rx=0 tx=0).stop 2023-11-16T16:47:49.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.884+0000 7fc573fff700 1 -- 172.21.15.2:0/3318917676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc56c1002c0 msgr2=0x7fc56c0748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:49.885 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.884+0000 7fc573fff700 1 --2- 172.21.15.2:0/3318917676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc56c1002c0 0x7fc56c0748a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc56800e970 tx=0x7fc56800ed30 comp rx=0 tx=0).stop 2023-11-16T16:47:49.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.884+0000 7fc573fff700 1 -- 172.21.15.2:0/3318917676 shutdown_connections 2023-11-16T16:47:49.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.884+0000 7fc573fff700 1 --2- 172.21.15.2:0/3318917676 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc5580a11d0 0x7fc5580a3680 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.884+0000 7fc573fff700 1 --2- 172.21.15.2:0/3318917676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc56c101e40 0x7fc56c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.884+0000 7fc573fff700 1 --2- 172.21.15.2:0/3318917676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc56c1002c0 0x7fc56c0748a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.884+0000 7fc573fff700 1 --2- 172.21.15.2:0/3318917676 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc56c1014b0 0x7fc56c072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:49.886 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.884+0000 7fc573fff700 1 -- 172.21.15.2:0/3318917676 >> 172.21.15.2:0/3318917676 conn(0x7fc56c0fb8a0 msgr2=0x7fc56c0fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:49.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.884+0000 7fc573fff700 1 -- 172.21.15.2:0/3318917676 shutdown_connections 2023-11-16T16:47:49.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:49.884+0000 7fc573fff700 1 -- 172.21.15.2:0/3318917676 wait complete. 2023-11-16T16:47:49.908 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2023-11-16T16:47:49.908 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-11-16T16:47:50.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.120+0000 7fbf5afff700 1 -- 172.21.15.2:0/3868329508 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf54105f80 msgr2=0x7fbf541083e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:50.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.120+0000 7fbf5afff700 1 --2- 172.21.15.2:0/3868329508 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf54105f80 0x7fbf541083e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c00b0a0 tx=0x7fbf4c00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.120+0000 7fbf5afff700 1 -- 172.21.15.2:0/3868329508 shutdown_connections 2023-11-16T16:47:50.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.120+0000 7fbf5afff700 1 --2- 172.21.15.2:0/3868329508 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf54105f80 0x7fbf541083e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.120+0000 7fbf5afff700 1 --2- 172.21.15.2:0/3868329508 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf54100d40 0x7fbf54103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.120+0000 7fbf5afff700 1 --2- 172.21.15.2:0/3868329508 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf54103660 0x7fbf54105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.120+0000 7fbf5afff700 1 -- 172.21.15.2:0/3868329508 >> 172.21.15.2:0/3868329508 conn(0x7fbf540fa7c0 msgr2=0x7fbf540fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:50.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.120+0000 7fbf5afff700 1 -- 172.21.15.2:0/3868329508 shutdown_connections 2023-11-16T16:47:50.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.120+0000 7fbf5afff700 1 -- 172.21.15.2:0/3868329508 wait complete. 2023-11-16T16:47:50.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.121+0000 7fbf5afff700 1 Processor -- start 2023-11-16T16:47:50.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.121+0000 7fbf5afff700 1 -- start start 2023-11-16T16:47:50.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.121+0000 7fbf5afff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf54100d40 0x7fbf54198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:50.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.121+0000 7fbf5afff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf54103660 0x7fbf54198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:50.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.121+0000 7fbf5afff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf54105f80 0x7fbf5419f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:50.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.121+0000 7fbf5afff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5419fea0 con 0x7fbf54103660 2023-11-16T16:47:50.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.121+0000 7fbf5afff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5419ffe0 con 0x7fbf54100d40 2023-11-16T16:47:50.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.122+0000 7fbf5afff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf541a0170 con 0x7fbf54105f80 2023-11-16T16:47:50.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.122+0000 7fbf58d9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf54100d40 0x7fbf54198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:50.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.122+0000 7fbf53fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf54103660 0x7fbf54198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:50.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.122+0000 7fbf58d9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf54100d40 0x7fbf54198320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:42740/0 (socket says 172.21.15.2:42740) 2023-11-16T16:47:50.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.122+0000 7fbf58d9b700 1 -- 172.21.15.2:0/1964549247 learned_addr learned my addr 172.21.15.2:0/1964549247 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:50.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.122+0000 7fbf5959c700 1 --2- 172.21.15.2:0/1964549247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf54105f80 0x7fbf5419f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:50.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.122+0000 7fbf53fff700 1 -- 172.21.15.2:0/1964549247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf54105f80 msgr2=0x7fbf5419f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:50.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.122+0000 7fbf53fff700 1 --2- 172.21.15.2:0/1964549247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf54105f80 0x7fbf5419f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.122+0000 7fbf53fff700 1 -- 172.21.15.2:0/1964549247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf54100d40 msgr2=0x7fbf54198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:50.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.122+0000 7fbf53fff700 1 --2- 172.21.15.2:0/1964549247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf54100d40 0x7fbf54198320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.122+0000 7fbf53fff700 1 -- 172.21.15.2:0/1964549247 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4c009d00 con 0x7fbf54103660 2023-11-16T16:47:50.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.122+0000 7fbf5959c700 1 --2- 172.21.15.2:0/1964549247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf54105f80 0x7fbf5419f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:50.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.122+0000 7fbf58d9b700 1 --2- 172.21.15.2:0/1964549247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf54100d40 0x7fbf54198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:50.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.123+0000 7fbf53fff700 1 --2- 172.21.15.2:0/1964549247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf54103660 0x7fbf54198860 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbf4000e9f0 tx=0x7fbf4000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:50.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.123+0000 7fbf51ffb700 1 -- 172.21.15.2:0/1964549247 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4000eee0 con 0x7fbf54103660 2023-11-16T16:47:50.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.123+0000 7fbf51ffb700 1 -- 172.21.15.2:0/1964549247 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf4000cbd0 con 0x7fbf54103660 2023-11-16T16:47:50.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.123+0000 7fbf51ffb700 1 -- 172.21.15.2:0/1964549247 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf40018680 con 0x7fbf54103660 2023-11-16T16:47:50.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.123+0000 7fbf5afff700 1 -- 172.21.15.2:0/1964549247 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5410bc70 con 0x7fbf54103660 2023-11-16T16:47:50.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.123+0000 7fbf5afff700 1 -- 172.21.15.2:0/1964549247 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5410bae0 con 0x7fbf54103660 2023-11-16T16:47:50.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.124+0000 7fbf5afff700 1 -- 172.21.15.2:0/1964549247 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf54107af0 con 0x7fbf54103660 2023-11-16T16:47:50.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.128+0000 7fbf51ffb700 1 -- 172.21.15.2:0/1964549247 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fbf4000cd40 con 0x7fbf54103660 2023-11-16T16:47:50.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.129+0000 7fbf51ffb700 1 --2- 172.21.15.2:0/1964549247 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbf440a11d0 0x7fbf440a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:50.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.129+0000 7fbf51ffb700 1 -- 172.21.15.2:0/1964549247 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbf40014070 con 0x7fbf54103660 2023-11-16T16:47:50.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.129+0000 7fbf58d9b700 1 --2- 172.21.15.2:0/1964549247 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbf440a11d0 0x7fbf440a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:50.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.130+0000 7fbf58d9b700 1 --2- 172.21.15.2:0/1964549247 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbf440a11d0 0x7fbf440a3680 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbf48011550 tx=0x7fbf48009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:50.132 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.132+0000 7fbf51ffb700 1 -- 172.21.15.2:0/1964549247 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbf40096460 con 0x7fbf54103660 2023-11-16T16:47:50.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:49 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3318917676' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-11-16T16:47:50.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:49 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3318917676' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-11-16T16:47:50.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:49 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3318917676' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-11-16T16:47:50.392 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.392+0000 7fbf5afff700 1 -- 172.21.15.2:0/1964549247 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7fbf54061700 con 0x7fbf54103660 2023-11-16T16:47:50.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.887+0000 7fbf51ffb700 1 -- 172.21.15.2:0/1964549247 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7fbf400260b0 con 0x7fbf54103660 2023-11-16T16:47:50.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.890+0000 7fbf5afff700 1 -- 172.21.15.2:0/1964549247 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbf440a11d0 msgr2=0x7fbf440a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:50.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.890+0000 7fbf5afff700 1 --2- 172.21.15.2:0/1964549247 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbf440a11d0 0x7fbf440a3680 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbf48011550 tx=0x7fbf48009260 comp rx=0 tx=0).stop 2023-11-16T16:47:50.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.890+0000 7fbf5afff700 1 -- 172.21.15.2:0/1964549247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf54103660 msgr2=0x7fbf54198860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:50.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.890+0000 7fbf5afff700 1 --2- 172.21.15.2:0/1964549247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf54103660 0x7fbf54198860 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbf4000e9f0 tx=0x7fbf4000ed00 comp rx=0 tx=0).stop 2023-11-16T16:47:50.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.890+0000 7fbf5afff700 1 -- 172.21.15.2:0/1964549247 shutdown_connections 2023-11-16T16:47:50.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.890+0000 7fbf5afff700 1 --2- 172.21.15.2:0/1964549247 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbf440a11d0 0x7fbf440a3680 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.890+0000 7fbf5afff700 1 --2- 172.21.15.2:0/1964549247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf54105f80 0x7fbf5419f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.890+0000 7fbf5afff700 1 --2- 172.21.15.2:0/1964549247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbf54100d40 0x7fbf54198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.890+0000 7fbf5afff700 1 --2- 172.21.15.2:0/1964549247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf54103660 0x7fbf54198860 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:50.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.890+0000 7fbf5afff700 1 -- 172.21.15.2:0/1964549247 >> 172.21.15.2:0/1964549247 conn(0x7fbf540fa7c0 msgr2=0x7fbf540ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:50.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.890+0000 7fbf5afff700 1 -- 172.21.15.2:0/1964549247 shutdown_connections 2023-11-16T16:47:50.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:50.890+0000 7fbf5afff700 1 -- 172.21.15.2:0/1964549247 wait complete. 2023-11-16T16:47:50.914 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-11-16T16:47:51.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.123+0000 7faa423ff700 1 -- 172.21.15.2:0/1819144950 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa3c101e40 msgr2=0x7faa3c108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:51.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.123+0000 7faa423ff700 1 --2- 172.21.15.2:0/1819144950 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa3c101e40 0x7faa3c108700 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faa3400b0a0 tx=0x7faa3400b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.123+0000 7faa423ff700 1 -- 172.21.15.2:0/1819144950 shutdown_connections 2023-11-16T16:47:51.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.123+0000 7faa423ff700 1 --2- 172.21.15.2:0/1819144950 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa3c101e40 0x7faa3c108700 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.123+0000 7faa423ff700 1 --2- 172.21.15.2:0/1819144950 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faa3c1002c0 0x7faa3c1006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.123+0000 7faa423ff700 1 --2- 172.21.15.2:0/1819144950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa3c1014b0 0x7faa3c101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.123+0000 7faa423ff700 1 -- 172.21.15.2:0/1819144950 >> 172.21.15.2:0/1819144950 conn(0x7faa3c0fb8a0 msgr2=0x7faa3c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:51.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.123+0000 7faa423ff700 1 -- 172.21.15.2:0/1819144950 shutdown_connections 2023-11-16T16:47:51.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.123+0000 7faa423ff700 1 -- 172.21.15.2:0/1819144950 wait complete. 2023-11-16T16:47:51.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.124+0000 7faa423ff700 1 Processor -- start 2023-11-16T16:47:51.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.124+0000 7faa423ff700 1 -- start start 2023-11-16T16:47:51.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.124+0000 7faa423ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa3c1002c0 0x7faa3c195f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:51.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.125+0000 7faa423ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa3c1014b0 0x7faa3c196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:51.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.125+0000 7faa423ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faa3c101e40 0x7faa3c19d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:51.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.125+0000 7faa423ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c19daa0 con 0x7faa3c1014b0 2023-11-16T16:47:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.125+0000 7faa423ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c19dbe0 con 0x7faa3c101e40 2023-11-16T16:47:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.125+0000 7faa423ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c19dd70 con 0x7faa3c1002c0 2023-11-16T16:47:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.125+0000 7faa3b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa3c1014b0 0x7faa3c196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.125+0000 7faa3b7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa3c1014b0 0x7faa3c196460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33208/0 (socket says 172.21.15.2:33208) 2023-11-16T16:47:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.125+0000 7faa3b7fe700 1 -- 172.21.15.2:0/3859890649 learned_addr learned my addr 172.21.15.2:0/3859890649 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.125+0000 7faa4099c700 1 --2- 172.21.15.2:0/3859890649 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faa3c101e40 0x7faa3c19d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.125+0000 7faa3bfff700 1 --2- 172.21.15.2:0/3859890649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa3c1002c0 0x7faa3c195f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.125+0000 7faa3b7fe700 1 -- 172.21.15.2:0/3859890649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa3c1002c0 msgr2=0x7faa3c195f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.125+0000 7faa3b7fe700 1 --2- 172.21.15.2:0/3859890649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa3c1002c0 0x7faa3c195f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.125+0000 7faa3b7fe700 1 -- 172.21.15.2:0/3859890649 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faa3c101e40 msgr2=0x7faa3c19d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.125+0000 7faa3b7fe700 1 --2- 172.21.15.2:0/3859890649 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faa3c101e40 0x7faa3c19d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.125+0000 7faa3b7fe700 1 -- 172.21.15.2:0/3859890649 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa30009520 con 0x7faa3c1014b0 2023-11-16T16:47:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.126+0000 7faa3b7fe700 1 --2- 172.21.15.2:0/3859890649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa3c1014b0 0x7faa3c196460 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7faa3000eb30 tx=0x7faa3000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.126+0000 7faa397fa700 1 -- 172.21.15.2:0/3859890649 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa30009880 con 0x7faa3c1014b0 2023-11-16T16:47:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.126+0000 7faa397fa700 1 -- 172.21.15.2:0/3859890649 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa300044e0 con 0x7faa3c1014b0 2023-11-16T16:47:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.126+0000 7faa423ff700 1 -- 172.21.15.2:0/3859890649 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa34009d00 con 0x7faa3c1014b0 2023-11-16T16:47:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.126+0000 7faa397fa700 1 -- 172.21.15.2:0/3859890649 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa30003d80 con 0x7faa3c1014b0 2023-11-16T16:47:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.126+0000 7faa423ff700 1 -- 172.21.15.2:0/3859890649 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa3c10bbc0 con 0x7faa3c1014b0 2023-11-16T16:47:51.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.127+0000 7faa423ff700 1 -- 172.21.15.2:0/3859890649 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa3c107150 con 0x7faa3c1014b0 2023-11-16T16:47:51.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.129+0000 7faa397fa700 1 -- 172.21.15.2:0/3859890649 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7faa30003f20 con 0x7faa3c1014b0 2023-11-16T16:47:51.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.130+0000 7faa397fa700 1 --2- 172.21.15.2:0/3859890649 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7faa240a11d0 0x7faa240a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:51.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.130+0000 7faa397fa700 1 -- 172.21.15.2:0/3859890649 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7faa30014070 con 0x7faa3c1014b0 2023-11-16T16:47:51.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.131+0000 7faa3bfff700 1 --2- 172.21.15.2:0/3859890649 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7faa240a11d0 0x7faa240a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:51.132 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.131+0000 7faa3bfff700 1 --2- 172.21.15.2:0/3859890649 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7faa240a11d0 0x7faa240a3680 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faa3c101310 tx=0x7faa2c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:51.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.134+0000 7faa397fa700 1 -- 172.21.15.2:0/3859890649 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7faa30096540 con 0x7faa3c1014b0 2023-11-16T16:47:51.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:50 smithi002 ceph-mon[128712]: pgmap v294: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s wr, 0 op/s 2023-11-16T16:47:51.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:50 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3318917676' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-11-16T16:47:51.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:50 smithi002 ceph-mon[128712]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:51.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:50 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1964549247' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-11-16T16:47:51.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:50 smithi141 ceph-mon[131184]: pgmap v294: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s wr, 0 op/s 2023-11-16T16:47:51.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:50 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3318917676' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-11-16T16:47:51.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:50 smithi141 ceph-mon[131184]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:51.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:50 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1964549247' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-11-16T16:47:51.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:50 smithi155 ceph-mon[131607]: pgmap v294: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s wr, 0 op/s 2023-11-16T16:47:51.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:50 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3318917676' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-11-16T16:47:51.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:50 smithi155 ceph-mon[131607]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:51.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:50 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1964549247' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-11-16T16:47:51.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.388+0000 7faa423ff700 1 -- 172.21.15.2:0/3859890649 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7faa3c02cfe0 con 0x7faa3c1014b0 2023-11-16T16:47:51.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.893+0000 7faa397fa700 1 -- 172.21.15.2:0/3859890649 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7faa3001e2e0 con 0x7faa3c1014b0 2023-11-16T16:47:51.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.896+0000 7faa423ff700 1 -- 172.21.15.2:0/3859890649 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7faa240a11d0 msgr2=0x7faa240a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:51.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.896+0000 7faa423ff700 1 --2- 172.21.15.2:0/3859890649 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7faa240a11d0 0x7faa240a3680 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faa3c101310 tx=0x7faa2c009260 comp rx=0 tx=0).stop 2023-11-16T16:47:51.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.896+0000 7faa423ff700 1 -- 172.21.15.2:0/3859890649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa3c1014b0 msgr2=0x7faa3c196460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:51.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.896+0000 7faa423ff700 1 --2- 172.21.15.2:0/3859890649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa3c1014b0 0x7faa3c196460 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7faa3000eb30 tx=0x7faa3000ee40 comp rx=0 tx=0).stop 2023-11-16T16:47:51.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.896+0000 7faa423ff700 1 -- 172.21.15.2:0/3859890649 shutdown_connections 2023-11-16T16:47:51.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.896+0000 7faa423ff700 1 --2- 172.21.15.2:0/3859890649 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7faa240a11d0 0x7faa240a3680 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.896+0000 7faa423ff700 1 --2- 172.21.15.2:0/3859890649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa3c1002c0 0x7faa3c195f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.896+0000 7faa423ff700 1 --2- 172.21.15.2:0/3859890649 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faa3c101e40 0x7faa3c19d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.896+0000 7faa423ff700 1 --2- 172.21.15.2:0/3859890649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa3c1014b0 0x7faa3c196460 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:51.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.896+0000 7faa423ff700 1 -- 172.21.15.2:0/3859890649 >> 172.21.15.2:0/3859890649 conn(0x7faa3c0fb8a0 msgr2=0x7faa3c0fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:51.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.896+0000 7faa423ff700 1 -- 172.21.15.2:0/3859890649 shutdown_connections 2023-11-16T16:47:51.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:51.896+0000 7faa423ff700 1 -- 172.21.15.2:0/3859890649 wait complete. 2023-11-16T16:47:51.923 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-11-16T16:47:52.134 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.133+0000 7f984c9ff700 1 -- 172.21.15.2:0/4085658010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98481002c0 msgr2=0x7f98481006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:52.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.133+0000 7f984c9ff700 1 --2- 172.21.15.2:0/4085658010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98481002c0 0x7f98481006d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9838009990 tx=0x7f9838009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.133+0000 7f984c9ff700 1 -- 172.21.15.2:0/4085658010 shutdown_connections 2023-11-16T16:47:52.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.133+0000 7f984c9ff700 1 --2- 172.21.15.2:0/4085658010 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9848101e40 0x7f9848108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.133+0000 7f984c9ff700 1 --2- 172.21.15.2:0/4085658010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98481002c0 0x7f98481006d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.133+0000 7f984c9ff700 1 --2- 172.21.15.2:0/4085658010 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98481014b0 0x7f9848101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.133+0000 7f984c9ff700 1 -- 172.21.15.2:0/4085658010 >> 172.21.15.2:0/4085658010 conn(0x7f98480fb8a0 msgr2=0x7f98480fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:52.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.133+0000 7f984c9ff700 1 -- 172.21.15.2:0/4085658010 shutdown_connections 2023-11-16T16:47:52.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.133+0000 7f984c9ff700 1 -- 172.21.15.2:0/4085658010 wait complete. 2023-11-16T16:47:52.137 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.134+0000 7f984c9ff700 1 Processor -- start 2023-11-16T16:47:52.137 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.135+0000 7f984c9ff700 1 -- start start 2023-11-16T16:47:52.137 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.135+0000 7f984c9ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98481002c0 0x7f98480748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:52.137 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.135+0000 7f984c9ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98481014b0 0x7f9848072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:52.137 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.135+0000 7f984c9ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9848101e40 0x7f9848073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:52.137 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.135+0000 7f984c9ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9848073bc0 con 0x7f98481002c0 2023-11-16T16:47:52.137 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.135+0000 7f984c9ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9848073d30 con 0x7f98481014b0 2023-11-16T16:47:52.138 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.135+0000 7f984c9ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98481a00f0 con 0x7f9848101e40 2023-11-16T16:47:52.138 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.135+0000 7f984659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98481002c0 0x7f98480748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:52.138 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.135+0000 7f984659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98481002c0 0x7f98480748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33224/0 (socket says 172.21.15.2:33224) 2023-11-16T16:47:52.138 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.135+0000 7f984659c700 1 -- 172.21.15.2:0/3775173864 learned_addr learned my addr 172.21.15.2:0/3775173864 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:52.138 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.135+0000 7f9845d9b700 1 --2- 172.21.15.2:0/3775173864 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98481014b0 0x7f9848072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:52.138 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.135+0000 7f9846d9d700 1 --2- 172.21.15.2:0/3775173864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9848101e40 0x7f9848073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:52.138 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.136+0000 7f984659c700 1 -- 172.21.15.2:0/3775173864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9848101e40 msgr2=0x7f9848073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:52.138 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.136+0000 7f984659c700 1 --2- 172.21.15.2:0/3775173864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9848101e40 0x7f9848073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.138 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.136+0000 7f984659c700 1 -- 172.21.15.2:0/3775173864 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98481014b0 msgr2=0x7f9848072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:52.138 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.136+0000 7f984659c700 1 --2- 172.21.15.2:0/3775173864 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98481014b0 0x7f9848072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.138 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.136+0000 7f984659c700 1 -- 172.21.15.2:0/3775173864 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98380095f0 con 0x7f98481002c0 2023-11-16T16:47:52.139 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.136+0000 7f984659c700 1 --2- 172.21.15.2:0/3775173864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98481002c0 0x7f98480748a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9838005740 tx=0x7f9838005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:52.139 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.136+0000 7f98377fe700 1 -- 172.21.15.2:0/3775173864 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983801d070 con 0x7f98481002c0 2023-11-16T16:47:52.139 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.136+0000 7f98377fe700 1 -- 172.21.15.2:0/3775173864 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9838021410 con 0x7f98481002c0 2023-11-16T16:47:52.139 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.136+0000 7f984c9ff700 1 -- 172.21.15.2:0/3775173864 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f984810bc90 con 0x7f98481002c0 2023-11-16T16:47:52.139 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.136+0000 7f98377fe700 1 -- 172.21.15.2:0/3775173864 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9838017650 con 0x7f98481002c0 2023-11-16T16:47:52.139 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.136+0000 7f984c9ff700 1 -- 172.21.15.2:0/3775173864 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f984810bb00 con 0x7f98481002c0 2023-11-16T16:47:52.139 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.137+0000 7f984c9ff700 1 -- 172.21.15.2:0/3775173864 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9848107850 con 0x7f98481002c0 2023-11-16T16:47:52.140 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.139+0000 7f98377fe700 1 -- 172.21.15.2:0/3775173864 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f9838021580 con 0x7f98481002c0 2023-11-16T16:47:52.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.141+0000 7f98377fe700 1 --2- 172.21.15.2:0/3775173864 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f98300a1150 0x7f98300a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:52.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.141+0000 7f98377fe700 1 -- 172.21.15.2:0/3775173864 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f98380cdda0 con 0x7f98481002c0 2023-11-16T16:47:52.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.141+0000 7f9845d9b700 1 --2- 172.21.15.2:0/3775173864 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f98300a1150 0x7f98300a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:52.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.142+0000 7f9845d9b700 1 --2- 172.21.15.2:0/3775173864 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f98300a1150 0x7f98300a3600 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f983c009c60 tx=0x7f983c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:52.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.144+0000 7f98377fe700 1 -- 172.21.15.2:0/3775173864 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9838097330 con 0x7f98481002c0 2023-11-16T16:47:52.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:51 smithi002 ceph-mon[128712]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-11-16T16:47:52.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:51 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1964549247' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-11-16T16:47:52.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:51 smithi002 ceph-mon[128712]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-11-16T16:47:52.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:51 smithi002 ceph-mon[128712]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:52.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:51 smithi002 ceph-mon[128712]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-11-16T16:47:52.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:51 smithi002 ceph-mon[128712]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-11-16T16:47:52.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:51 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3859890649' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-11-16T16:47:52.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:51 smithi141 ceph-mon[131184]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-11-16T16:47:52.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:51 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1964549247' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-11-16T16:47:52.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:51 smithi141 ceph-mon[131184]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-11-16T16:47:52.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:51 smithi141 ceph-mon[131184]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:52.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:51 smithi141 ceph-mon[131184]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-11-16T16:47:52.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:51 smithi141 ceph-mon[131184]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-11-16T16:47:52.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:51 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3859890649' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-11-16T16:47:52.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:51 smithi155 ceph-mon[131607]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-11-16T16:47:52.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:51 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1964549247' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-11-16T16:47:52.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:51 smithi155 ceph-mon[131607]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-11-16T16:47:52.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:51 smithi155 ceph-mon[131607]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-11-16T16:47:52.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:51 smithi155 ceph-mon[131607]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-11-16T16:47:52.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:51 smithi155 ceph-mon[131607]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-11-16T16:47:52.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:51 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3859890649' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-11-16T16:47:52.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.399+0000 7f984c9ff700 1 -- 172.21.15.2:0/3775173864 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f984802cfe0 con 0x7f98481002c0 2023-11-16T16:47:52.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.900+0000 7f98377fe700 1 -- 172.21.15.2:0/3775173864 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7f98380260b0 con 0x7f98481002c0 2023-11-16T16:47:52.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.903+0000 7f984c9ff700 1 -- 172.21.15.2:0/3775173864 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f98300a1150 msgr2=0x7f98300a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:52.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.903+0000 7f984c9ff700 1 --2- 172.21.15.2:0/3775173864 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f98300a1150 0x7f98300a3600 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f983c009c60 tx=0x7f983c009260 comp rx=0 tx=0).stop 2023-11-16T16:47:52.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.903+0000 7f984c9ff700 1 -- 172.21.15.2:0/3775173864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98481002c0 msgr2=0x7f98480748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:52.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.903+0000 7f984c9ff700 1 --2- 172.21.15.2:0/3775173864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98481002c0 0x7f98480748a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9838005740 tx=0x7f9838005770 comp rx=0 tx=0).stop 2023-11-16T16:47:52.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.903+0000 7f984c9ff700 1 -- 172.21.15.2:0/3775173864 shutdown_connections 2023-11-16T16:47:52.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.903+0000 7f984c9ff700 1 --2- 172.21.15.2:0/3775173864 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f98300a1150 0x7f98300a3600 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.903+0000 7f984c9ff700 1 --2- 172.21.15.2:0/3775173864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9848101e40 0x7f9848073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.906 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.903+0000 7f984c9ff700 1 --2- 172.21.15.2:0/3775173864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98481002c0 0x7f98480748a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.906 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.903+0000 7f984c9ff700 1 --2- 172.21.15.2:0/3775173864 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98481014b0 0x7f9848072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:52.906 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.903+0000 7f984c9ff700 1 -- 172.21.15.2:0/3775173864 >> 172.21.15.2:0/3775173864 conn(0x7f98480fb8a0 msgr2=0x7f9848104990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:52.906 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.903+0000 7f984c9ff700 1 -- 172.21.15.2:0/3775173864 shutdown_connections 2023-11-16T16:47:52.906 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:52.903+0000 7f984c9ff700 1 -- 172.21.15.2:0/3775173864 wait complete. 2023-11-16T16:47:52.927 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2023-11-16T16:47:52.928 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 '' 2023-11-16T16:47:53.140 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.138+0000 7f63181ff700 1 -- 172.21.15.2:0/3657444395 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63101014b0 msgr2=0x7f6310101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.140 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.138+0000 7f63181ff700 1 --2- 172.21.15.2:0/3657444395 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63101014b0 0x7f6310101900 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6300009990 tx=0x7f6300009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.140 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.138+0000 7f63181ff700 1 -- 172.21.15.2:0/3657444395 shutdown_connections 2023-11-16T16:47:53.140 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.138+0000 7f63181ff700 1 --2- 172.21.15.2:0/3657444395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63101002c0 0x7f63101006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.140 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.138+0000 7f63181ff700 1 --2- 172.21.15.2:0/3657444395 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6310101e40 0x7f6310108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.138+0000 7f63181ff700 1 --2- 172.21.15.2:0/3657444395 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63101014b0 0x7f6310101900 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.138+0000 7f63181ff700 1 -- 172.21.15.2:0/3657444395 >> 172.21.15.2:0/3657444395 conn(0x7f63100fb8a0 msgr2=0x7f63100fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:53.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.138+0000 7f63181ff700 1 -- 172.21.15.2:0/3657444395 shutdown_connections 2023-11-16T16:47:53.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.138+0000 7f63181ff700 1 -- 172.21.15.2:0/3657444395 wait complete. 2023-11-16T16:47:53.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.140+0000 7f63181ff700 1 Processor -- start 2023-11-16T16:47:53.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.140+0000 7f63181ff700 1 -- start start 2023-11-16T16:47:53.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.140+0000 7f63181ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63101002c0 0x7f6310196000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:53.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.140+0000 7f63181ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63101014b0 0x7f6310196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:53.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.140+0000 7f63181ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6310101e40 0x7f631019d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:53.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.140+0000 7f63181ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631019db80 con 0x7f63101014b0 2023-11-16T16:47:53.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.140+0000 7f63181ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631019dcc0 con 0x7f6310101e40 2023-11-16T16:47:53.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.140+0000 7f63181ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631019de50 con 0x7f63101002c0 2023-11-16T16:47:53.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.141+0000 7f631579a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63101014b0 0x7f6310196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:53.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.141+0000 7f631579a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63101014b0 0x7f6310196540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33242/0 (socket says 172.21.15.2:33242) 2023-11-16T16:47:53.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.141+0000 7f631579a700 1 -- 172.21.15.2:0/2588053059 learned_addr learned my addr 172.21.15.2:0/2588053059 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:53.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.141+0000 7f6315f9b700 1 --2- 172.21.15.2:0/2588053059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63101002c0 0x7f6310196000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:53.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.141+0000 7f631679c700 1 --2- 172.21.15.2:0/2588053059 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6310101e40 0x7f631019d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:53.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.141+0000 7f631579a700 1 -- 172.21.15.2:0/2588053059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63101002c0 msgr2=0x7f6310196000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.141+0000 7f631579a700 1 --2- 172.21.15.2:0/2588053059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63101002c0 0x7f6310196000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.141+0000 7f631579a700 1 -- 172.21.15.2:0/2588053059 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6310101e40 msgr2=0x7f631019d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.141+0000 7f631579a700 1 --2- 172.21.15.2:0/2588053059 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6310101e40 0x7f631019d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.141+0000 7f631579a700 1 -- 172.21.15.2:0/2588053059 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63000095f0 con 0x7f63101014b0 2023-11-16T16:47:53.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.141+0000 7f631579a700 1 --2- 172.21.15.2:0/2588053059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63101014b0 0x7f6310196540 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6300005f50 tx=0x7f63000057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:53.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.141+0000 7f6306ffd700 1 -- 172.21.15.2:0/2588053059 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630001d070 con 0x7f63101014b0 2023-11-16T16:47:53.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.142+0000 7f6306ffd700 1 -- 172.21.15.2:0/2588053059 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6300021410 con 0x7f63101014b0 2023-11-16T16:47:53.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.142+0000 7f63181ff700 1 -- 172.21.15.2:0/2588053059 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f631010bc30 con 0x7f63101014b0 2023-11-16T16:47:53.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.142+0000 7f6306ffd700 1 -- 172.21.15.2:0/2588053059 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6300017650 con 0x7f63101014b0 2023-11-16T16:47:53.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.142+0000 7f63181ff700 1 -- 172.21.15.2:0/2588053059 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f631010baa0 con 0x7f63101014b0 2023-11-16T16:47:53.147 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.143+0000 7f63181ff700 1 -- 172.21.15.2:0/2588053059 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f631019e4d0 con 0x7f63101014b0 2023-11-16T16:47:53.147 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.145+0000 7f6306ffd700 1 -- 172.21.15.2:0/2588053059 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f6300021580 con 0x7f63101014b0 2023-11-16T16:47:53.148 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.146+0000 7f6306ffd700 1 --2- 172.21.15.2:0/2588053059 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62fc0a11d0 0x7f62fc0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:53.148 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.146+0000 7f6306ffd700 1 -- 172.21.15.2:0/2588053059 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f63000cdde0 con 0x7f63101014b0 2023-11-16T16:47:53.148 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.146+0000 7f6315f9b700 1 --2- 172.21.15.2:0/2588053059 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62fc0a11d0 0x7f62fc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:53.148 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.147+0000 7f6315f9b700 1 --2- 172.21.15.2:0/2588053059 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62fc0a11d0 0x7f62fc0a3680 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6310101310 tx=0x7f630c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:53.151 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.150+0000 7f6306ffd700 1 -- 172.21.15.2:0/2588053059 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6300097250 con 0x7f63101014b0 2023-11-16T16:47:53.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:52 smithi002 ceph-mon[128712]: pgmap v295: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s wr, 1 op/s 2023-11-16T16:47:53.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:52 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] up:active 2023-11-16T16:47:53.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:52 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3859890649' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-11-16T16:47:53.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:52 smithi002 ceph-mon[128712]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-11-16T16:47:53.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:52 smithi002 ceph-mon[128712]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-11-16T16:47:53.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:52 smithi002 ceph-mon[128712]: Cluster is now healthy 2023-11-16T16:47:53.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:52 smithi002 ceph-mon[128712]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-11-16T16:47:53.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:52 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-11-16T16:47:53.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:52 smithi002 ceph-mon[128712]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-11-16T16:47:53.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:52 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3775173864' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-11-16T16:47:53.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:52 smithi141 ceph-mon[131184]: pgmap v295: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s wr, 1 op/s 2023-11-16T16:47:53.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:52 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] up:active 2023-11-16T16:47:53.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:52 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3859890649' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-11-16T16:47:53.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:52 smithi141 ceph-mon[131184]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-11-16T16:47:53.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:52 smithi141 ceph-mon[131184]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-11-16T16:47:53.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:52 smithi141 ceph-mon[131184]: Cluster is now healthy 2023-11-16T16:47:53.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:52 smithi141 ceph-mon[131184]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-11-16T16:47:53.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:52 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-11-16T16:47:53.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:52 smithi141 ceph-mon[131184]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-11-16T16:47:53.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:52 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3775173864' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-11-16T16:47:53.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:52 smithi155 ceph-mon[131607]: pgmap v295: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s wr, 1 op/s 2023-11-16T16:47:53.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:52 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] up:active 2023-11-16T16:47:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:52 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3859890649' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-11-16T16:47:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:52 smithi155 ceph-mon[131607]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-11-16T16:47:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:52 smithi155 ceph-mon[131607]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-11-16T16:47:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:52 smithi155 ceph-mon[131607]: Cluster is now healthy 2023-11-16T16:47:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:52 smithi155 ceph-mon[131607]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-11-16T16:47:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:52 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-11-16T16:47:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:52 smithi155 ceph-mon[131607]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-11-16T16:47:53.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:52 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3775173864' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-11-16T16:47:53.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.392+0000 7f63181ff700 1 -- 172.21.15.2:0/2588053059 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f631019eb10 con 0x7f62fc0a11d0 2023-11-16T16:47:53.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.440+0000 7f6306ffd700 1 -- 172.21.15.2:0/2588053059 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f631019eb10 con 0x7f62fc0a11d0 2023-11-16T16:47:53.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.442+0000 7f63181ff700 1 -- 172.21.15.2:0/2588053059 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62fc0a11d0 msgr2=0x7f62fc0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.442+0000 7f63181ff700 1 --2- 172.21.15.2:0/2588053059 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62fc0a11d0 0x7f62fc0a3680 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6310101310 tx=0x7f630c009260 comp rx=0 tx=0).stop 2023-11-16T16:47:53.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.442+0000 7f63181ff700 1 -- 172.21.15.2:0/2588053059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63101014b0 msgr2=0x7f6310196540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.442+0000 7f63181ff700 1 --2- 172.21.15.2:0/2588053059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63101014b0 0x7f6310196540 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6300005f50 tx=0x7f63000057b0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.442+0000 7f63181ff700 1 -- 172.21.15.2:0/2588053059 shutdown_connections 2023-11-16T16:47:53.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.442+0000 7f63181ff700 1 --2- 172.21.15.2:0/2588053059 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62fc0a11d0 0x7f62fc0a3680 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.442+0000 7f63181ff700 1 --2- 172.21.15.2:0/2588053059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f63101002c0 0x7f6310196000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.442+0000 7f63181ff700 1 --2- 172.21.15.2:0/2588053059 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6310101e40 0x7f631019d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.442+0000 7f63181ff700 1 --2- 172.21.15.2:0/2588053059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63101014b0 0x7f6310196540 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.442+0000 7f63181ff700 1 -- 172.21.15.2:0/2588053059 >> 172.21.15.2:0/2588053059 conn(0x7f63100fb8a0 msgr2=0x7f63100fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:53.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.443+0000 7f63181ff700 1 -- 172.21.15.2:0/2588053059 shutdown_connections 2023-11-16T16:47:53.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.443+0000 7f63181ff700 1 -- 172.21.15.2:0/2588053059 wait complete. 2023-11-16T16:47:53.463 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_0 2023-11-16T16:47:53.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:47:53 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:47:53.399+0000 7f9047b98700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:47:53.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:47:53 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:47:53.399+0000 7f9047b98700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:47:53.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:47:53 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:47:53.399+0000 7f9047b98700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:47:53.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:47:53 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:47:53.399+0000 7f9047b98700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:47:53.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:47:53 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:47:53.399+0000 7f9047b98700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:47:53.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.669+0000 7faf365ff700 1 -- 172.21.15.2:0/1586265239 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf301014b0 msgr2=0x7faf30101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.669+0000 7faf365ff700 1 --2- 172.21.15.2:0/1586265239 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf301014b0 0x7faf30101900 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7faf24009990 tx=0x7faf24009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.670+0000 7faf365ff700 1 -- 172.21.15.2:0/1586265239 shutdown_connections 2023-11-16T16:47:53.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.670+0000 7faf365ff700 1 --2- 172.21.15.2:0/1586265239 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf30101e40 0x7faf30108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.670+0000 7faf365ff700 1 --2- 172.21.15.2:0/1586265239 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf301002c0 0x7faf301006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.670+0000 7faf365ff700 1 --2- 172.21.15.2:0/1586265239 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf301014b0 0x7faf30101900 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.670+0000 7faf365ff700 1 -- 172.21.15.2:0/1586265239 >> 172.21.15.2:0/1586265239 conn(0x7faf300fb8a0 msgr2=0x7faf300fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:53.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.670+0000 7faf365ff700 1 -- 172.21.15.2:0/1586265239 shutdown_connections 2023-11-16T16:47:53.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.670+0000 7faf365ff700 1 -- 172.21.15.2:0/1586265239 wait complete. 2023-11-16T16:47:53.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.671+0000 7faf365ff700 1 Processor -- start 2023-11-16T16:47:53.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.671+0000 7faf365ff700 1 -- start start 2023-11-16T16:47:53.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.672+0000 7faf365ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf301002c0 0x7faf30196000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:53.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.672+0000 7faf365ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf301014b0 0x7faf30196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:53.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.672+0000 7faf365ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf30101e40 0x7faf3019d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:53.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.672+0000 7faf365ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3019db80 con 0x7faf301014b0 2023-11-16T16:47:53.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.672+0000 7faf365ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3019dcc0 con 0x7faf30101e40 2023-11-16T16:47:53.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.672+0000 7faf365ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3019de50 con 0x7faf301002c0 2023-11-16T16:47:53.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.672+0000 7faf2f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf301014b0 0x7faf30196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:53.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.672+0000 7faf2f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf301014b0 0x7faf30196540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33280/0 (socket says 172.21.15.2:33280) 2023-11-16T16:47:53.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.672+0000 7faf2f7fe700 1 -- 172.21.15.2:0/3151957219 learned_addr learned my addr 172.21.15.2:0/3151957219 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:53.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.672+0000 7faf2ffff700 1 --2- 172.21.15.2:0/3151957219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf301002c0 0x7faf30196000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:53.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.672+0000 7faf34b9c700 1 --2- 172.21.15.2:0/3151957219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf30101e40 0x7faf3019d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:53.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.673+0000 7faf2f7fe700 1 -- 172.21.15.2:0/3151957219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf301002c0 msgr2=0x7faf30196000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.673+0000 7faf2f7fe700 1 --2- 172.21.15.2:0/3151957219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf301002c0 0x7faf30196000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.673+0000 7faf2f7fe700 1 -- 172.21.15.2:0/3151957219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf30101e40 msgr2=0x7faf3019d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.673+0000 7faf2f7fe700 1 --2- 172.21.15.2:0/3151957219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf30101e40 0x7faf3019d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.673+0000 7faf2f7fe700 1 -- 172.21.15.2:0/3151957219 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf20009520 con 0x7faf301014b0 2023-11-16T16:47:53.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.673+0000 7faf34b9c700 1 --2- 172.21.15.2:0/3151957219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf30101e40 0x7faf3019d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:53.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.673+0000 7faf2ffff700 1 --2- 172.21.15.2:0/3151957219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf301002c0 0x7faf30196000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:53.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.673+0000 7faf2f7fe700 1 --2- 172.21.15.2:0/3151957219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf301014b0 0x7faf30196540 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7faf24000c00 tx=0x7faf240047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:53.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.673+0000 7faf2d7fa700 1 -- 172.21.15.2:0/3151957219 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf2401d070 con 0x7faf301014b0 2023-11-16T16:47:53.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.673+0000 7faf2d7fa700 1 -- 172.21.15.2:0/3151957219 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faf24022470 con 0x7faf301014b0 2023-11-16T16:47:53.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.673+0000 7faf365ff700 1 -- 172.21.15.2:0/3151957219 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf240095f0 con 0x7faf301014b0 2023-11-16T16:47:53.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.673+0000 7faf2d7fa700 1 -- 172.21.15.2:0/3151957219 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf24017650 con 0x7faf301014b0 2023-11-16T16:47:53.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.673+0000 7faf365ff700 1 -- 172.21.15.2:0/3151957219 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf3010bc30 con 0x7faf301014b0 2023-11-16T16:47:53.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.674+0000 7faf365ff700 1 -- 172.21.15.2:0/3151957219 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf3010bdc0 con 0x7faf301014b0 2023-11-16T16:47:53.678 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.676+0000 7faf2d7fa700 1 -- 172.21.15.2:0/3151957219 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 135714+0+0 (secure 0 0 0) 0x7faf240225e0 con 0x7faf301014b0 2023-11-16T16:47:53.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.678+0000 7faf2d7fa700 1 --2- 172.21.15.2:0/3151957219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7faf180a11d0 0x7faf180a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:53.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.678+0000 7faf2d7fa700 1 -- 172.21.15.2:0/3151957219 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7faf240ce1a0 con 0x7faf301014b0 2023-11-16T16:47:53.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.678+0000 7faf2ffff700 1 --2- 172.21.15.2:0/3151957219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7faf180a11d0 0x7faf180a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:53.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.678+0000 7faf2ffff700 1 --2- 172.21.15.2:0/3151957219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7faf180a11d0 0x7faf180a3680 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faf20009fd0 tx=0x7faf20009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:53.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.681+0000 7faf2d7fa700 1 -- 172.21.15.2:0/3151957219 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7faf24097610 con 0x7faf301014b0 2023-11-16T16:47:53.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.918+0000 7faf2d7fa700 1 -- 172.21.15.2:0/3151957219 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7faf240979e0 con 0x7faf301014b0 2023-11-16T16:47:53.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.937+0000 7faf365ff700 1 -- 172.21.15.2:0/3151957219 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7faf30060f30 con 0x7faf180a11d0 2023-11-16T16:47:53.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.942+0000 7faf2d7fa700 1 -- 172.21.15.2:0/3151957219 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7faf30060f30 con 0x7faf180a11d0 2023-11-16T16:47:53.943 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv_0/f592ed5e-14f1-422a-b1bf-5606a9096610 2023-11-16T16:47:53.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.944+0000 7faf365ff700 1 -- 172.21.15.2:0/3151957219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7faf180a11d0 msgr2=0x7faf180a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.944+0000 7faf365ff700 1 --2- 172.21.15.2:0/3151957219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7faf180a11d0 0x7faf180a3680 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faf20009fd0 tx=0x7faf20009260 comp rx=0 tx=0).stop 2023-11-16T16:47:53.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.944+0000 7faf365ff700 1 -- 172.21.15.2:0/3151957219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf301014b0 msgr2=0x7faf30196540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:53.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.944+0000 7faf365ff700 1 --2- 172.21.15.2:0/3151957219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf301014b0 0x7faf30196540 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7faf24000c00 tx=0x7faf240047a0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.944+0000 7faf365ff700 1 -- 172.21.15.2:0/3151957219 shutdown_connections 2023-11-16T16:47:53.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.944+0000 7faf365ff700 1 --2- 172.21.15.2:0/3151957219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7faf180a11d0 0x7faf180a3680 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.944+0000 7faf365ff700 1 --2- 172.21.15.2:0/3151957219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faf301002c0 0x7faf30196000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.944+0000 7faf365ff700 1 --2- 172.21.15.2:0/3151957219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7faf30101e40 0x7faf3019d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.944+0000 7faf365ff700 1 --2- 172.21.15.2:0/3151957219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf301014b0 0x7faf30196540 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:53.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.944+0000 7faf365ff700 1 -- 172.21.15.2:0/3151957219 >> 172.21.15.2:0/3151957219 conn(0x7faf300fb8a0 msgr2=0x7faf300fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:53.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.944+0000 7faf365ff700 1 -- 172.21.15.2:0/3151957219 shutdown_connections 2023-11-16T16:47:53.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:53.944+0000 7faf365ff700 1 -- 172.21.15.2:0/3151957219 wait complete. 2023-11-16T16:47:53.964 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 '' 2023-11-16T16:47:54.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:53 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] up:active 2023-11-16T16:47:54.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:53 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3775173864' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-11-16T16:47:54.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:53 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-11-16T16:47:54.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:53 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:47:54.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.171+0000 7effe39ff700 1 -- 172.21.15.2:0/1016502527 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effdc1002c0 msgr2=0x7effdc1006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:54.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.171+0000 7effe39ff700 1 --2- 172.21.15.2:0/1016502527 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effdc1002c0 0x7effdc1006d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7effd8009990 tx=0x7effd8009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.171+0000 7effe39ff700 1 -- 172.21.15.2:0/1016502527 shutdown_connections 2023-11-16T16:47:54.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.171+0000 7effe39ff700 1 --2- 172.21.15.2:0/1016502527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effdc101e40 0x7effdc108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.171+0000 7effe39ff700 1 --2- 172.21.15.2:0/1016502527 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effdc1002c0 0x7effdc1006d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.171+0000 7effe39ff700 1 --2- 172.21.15.2:0/1016502527 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7effdc1014b0 0x7effdc101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.171+0000 7effe39ff700 1 -- 172.21.15.2:0/1016502527 >> 172.21.15.2:0/1016502527 conn(0x7effdc0fb8a0 msgr2=0x7effdc0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:54.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.171+0000 7effe39ff700 1 -- 172.21.15.2:0/1016502527 shutdown_connections 2023-11-16T16:47:54.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.171+0000 7effe39ff700 1 -- 172.21.15.2:0/1016502527 wait complete. 2023-11-16T16:47:54.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.172+0000 7effe39ff700 1 Processor -- start 2023-11-16T16:47:54.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.172+0000 7effe39ff700 1 -- start start 2023-11-16T16:47:54.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.173+0000 7effe39ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7effdc1002c0 0x7effdc198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:54.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.173+0000 7effe39ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effdc1014b0 0x7effdc198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:54.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.173+0000 7effe39ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effdc101e40 0x7effdc19f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:54.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.173+0000 7effe39ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc19fd90 con 0x7effdc101e40 2023-11-16T16:47:54.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.173+0000 7effe39ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc19fed0 con 0x7effdc1002c0 2023-11-16T16:47:54.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.173+0000 7effe39ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc1a0060 con 0x7effdc1014b0 2023-11-16T16:47:54.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.173+0000 7effe1f9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effdc101e40 0x7effdc19f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:54.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.173+0000 7effe0f9a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effdc1014b0 0x7effdc198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:54.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.173+0000 7effe179b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7effdc1002c0 0x7effdc198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:54.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.173+0000 7effe1f9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effdc101e40 0x7effdc19f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33308/0 (socket says 172.21.15.2:33308) 2023-11-16T16:47:54.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.173+0000 7effe1f9c700 1 -- 172.21.15.2:0/1294999814 learned_addr learned my addr 172.21.15.2:0/1294999814 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:54.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.173+0000 7effe0f9a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effdc1014b0 0x7effdc198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:47086/0 (socket says 172.21.15.2:47086) 2023-11-16T16:47:54.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.174+0000 7effe0f9a700 1 -- 172.21.15.2:0/1294999814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7effdc1002c0 msgr2=0x7effdc198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:54.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.174+0000 7effe0f9a700 1 --2- 172.21.15.2:0/1294999814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7effdc1002c0 0x7effdc198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.174+0000 7effe0f9a700 1 -- 172.21.15.2:0/1294999814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effdc101e40 msgr2=0x7effdc19f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:54.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.174+0000 7effe0f9a700 1 --2- 172.21.15.2:0/1294999814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effdc101e40 0x7effdc19f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.174+0000 7effe0f9a700 1 -- 172.21.15.2:0/1294999814 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd80095f0 con 0x7effdc1014b0 2023-11-16T16:47:54.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.174+0000 7effe179b700 1 --2- 172.21.15.2:0/1294999814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7effdc1002c0 0x7effdc198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:54.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.174+0000 7effe1f9c700 1 --2- 172.21.15.2:0/1294999814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effdc101e40 0x7effdc19f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:54.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.174+0000 7effe0f9a700 1 --2- 172.21.15.2:0/1294999814 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effdc1014b0 0x7effdc198750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7effcc00e9f0 tx=0x7effcc00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:54.177 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.174+0000 7effd27fc700 1 -- 172.21.15.2:0/1294999814 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc00c9c0 con 0x7effdc1014b0 2023-11-16T16:47:54.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.174+0000 7effe39ff700 1 -- 172.21.15.2:0/1294999814 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effdc10bc20 con 0x7effdc1014b0 2023-11-16T16:47:54.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.174+0000 7effd27fc700 1 -- 172.21.15.2:0/1294999814 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7effcc010430 con 0x7effdc1014b0 2023-11-16T16:47:54.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.174+0000 7effd27fc700 1 -- 172.21.15.2:0/1294999814 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effcc018680 con 0x7effdc1014b0 2023-11-16T16:47:54.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.174+0000 7effe39ff700 1 -- 172.21.15.2:0/1294999814 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effdc10ba90 con 0x7effdc1014b0 2023-11-16T16:47:54.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.175+0000 7effe39ff700 1 -- 172.21.15.2:0/1294999814 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effdc192980 con 0x7effdc1014b0 2023-11-16T16:47:54.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.177+0000 7effd27fc700 1 -- 172.21.15.2:0/1294999814 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7effcc010860 con 0x7effdc1014b0 2023-11-16T16:47:54.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.179+0000 7effd27fc700 1 --2- 172.21.15.2:0/1294999814 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7effc80a11e0 0x7effc80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:54.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.179+0000 7effd27fc700 1 -- 172.21.15.2:0/1294999814 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7effcc014070 con 0x7effdc1014b0 2023-11-16T16:47:54.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.179+0000 7effe179b700 1 --2- 172.21.15.2:0/1294999814 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7effc80a11e0 0x7effc80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:54.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.180+0000 7effe179b700 1 --2- 172.21.15.2:0/1294999814 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7effc80a11e0 0x7effc80a3690 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7effd800b2e0 tx=0x7effd8005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:54.182 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.181+0000 7effd27fc700 1 -- 172.21.15.2:0/1294999814 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7effcc096370 con 0x7effdc1014b0 2023-11-16T16:47:54.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:53 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] up:active 2023-11-16T16:47:54.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:53 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3775173864' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-11-16T16:47:54.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:53 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-11-16T16:47:54.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:53 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:47:54.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:53 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] up:active 2023-11-16T16:47:54.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:53 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3775173864' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-11-16T16:47:54.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:53 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-11-16T16:47:54.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:53 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:47:54.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.425+0000 7effe39ff700 1 -- 172.21.15.2:0/1294999814 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7effdc061bc0 con 0x7effc80a11e0 2023-11-16T16:47:54.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.451+0000 7effd27fc700 1 -- 172.21.15.2:0/1294999814 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7effdc061bc0 con 0x7effc80a11e0 2023-11-16T16:47:54.455 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.453+0000 7effe39ff700 1 -- 172.21.15.2:0/1294999814 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7effc80a11e0 msgr2=0x7effc80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:54.455 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.453+0000 7effe39ff700 1 --2- 172.21.15.2:0/1294999814 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7effc80a11e0 0x7effc80a3690 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7effd800b2e0 tx=0x7effd8005960 comp rx=0 tx=0).stop 2023-11-16T16:47:54.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.453+0000 7effe39ff700 1 -- 172.21.15.2:0/1294999814 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effdc1014b0 msgr2=0x7effdc198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:54.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.453+0000 7effe39ff700 1 --2- 172.21.15.2:0/1294999814 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effdc1014b0 0x7effdc198750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7effcc00e9f0 tx=0x7effcc00ed00 comp rx=0 tx=0).stop 2023-11-16T16:47:54.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.453+0000 7effe39ff700 1 -- 172.21.15.2:0/1294999814 shutdown_connections 2023-11-16T16:47:54.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.453+0000 7effe39ff700 1 --2- 172.21.15.2:0/1294999814 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7effc80a11e0 0x7effc80a3690 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.453+0000 7effe39ff700 1 --2- 172.21.15.2:0/1294999814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effdc101e40 0x7effdc19f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.453+0000 7effe39ff700 1 --2- 172.21.15.2:0/1294999814 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7effdc1002c0 0x7effdc198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.453+0000 7effe39ff700 1 --2- 172.21.15.2:0/1294999814 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effdc1014b0 0x7effdc198750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.453+0000 7effe39ff700 1 -- 172.21.15.2:0/1294999814 >> 172.21.15.2:0/1294999814 conn(0x7effdc0fb8a0 msgr2=0x7effdc106db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:54.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.453+0000 7effe39ff700 1 -- 172.21.15.2:0/1294999814 shutdown_connections 2023-11-16T16:47:54.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.453+0000 7effe39ff700 1 -- 172.21.15.2:0/1294999814 wait complete. 2023-11-16T16:47:54.474 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_1 2023-11-16T16:47:54.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.678+0000 7f2343fff700 1 -- 172.21.15.2:0/136607791 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f233c1014b0 msgr2=0x7f233c101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:54.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.678+0000 7f2343fff700 1 --2- 172.21.15.2:0/136607791 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f233c1014b0 0x7f233c101900 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f232c009990 tx=0x7f232c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.680 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.678+0000 7f2343fff700 1 -- 172.21.15.2:0/136607791 shutdown_connections 2023-11-16T16:47:54.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.678+0000 7f2343fff700 1 --2- 172.21.15.2:0/136607791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f233c101e40 0x7f233c108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.678+0000 7f2343fff700 1 --2- 172.21.15.2:0/136607791 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f233c1002c0 0x7f233c1006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.678+0000 7f2343fff700 1 --2- 172.21.15.2:0/136607791 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f233c1014b0 0x7f233c101900 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.678+0000 7f2343fff700 1 -- 172.21.15.2:0/136607791 >> 172.21.15.2:0/136607791 conn(0x7f233c0fb8a0 msgr2=0x7f233c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:54.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.678+0000 7f2343fff700 1 -- 172.21.15.2:0/136607791 shutdown_connections 2023-11-16T16:47:54.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.678+0000 7f2343fff700 1 -- 172.21.15.2:0/136607791 wait complete. 2023-11-16T16:47:54.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.679+0000 7f2343fff700 1 Processor -- start 2023-11-16T16:47:54.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.679+0000 7f2343fff700 1 -- start start 2023-11-16T16:47:54.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.680+0000 7f2343fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f233c1002c0 0x7f233c196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:54.683 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.680+0000 7f2343fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f233c1014b0 0x7f233c196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:54.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.680+0000 7f2343fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f233c101e40 0x7f233c19d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:54.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.680+0000 7f2343fff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233c19dbc0 con 0x7f233c101e40 2023-11-16T16:47:54.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.680+0000 7f2343fff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233c19dd00 con 0x7f233c1014b0 2023-11-16T16:47:54.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.680+0000 7f2343fff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233c19de90 con 0x7f233c1002c0 2023-11-16T16:47:54.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.680+0000 7f234259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f233c101e40 0x7f233c19d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:54.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.680+0000 7f234259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f233c101e40 0x7f233c19d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58808/0 (socket says 172.21.15.2:58808) 2023-11-16T16:47:54.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.680+0000 7f234259c700 1 -- 172.21.15.2:0/3208501447 learned_addr learned my addr 172.21.15.2:0/3208501447 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:54.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.680+0000 7f234159a700 1 --2- 172.21.15.2:0/3208501447 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f233c1014b0 0x7f233c196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:54.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.680+0000 7f2341d9b700 1 --2- 172.21.15.2:0/3208501447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f233c1002c0 0x7f233c196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:54.684 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.680+0000 7f234259c700 1 -- 172.21.15.2:0/3208501447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f233c1002c0 msgr2=0x7f233c196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:54.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.680+0000 7f234259c700 1 --2- 172.21.15.2:0/3208501447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f233c1002c0 0x7f233c196040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.680+0000 7f234259c700 1 -- 172.21.15.2:0/3208501447 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f233c1014b0 msgr2=0x7f233c196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:54.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.680+0000 7f234259c700 1 --2- 172.21.15.2:0/3208501447 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f233c1014b0 0x7f233c196580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.680+0000 7f234259c700 1 -- 172.21.15.2:0/3208501447 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2338009520 con 0x7f233c101e40 2023-11-16T16:47:54.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.681+0000 7f234259c700 1 --2- 172.21.15.2:0/3208501447 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f233c101e40 0x7f233c19d5f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f233400ef30 tx=0x7f233400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:54.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.681+0000 7f2332ffd700 1 -- 172.21.15.2:0/3208501447 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2334018070 con 0x7f233c101e40 2023-11-16T16:47:54.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.681+0000 7f2332ffd700 1 -- 172.21.15.2:0/3208501447 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f233400f040 con 0x7f233c101e40 2023-11-16T16:47:54.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.681+0000 7f2343fff700 1 -- 172.21.15.2:0/3208501447 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f232c0095f0 con 0x7f233c101e40 2023-11-16T16:47:54.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.681+0000 7f2332ffd700 1 -- 172.21.15.2:0/3208501447 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233400cd00 con 0x7f233c101e40 2023-11-16T16:47:54.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.681+0000 7f2343fff700 1 -- 172.21.15.2:0/3208501447 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f233c10bc70 con 0x7f233c101e40 2023-11-16T16:47:54.686 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.682+0000 7f2343fff700 1 -- 172.21.15.2:0/3208501447 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f233c1071f0 con 0x7f233c101e40 2023-11-16T16:47:54.686 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.684+0000 7f2332ffd700 1 -- 172.21.15.2:0/3208501447 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f2334003a00 con 0x7f233c101e40 2023-11-16T16:47:54.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.686+0000 7f2332ffd700 1 --2- 172.21.15.2:0/3208501447 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f23280a1260 0x7f23280a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:54.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.686+0000 7f2332ffd700 1 -- 172.21.15.2:0/3208501447 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f23340cc890 con 0x7f233c101e40 2023-11-16T16:47:54.688 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.687+0000 7f2341d9b700 1 --2- 172.21.15.2:0/3208501447 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f23280a1260 0x7f23280a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:54.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.687+0000 7f2341d9b700 1 --2- 172.21.15.2:0/3208501447 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f23280a1260 0x7f23280a3710 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f233c101310 tx=0x7f2338009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:54.692 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.691+0000 7f2332ffd700 1 -- 172.21.15.2:0/3208501447 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2334095ba0 con 0x7f233c101e40 2023-11-16T16:47:54.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.941+0000 7f2343fff700 1 -- 172.21.15.2:0/3208501447 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f233c060f30 con 0x7f23280a1260 2023-11-16T16:47:54.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.945+0000 7f2332ffd700 1 -- 172.21.15.2:0/3208501447 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f233c060f30 con 0x7f23280a1260 2023-11-16T16:47:54.947 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv_1/81f7d666-f2e8-486f-8c44-a77b2ceae0af 2023-11-16T16:47:54.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.948+0000 7f2343fff700 1 -- 172.21.15.2:0/3208501447 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f23280a1260 msgr2=0x7f23280a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:54.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.948+0000 7f2343fff700 1 --2- 172.21.15.2:0/3208501447 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f23280a1260 0x7f23280a3710 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f233c101310 tx=0x7f2338009260 comp rx=0 tx=0).stop 2023-11-16T16:47:54.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.948+0000 7f2343fff700 1 -- 172.21.15.2:0/3208501447 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f233c101e40 msgr2=0x7f233c19d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:54.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.948+0000 7f2343fff700 1 --2- 172.21.15.2:0/3208501447 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f233c101e40 0x7f233c19d5f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f233400ef30 tx=0x7f233400c3a0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.948+0000 7f2343fff700 1 -- 172.21.15.2:0/3208501447 shutdown_connections 2023-11-16T16:47:54.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.948+0000 7f2343fff700 1 --2- 172.21.15.2:0/3208501447 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f23280a1260 0x7f23280a3710 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.948+0000 7f2343fff700 1 --2- 172.21.15.2:0/3208501447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f233c1002c0 0x7f233c196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.948+0000 7f2343fff700 1 --2- 172.21.15.2:0/3208501447 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f233c101e40 0x7f233c19d5f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.948+0000 7f2343fff700 1 --2- 172.21.15.2:0/3208501447 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f233c1014b0 0x7f233c196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:54.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.948+0000 7f2343fff700 1 -- 172.21.15.2:0/3208501447 >> 172.21.15.2:0/3208501447 conn(0x7f233c0fb8a0 msgr2=0x7f233c0fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:54.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.948+0000 7f2343fff700 1 -- 172.21.15.2:0/3208501447 shutdown_connections 2023-11-16T16:47:54.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:54.948+0000 7f2343fff700 1 -- 172.21.15.2:0/3208501447 wait complete. 2023-11-16T16:47:54.969 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:47:55.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.172+0000 7f5e4a7ff700 1 -- 172.21.15.2:0/2709171881 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e44101e10 msgr2=0x7f5e441086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:55.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.172+0000 7f5e4a7ff700 1 --2- 172.21.15.2:0/2709171881 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e44101e10 0x7f5e441086d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c00b0a0 tx=0x7f5e3c00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.172+0000 7f5e4a7ff700 1 -- 172.21.15.2:0/2709171881 shutdown_connections 2023-11-16T16:47:55.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.172+0000 7f5e4a7ff700 1 --2- 172.21.15.2:0/2709171881 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e44101e10 0x7f5e441086d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.172+0000 7f5e4a7ff700 1 --2- 172.21.15.2:0/2709171881 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e44100290 0x7f5e441006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.172+0000 7f5e4a7ff700 1 --2- 172.21.15.2:0/2709171881 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e44101480 0x7f5e441018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.172+0000 7f5e4a7ff700 1 -- 172.21.15.2:0/2709171881 >> 172.21.15.2:0/2709171881 conn(0x7f5e440fb870 msgr2=0x7f5e440fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:55.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.172+0000 7f5e4a7ff700 1 -- 172.21.15.2:0/2709171881 shutdown_connections 2023-11-16T16:47:55.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.173+0000 7f5e4a7ff700 1 -- 172.21.15.2:0/2709171881 wait complete. 2023-11-16T16:47:55.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.174+0000 7f5e4a7ff700 1 Processor -- start 2023-11-16T16:47:55.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.174+0000 7f5e4a7ff700 1 -- start start 2023-11-16T16:47:55.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.174+0000 7f5e4a7ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e44100290 0x7f5e44198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:55.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.174+0000 7f5e4a7ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e44101480 0x7f5e44198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:55.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.174+0000 7f5e4a7ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e44101e10 0x7f5e4419f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:55.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.174+0000 7f5e4a7ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4419fdd0 con 0x7f5e44101480 2023-11-16T16:47:55.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.174+0000 7f5e4a7ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4419ff10 con 0x7f5e44100290 2023-11-16T16:47:55.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.174+0000 7f5e4a7ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e441a00a0 con 0x7f5e44101e10 2023-11-16T16:47:55.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.174+0000 7f5e437fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e44101480 0x7f5e44198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:55.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.174+0000 7f5e43fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e44100290 0x7f5e44198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:55.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.175+0000 7f5e43fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e44100290 0x7f5e44198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:38714/0 (socket says 172.21.15.2:38714) 2023-11-16T16:47:55.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.175+0000 7f5e437fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e44101480 0x7f5e44198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58820/0 (socket says 172.21.15.2:58820) 2023-11-16T16:47:55.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.175+0000 7f5e43fff700 1 -- 172.21.15.2:0/1531307612 learned_addr learned my addr 172.21.15.2:0/1531307612 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:55.180 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.175+0000 7f5e48d9c700 1 --2- 172.21.15.2:0/1531307612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e44101e10 0x7f5e4419f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:55.180 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.175+0000 7f5e437fe700 1 -- 172.21.15.2:0/1531307612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e44101e10 msgr2=0x7f5e4419f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:55.180 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.175+0000 7f5e437fe700 1 --2- 172.21.15.2:0/1531307612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e44101e10 0x7f5e4419f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.180 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.175+0000 7f5e437fe700 1 -- 172.21.15.2:0/1531307612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e44100290 msgr2=0x7f5e44198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:55.180 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.175+0000 7f5e437fe700 1 --2- 172.21.15.2:0/1531307612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e44100290 0x7f5e44198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.180 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.175+0000 7f5e437fe700 1 -- 172.21.15.2:0/1531307612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c009d00 con 0x7f5e44101480 2023-11-16T16:47:55.180 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.175+0000 7f5e43fff700 1 --2- 172.21.15.2:0/1531307612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e44100290 0x7f5e44198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:55.180 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.175+0000 7f5e437fe700 1 --2- 172.21.15.2:0/1531307612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e44101480 0x7f5e44198790 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5e3800d950 tx=0x7f5e3800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:55.180 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.175+0000 7f5e417fa700 1 -- 172.21.15.2:0/1531307612 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3800f890 con 0x7f5e44101480 2023-11-16T16:47:55.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.175+0000 7f5e417fa700 1 -- 172.21.15.2:0/1531307612 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e38010460 con 0x7f5e44101480 2023-11-16T16:47:55.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.175+0000 7f5e4a7ff700 1 -- 172.21.15.2:0/1531307612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e4410bc70 con 0x7f5e44101480 2023-11-16T16:47:55.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.175+0000 7f5e417fa700 1 -- 172.21.15.2:0/1531307612 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3800e610 con 0x7f5e44101480 2023-11-16T16:47:55.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.176+0000 7f5e4a7ff700 1 -- 172.21.15.2:0/1531307612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e4410bae0 con 0x7f5e44101480 2023-11-16T16:47:55.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.177+0000 7f5e4a7ff700 1 -- 172.21.15.2:0/1531307612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e441a0840 con 0x7f5e44101480 2023-11-16T16:47:55.182 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.179+0000 7f5e417fa700 1 -- 172.21.15.2:0/1531307612 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f5e380105d0 con 0x7f5e44101480 2023-11-16T16:47:55.182 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.180+0000 7f5e417fa700 1 --2- 172.21.15.2:0/1531307612 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5e2c0a1260 0x7f5e2c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:55.182 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.180+0000 7f5e417fa700 1 -- 172.21.15.2:0/1531307612 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5e380cca00 con 0x7f5e44101480 2023-11-16T16:47:55.183 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.181+0000 7f5e43fff700 1 --2- 172.21.15.2:0/1531307612 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5e2c0a1260 0x7f5e2c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:55.183 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.181+0000 7f5e43fff700 1 --2- 172.21.15.2:0/1531307612 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5e2c0a1260 0x7f5e2c0a3710 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5e34005950 tx=0x7f5e34009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:55.186 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.185+0000 7f5e417fa700 1 -- 172.21.15.2:0/1531307612 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5e38095c80 con 0x7f5e44101480 2023-11-16T16:47:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:54 smithi141 ceph-mon[131184]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:54 smithi141 ceph-mon[131184]: pgmap v296: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 823 B/s wr, 3 op/s 2023-11-16T16:47:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:54 smithi141 ceph-mon[131184]: mgrmap e17: x(active, since 7m), standbys: z, y 2023-11-16T16:47:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:54 smithi141 ceph-mon[131184]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:55.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:54 smithi155 ceph-mon[131607]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:55.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:54 smithi155 ceph-mon[131607]: pgmap v296: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 823 B/s wr, 3 op/s 2023-11-16T16:47:55.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:54 smithi155 ceph-mon[131607]: mgrmap e17: x(active, since 7m), standbys: z, y 2023-11-16T16:47:55.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:54 smithi155 ceph-mon[131607]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:55.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:54 smithi002 ceph-mon[128712]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:55.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:54 smithi002 ceph-mon[128712]: pgmap v296: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 823 B/s wr, 3 op/s 2023-11-16T16:47:55.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:54 smithi002 ceph-mon[128712]: mgrmap e17: x(active, since 7m), standbys: z, y 2023-11-16T16:47:55.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:54 smithi002 ceph-mon[128712]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:55.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.439+0000 7f5e4a7ff700 1 -- 172.21.15.2:0/1531307612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e441a0df0 con 0x7f5e44101480 2023-11-16T16:47:55.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.440+0000 7f5e417fa700 1 -- 172.21.15.2:0/1531307612 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10808 (secure 0 0 0) 0x7f5e380992a0 con 0x7f5e44101480 2023-11-16T16:47:55.442 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:55.442 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/1610211225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":1610211225},{"type":"v1","addr":"172.21.15.2:6837","nonce":1610211225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14538,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2583261818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2583261818},{"type":"v1","addr":"172.21.15.2:6839","nonce":2583261818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14550,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3170991923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3170991923},{"type":"v1","addr":"172.21.15.2:6841","nonce":3170991923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:26.349176+0000","modified":"2023-11-16T16:47:52.901168+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24544,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.141:6835/3516869188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3516869188},{"type":"v1","addr":"172.21.15.141:6835","nonce":3516869188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/2527313191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":2527313191},{"type":"v1","addr":"172.21.15.155:6835","nonce":2527313191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24523":{"gid":24523,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1607756751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1607756751},{"type":"v1","addr":"172.21.15.155:6837","nonce":1607756751}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.141:6837/2934891475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6836","nonce":2934891475},{"type":"v1","addr":"172.21.15.141:6837","nonce":2934891475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24544":{"gid":24544,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1158457009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1158457009},{"type":"v1","addr":"172.21.15.155:6839","nonce":1158457009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6839/3916358272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6838","nonce":3916358272},{"type":"v1","addr":"172.21.15.141:6839","nonce":3916358272}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:47:55.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.442+0000 7f5e4a7ff700 1 -- 172.21.15.2:0/1531307612 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5e2c0a1260 msgr2=0x7f5e2c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:55.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.442+0000 7f5e4a7ff700 1 --2- 172.21.15.2:0/1531307612 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5e2c0a1260 0x7f5e2c0a3710 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5e34005950 tx=0x7f5e34009260 comp rx=0 tx=0).stop 2023-11-16T16:47:55.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.442+0000 7f5e4a7ff700 1 -- 172.21.15.2:0/1531307612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e44101480 msgr2=0x7f5e44198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:55.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.442+0000 7f5e4a7ff700 1 --2- 172.21.15.2:0/1531307612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e44101480 0x7f5e44198790 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5e3800d950 tx=0x7f5e3800dc60 comp rx=0 tx=0).stop 2023-11-16T16:47:55.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.442+0000 7f5e4a7ff700 1 -- 172.21.15.2:0/1531307612 shutdown_connections 2023-11-16T16:47:55.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.442+0000 7f5e4a7ff700 1 --2- 172.21.15.2:0/1531307612 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5e2c0a1260 0x7f5e2c0a3710 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.442+0000 7f5e4a7ff700 1 --2- 172.21.15.2:0/1531307612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e44101e10 0x7f5e4419f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.442+0000 7f5e4a7ff700 1 --2- 172.21.15.2:0/1531307612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e44100290 0x7f5e44198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.442+0000 7f5e4a7ff700 1 --2- 172.21.15.2:0/1531307612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e44101480 0x7f5e44198790 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.442+0000 7f5e4a7ff700 1 -- 172.21.15.2:0/1531307612 >> 172.21.15.2:0/1531307612 conn(0x7f5e440fb870 msgr2=0x7f5e44106d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:55.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.442+0000 7f5e4a7ff700 1 -- 172.21.15.2:0/1531307612 shutdown_connections 2023-11-16T16:47:55.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.442+0000 7f5e4a7ff700 1 -- 172.21.15.2:0/1531307612 wait complete. 2023-11-16T16:47:55.446 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2023-11-16T16:47:55.463 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:47:55.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.666+0000 7f683319d700 1 -- 172.21.15.2:0/4178864875 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f682c101cf0 msgr2=0x7f682c108590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:55.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.666+0000 7f683319d700 1 --2- 172.21.15.2:0/4178864875 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f682c101cf0 0x7f682c108590 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f682400b0a0 tx=0x7f682400b3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.666+0000 7f683319d700 1 -- 172.21.15.2:0/4178864875 shutdown_connections 2023-11-16T16:47:55.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.666+0000 7f683319d700 1 --2- 172.21.15.2:0/4178864875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f682c1000b0 0x7f682c1004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.666+0000 7f683319d700 1 --2- 172.21.15.2:0/4178864875 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f682c101cf0 0x7f682c108590 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.666+0000 7f683319d700 1 --2- 172.21.15.2:0/4178864875 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f682c101340 0x7f682c1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.666+0000 7f683319d700 1 -- 172.21.15.2:0/4178864875 >> 172.21.15.2:0/4178864875 conn(0x7f682c0fb6d0 msgr2=0x7f682c0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:55.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.666+0000 7f683319d700 1 -- 172.21.15.2:0/4178864875 shutdown_connections 2023-11-16T16:47:55.670 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.666+0000 7f683319d700 1 -- 172.21.15.2:0/4178864875 wait complete. 2023-11-16T16:47:55.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.667+0000 7f683319d700 1 Processor -- start 2023-11-16T16:47:55.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.668+0000 7f683319d700 1 -- start start 2023-11-16T16:47:55.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.668+0000 7f683319d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f682c1000b0 0x7f682c0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:55.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.668+0000 7f683319d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f682c101340 0x7f682c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:55.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.668+0000 7f683319d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f682c101cf0 0x7f682c073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:55.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.668+0000 7f683319d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682c073bc0 con 0x7f682c101340 2023-11-16T16:47:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.668+0000 7f683319d700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682c073d30 con 0x7f682c1000b0 2023-11-16T16:47:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.668+0000 7f683319d700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682c19ff20 con 0x7f682c101cf0 2023-11-16T16:47:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.668+0000 7f683199a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f682c101340 0x7f682c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.668+0000 7f683199a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f682c101340 0x7f682c072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58846/0 (socket says 172.21.15.2:58846) 2023-11-16T16:47:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.668+0000 7f683199a700 1 -- 172.21.15.2:0/36649579 learned_addr learned my addr 172.21.15.2:0/36649579 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.668+0000 7f683299c700 1 --2- 172.21.15.2:0/36649579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f682c101cf0 0x7f682c073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.668+0000 7f683219b700 1 --2- 172.21.15.2:0/36649579 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f682c1000b0 0x7f682c0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.669+0000 7f683199a700 1 -- 172.21.15.2:0/36649579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f682c101cf0 msgr2=0x7f682c073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.669+0000 7f683199a700 1 --2- 172.21.15.2:0/36649579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f682c101cf0 0x7f682c073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.669+0000 7f683199a700 1 -- 172.21.15.2:0/36649579 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f682c1000b0 msgr2=0x7f682c0748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.669+0000 7f683199a700 1 --2- 172.21.15.2:0/36649579 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f682c1000b0 0x7f682c0748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.669+0000 7f683199a700 1 -- 172.21.15.2:0/36649579 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6824009d00 con 0x7f682c101340 2023-11-16T16:47:55.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.669+0000 7f683299c700 1 --2- 172.21.15.2:0/36649579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f682c101cf0 0x7f682c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:55.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.669+0000 7f683199a700 1 --2- 172.21.15.2:0/36649579 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f682c101340 0x7f682c072ee0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f681c00e9a0 tx=0x7f681c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:55.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.669+0000 7f68237fe700 1 -- 172.21.15.2:0/36649579 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c009790 con 0x7f682c101340 2023-11-16T16:47:55.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.669+0000 7f68237fe700 1 -- 172.21.15.2:0/36649579 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f681c010430 con 0x7f682c101340 2023-11-16T16:47:55.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.669+0000 7f68237fe700 1 -- 172.21.15.2:0/36649579 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c018680 con 0x7f682c101340 2023-11-16T16:47:55.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.669+0000 7f683319d700 1 -- 172.21.15.2:0/36649579 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f682c10bb00 con 0x7f682c101340 2023-11-16T16:47:55.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.669+0000 7f683319d700 1 -- 172.21.15.2:0/36649579 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f682c10b970 con 0x7f682c101340 2023-11-16T16:47:55.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.670+0000 7f683319d700 1 -- 172.21.15.2:0/36649579 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f682c107640 con 0x7f682c101340 2023-11-16T16:47:55.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.672+0000 7f68237fe700 1 -- 172.21.15.2:0/36649579 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f681c0105a0 con 0x7f682c101340 2023-11-16T16:47:55.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.674+0000 7f68237fe700 1 --2- 172.21.15.2:0/36649579 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f68180a1260 0x7f68180a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:55.676 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.675+0000 7f68237fe700 1 -- 172.21.15.2:0/36649579 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f681c014070 con 0x7f682c101340 2023-11-16T16:47:55.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.675+0000 7f683219b700 1 --2- 172.21.15.2:0/36649579 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f68180a1260 0x7f68180a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:55.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.675+0000 7f683219b700 1 --2- 172.21.15.2:0/36649579 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f68180a1260 0x7f68180a3710 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6828009c60 tx=0x7f6828009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:55.679 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.677+0000 7f68237fe700 1 -- 172.21.15.2:0/36649579 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f681c096220 con 0x7f682c101340 2023-11-16T16:47:55.893 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.891+0000 7f683319d700 1 -- 172.21.15.2:0/36649579 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f682c061700 con 0x7f682c101340 2023-11-16T16:47:55.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.893+0000 7f68237fe700 1 -- 172.21.15.2:0/36649579 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24379 (secure 0 0 0) 0x7f681c099840 con 0x7f682c101340 2023-11-16T16:47:55.895 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:55.895 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":91,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:47:44.849645+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:24.458452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:25.786005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:29.313181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:47:55.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.895+0000 7f683319d700 1 -- 172.21.15.2:0/36649579 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f68180a1260 msgr2=0x7f68180a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:55.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.895+0000 7f683319d700 1 --2- 172.21.15.2:0/36649579 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f68180a1260 0x7f68180a3710 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6828009c60 tx=0x7f6828009260 comp rx=0 tx=0).stop 2023-11-16T16:47:55.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.895+0000 7f683319d700 1 -- 172.21.15.2:0/36649579 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f682c101340 msgr2=0x7f682c072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:55.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.895+0000 7f683319d700 1 --2- 172.21.15.2:0/36649579 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f682c101340 0x7f682c072ee0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f681c00e9a0 tx=0x7f681c00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.895+0000 7f683319d700 1 -- 172.21.15.2:0/36649579 shutdown_connections 2023-11-16T16:47:55.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.895+0000 7f683319d700 1 --2- 172.21.15.2:0/36649579 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f68180a1260 0x7f68180a3710 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.895+0000 7f683319d700 1 --2- 172.21.15.2:0/36649579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f682c101cf0 0x7f682c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.895+0000 7f683319d700 1 --2- 172.21.15.2:0/36649579 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f682c1000b0 0x7f682c0748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.895+0000 7f683319d700 1 --2- 172.21.15.2:0/36649579 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f682c101340 0x7f682c072ee0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:55.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.895+0000 7f683319d700 1 -- 172.21.15.2:0/36649579 >> 172.21.15.2:0/36649579 conn(0x7f682c0fb6d0 msgr2=0x7f682c104780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:55.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.895+0000 7f683319d700 1 -- 172.21.15.2:0/36649579 shutdown_connections 2023-11-16T16:47:55.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:55.895+0000 7f683319d700 1 -- 172.21.15.2:0/36649579 wait complete. 2023-11-16T16:47:55.917 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-16T16:47:55.917 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T16:47:56.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.119+0000 7f2ce1bff700 1 -- 172.21.15.2:0/1825882969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cdc101480 msgr2=0x7f2cdc1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:56.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.119+0000 7f2ce1bff700 1 --2- 172.21.15.2:0/1825882969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cdc101480 0x7f2cdc1018d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0009990 tx=0x7f2cd0009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.119+0000 7f2ce1bff700 1 -- 172.21.15.2:0/1825882969 shutdown_connections 2023-11-16T16:47:56.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.119+0000 7f2ce1bff700 1 --2- 172.21.15.2:0/1825882969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2cdc100290 0x7f2cdc1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.119+0000 7f2ce1bff700 1 --2- 172.21.15.2:0/1825882969 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cdc101e10 0x7f2cdc1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.119+0000 7f2ce1bff700 1 --2- 172.21.15.2:0/1825882969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cdc101480 0x7f2cdc1018d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.119+0000 7f2ce1bff700 1 -- 172.21.15.2:0/1825882969 >> 172.21.15.2:0/1825882969 conn(0x7f2cdc0fb870 msgr2=0x7f2cdc0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:56.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.119+0000 7f2ce1bff700 1 -- 172.21.15.2:0/1825882969 shutdown_connections 2023-11-16T16:47:56.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.119+0000 7f2ce1bff700 1 -- 172.21.15.2:0/1825882969 wait complete. 2023-11-16T16:47:56.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.120+0000 7f2ce1bff700 1 Processor -- start 2023-11-16T16:47:56.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.121+0000 7f2ce1bff700 1 -- start start 2023-11-16T16:47:56.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.121+0000 7f2ce1bff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2cdc100290 0x7f2cdc195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:56.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.121+0000 7f2ce1bff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cdc101480 0x7f2cdc1964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:56.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.121+0000 7f2ce1bff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cdc101e10 0x7f2cdc19d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:56.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.121+0000 7f2ce1bff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc19db10 con 0x7f2cdc101480 2023-11-16T16:47:56.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.121+0000 7f2ce1bff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc19dc50 con 0x7f2cdc101e10 2023-11-16T16:47:56.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.121+0000 7f2ce1bff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc19dde0 con 0x7f2cdc100290 2023-11-16T16:47:56.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.121+0000 7f2cdb7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2cdc100290 0x7f2cdc195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:56.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.121+0000 7f2cdaffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cdc101480 0x7f2cdc1964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:56.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.121+0000 7f2cdb7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2cdc100290 0x7f2cdc195f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:46956/0 (socket says 172.21.15.2:46956) 2023-11-16T16:47:56.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.121+0000 7f2cdb7fe700 1 -- 172.21.15.2:0/76141169 learned_addr learned my addr 172.21.15.2:0/76141169 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:56.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.121+0000 7f2cdbfff700 1 --2- 172.21.15.2:0/76141169 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cdc101e10 0x7f2cdc19d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:56.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.121+0000 7f2cdaffd700 1 -- 172.21.15.2:0/76141169 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2cdc100290 msgr2=0x7f2cdc195f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:56.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.122+0000 7f2cdaffd700 1 --2- 172.21.15.2:0/76141169 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2cdc100290 0x7f2cdc195f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.122+0000 7f2cdaffd700 1 -- 172.21.15.2:0/76141169 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cdc101e10 msgr2=0x7f2cdc19d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:56.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.122+0000 7f2cdaffd700 1 --2- 172.21.15.2:0/76141169 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cdc101e10 0x7f2cdc19d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.122+0000 7f2cdaffd700 1 -- 172.21.15.2:0/76141169 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd00095f0 con 0x7f2cdc101480 2023-11-16T16:47:56.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.122+0000 7f2cdbfff700 1 --2- 172.21.15.2:0/76141169 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cdc101e10 0x7f2cdc19d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:47:56.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.122+0000 7f2cdaffd700 1 --2- 172.21.15.2:0/76141169 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cdc101480 0x7f2cdc1964d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0009990 tx=0x7f2cd0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:56.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.122+0000 7f2cd8ff9700 1 -- 172.21.15.2:0/76141169 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd001d070 con 0x7f2cdc101480 2023-11-16T16:47:56.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.122+0000 7f2cd8ff9700 1 -- 172.21.15.2:0/76141169 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2cd0004020 con 0x7f2cdc101480 2023-11-16T16:47:56.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.122+0000 7f2ce1bff700 1 -- 172.21.15.2:0/76141169 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cdc10bc50 con 0x7f2cdc101480 2023-11-16T16:47:56.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.122+0000 7f2cd8ff9700 1 -- 172.21.15.2:0/76141169 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0022c70 con 0x7f2cdc101480 2023-11-16T16:47:56.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.122+0000 7f2ce1bff700 1 -- 172.21.15.2:0/76141169 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cdc10bac0 con 0x7f2cdc101480 2023-11-16T16:47:56.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.123+0000 7f2ce1bff700 1 -- 172.21.15.2:0/76141169 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cdc067a60 con 0x7f2cdc101480 2023-11-16T16:47:56.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.125+0000 7f2cd8ff9700 1 -- 172.21.15.2:0/76141169 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f2cd0003b60 con 0x7f2cdc101480 2023-11-16T16:47:56.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.127+0000 7f2cd8ff9700 1 --2- 172.21.15.2:0/76141169 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2cc40a1260 0x7f2cc40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:56.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.127+0000 7f2cd8ff9700 1 -- 172.21.15.2:0/76141169 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2cd00ce2a0 con 0x7f2cdc101480 2023-11-16T16:47:56.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.127+0000 7f2cdb7fe700 1 --2- 172.21.15.2:0/76141169 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2cc40a1260 0x7f2cc40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:56.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.127+0000 7f2cdb7fe700 1 --2- 172.21.15.2:0/76141169 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2cc40a1260 0x7f2cc40a3710 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00bd70 tx=0x7f2ccc00b4a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:56.132 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.130+0000 7f2cd8ff9700 1 -- 172.21.15.2:0/76141169 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2cd00975b0 con 0x7f2cdc101480 2023-11-16T16:47:56.265 INFO:journalctl@ceph.mds.b.smithi141.stdout:Nov 16 16:47:56 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-b[156414]: 2023-11-16T16:47:56.081+0000 7fdde4a6c700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-11-16T16:47:56.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:55 smithi141 ceph-mon[131184]: from='client.24832 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:56.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:55 smithi141 ceph-mon[131184]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:56.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:55 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1531307612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:56.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:55 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/36649579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:56.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.335+0000 7f2ce1bff700 1 -- 172.21.15.2:0/76141169 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2cdc060f30 con 0x7f2cc40a1260 2023-11-16T16:47:56.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.351+0000 7f2cd8ff9700 1 -- 172.21.15.2:0/76141169 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422659 (secure 0 0 0) 0x7f2cdc060f30 con 0x7f2cc40a1260 2023-11-16T16:47:56.361 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:56.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.356+0000 7f2ce1bff700 1 -- 172.21.15.2:0/76141169 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2cc40a1260 msgr2=0x7f2cc40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:56.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.356+0000 7f2ce1bff700 1 --2- 172.21.15.2:0/76141169 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2cc40a1260 0x7f2cc40a3710 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00bd70 tx=0x7f2ccc00b4a0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.356+0000 7f2ce1bff700 1 -- 172.21.15.2:0/76141169 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cdc101480 msgr2=0x7f2cdc1964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:56.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.356+0000 7f2ce1bff700 1 --2- 172.21.15.2:0/76141169 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cdc101480 0x7f2cdc1964d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0009990 tx=0x7f2cd0009580 comp rx=0 tx=0).stop 2023-11-16T16:47:56.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.357+0000 7f2ce1bff700 1 -- 172.21.15.2:0/76141169 shutdown_connections 2023-11-16T16:47:56.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.357+0000 7f2ce1bff700 1 --2- 172.21.15.2:0/76141169 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2cc40a1260 0x7f2cc40a3710 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.357+0000 7f2ce1bff700 1 --2- 172.21.15.2:0/76141169 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2cdc100290 0x7f2cdc195f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.357+0000 7f2ce1bff700 1 --2- 172.21.15.2:0/76141169 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2cdc101e10 0x7f2cdc19d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.357+0000 7f2ce1bff700 1 --2- 172.21.15.2:0/76141169 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cdc101480 0x7f2cdc1964d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.357+0000 7f2ce1bff700 1 -- 172.21.15.2:0/76141169 >> 172.21.15.2:0/76141169 conn(0x7f2cdc0fb870 msgr2=0x7f2cdc0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:56.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.357+0000 7f2ce1bff700 1 -- 172.21.15.2:0/76141169 shutdown_connections 2023-11-16T16:47:56.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.357+0000 7f2ce1bff700 1 -- 172.21.15.2:0/76141169 wait complete. 2023-11-16T16:47:56.369 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-16T16:47:56.370 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:55 smithi155 ceph-mon[131607]: from='client.24832 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:56.370 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:55 smithi155 ceph-mon[131607]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:56.370 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:55 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1531307612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:56.370 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:55 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/36649579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:56.375 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:55 smithi002 ceph-mon[128712]: from='client.24832 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:56.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:55 smithi002 ceph-mon[128712]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:47:56.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:55 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1531307612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:47:56.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:55 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/36649579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:47:56.378 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2023-11-16T16:47:55.793582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":615881,"num_objects":55,"num_object_clones":0,"num_object_copies":167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":164,"num_read_kb":160,"num_write":304,"num_write_kb":2995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":245,"ondisk_log_size":245,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":80952,"kb_used_data":11928,"kb_used_omap":0,"kb_used_meta":68928,"kb_avail":1124860872,"statfs":{"total":1151940427776,"available":1151857532928,"internally_reserved":0,"allocated":12214272,"data_stored":7776827,"data_compressed":320145,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":0,"internal_metadata":70582272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21062,"num_objects":31,"num_object_clones":0,"num_object_copies":95,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":28,"num_read_kb":41,"num_write":75,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.941846"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472424+0000","last_change":"2023-11-16T16:47:26.367598+0000","last_active":"2023-11-16T16:47:45.472424+0000","last_peered":"2023-11-16T16:47:45.472424+0000","last_clean":"2023-11-16T16:47:45.472424+0000","last_became_active":"2023-11-16T16:47:26.366233+0000","last_became_peered":"2023-11-16T16:47:26.366233+0000","last_unstale":"2023-11-16T16:47:45.472424+0000","last_undegraded":"2023-11-16T16:47:45.472424+0000","last_fullsized":"2023-11-16T16:47:45.472424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:27:47.751645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.861310+0000","last_change":"2023-11-16T16:47:26.371984+0000","last_active":"2023-11-16T16:47:44.861310+0000","last_peered":"2023-11-16T16:47:44.861310+0000","last_clean":"2023-11-16T16:47:44.861310+0000","last_became_active":"2023-11-16T16:47:26.371628+0000","last_became_peered":"2023-11-16T16:47:26.371628+0000","last_unstale":"2023-11-16T16:47:44.861310+0000","last_undegraded":"2023-11-16T16:47:44.861310+0000","last_fullsized":"2023-11-16T16:47:44.861310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:07:37.389666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.860205+0000","last_change":"2023-11-16T16:47:26.374109+0000","last_active":"2023-11-16T16:47:44.860205+0000","last_peered":"2023-11-16T16:47:44.860205+0000","last_clean":"2023-11-16T16:47:44.860205+0000","last_became_active":"2023-11-16T16:47:26.373759+0000","last_became_peered":"2023-11-16T16:47:26.373759+0000","last_unstale":"2023-11-16T16:47:44.860205+0000","last_undegraded":"2023-11-16T16:47:44.860205+0000","last_fullsized":"2023-11-16T16:47:44.860205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:38:53.188578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855484+0000","last_change":"2023-11-16T16:47:26.378404+0000","last_active":"2023-11-16T16:47:44.855484+0000","last_peered":"2023-11-16T16:47:44.855484+0000","last_clean":"2023-11-16T16:47:44.855484+0000","last_became_active":"2023-11-16T16:47:26.378067+0000","last_became_peered":"2023-11-16T16:47:26.378067+0000","last_unstale":"2023-11-16T16:47:44.855484+0000","last_undegraded":"2023-11-16T16:47:44.855484+0000","last_fullsized":"2023-11-16T16:47:44.855484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:58:13.078678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856409+0000","last_change":"2023-11-16T16:47:26.370066+0000","last_active":"2023-11-16T16:47:44.856409+0000","last_peered":"2023-11-16T16:47:44.856409+0000","last_clean":"2023-11-16T16:47:44.856409+0000","last_became_active":"2023-11-16T16:47:26.369789+0000","last_became_peered":"2023-11-16T16:47:26.369789+0000","last_unstale":"2023-11-16T16:47:44.856409+0000","last_undegraded":"2023-11-16T16:47:44.856409+0000","last_fullsized":"2023-11-16T16:47:44.856409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:33:38.139992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472727+0000","last_change":"2023-11-16T16:47:26.379403+0000","last_active":"2023-11-16T16:47:45.472727+0000","last_peered":"2023-11-16T16:47:45.472727+0000","last_clean":"2023-11-16T16:47:45.472727+0000","last_became_active":"2023-11-16T16:47:26.379086+0000","last_became_peered":"2023-11-16T16:47:26.379086+0000","last_unstale":"2023-11-16T16:47:45.472727+0000","last_undegraded":"2023-11-16T16:47:45.472727+0000","last_fullsized":"2023-11-16T16:47:45.472727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:20:51.896082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857929+0000","last_change":"2023-11-16T16:47:26.377555+0000","last_active":"2023-11-16T16:47:44.857929+0000","last_peered":"2023-11-16T16:47:44.857929+0000","last_clean":"2023-11-16T16:47:44.857929+0000","last_became_active":"2023-11-16T16:47:26.377274+0000","last_became_peered":"2023-11-16T16:47:26.377274+0000","last_unstale":"2023-11-16T16:47:44.857929+0000","last_undegraded":"2023-11-16T16:47:44.857929+0000","last_fullsized":"2023-11-16T16:47:44.857929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:40:11.254785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.412630+0000","last_change":"2023-11-16T16:47:26.380842+0000","last_active":"2023-11-16T16:47:45.412630+0000","last_peered":"2023-11-16T16:47:45.412630+0000","last_clean":"2023-11-16T16:47:45.412630+0000","last_became_active":"2023-11-16T16:47:26.380527+0000","last_became_peered":"2023-11-16T16:47:26.380527+0000","last_unstale":"2023-11-16T16:47:45.412630+0000","last_undegraded":"2023-11-16T16:47:45.412630+0000","last_fullsized":"2023-11-16T16:47:45.412630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:56:19.663855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858099+0000","last_change":"2023-11-16T16:47:26.371416+0000","last_active":"2023-11-16T16:47:44.858099+0000","last_peered":"2023-11-16T16:47:44.858099+0000","last_clean":"2023-11-16T16:47:44.858099+0000","last_became_active":"2023-11-16T16:47:26.371096+0000","last_became_peered":"2023-11-16T16:47:26.371096+0000","last_unstale":"2023-11-16T16:47:44.858099+0000","last_undegraded":"2023-11-16T16:47:44.858099+0000","last_fullsized":"2023-11-16T16:47:44.858099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:35:21.284705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473525+0000","last_change":"2023-11-16T16:47:26.380828+0000","last_active":"2023-11-16T16:47:45.473525+0000","last_peered":"2023-11-16T16:47:45.473525+0000","last_clean":"2023-11-16T16:47:45.473525+0000","last_became_active":"2023-11-16T16:47:26.377495+0000","last_became_peered":"2023-11-16T16:47:26.377495+0000","last_unstale":"2023-11-16T16:47:45.473525+0000","last_undegraded":"2023-11-16T16:47:45.473525+0000","last_fullsized":"2023-11-16T16:47:45.473525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:58:53.606878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.859142+0000","last_change":"2023-11-16T16:47:26.374127+0000","last_active":"2023-11-16T16:47:44.859142+0000","last_peered":"2023-11-16T16:47:44.859142+0000","last_clean":"2023-11-16T16:47:44.859142+0000","last_became_active":"2023-11-16T16:47:26.373794+0000","last_became_peered":"2023-11-16T16:47:26.373794+0000","last_unstale":"2023-11-16T16:47:44.859142+0000","last_undegraded":"2023-11-16T16:47:44.859142+0000","last_fullsized":"2023-11-16T16:47:44.859142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:39:54.776347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855500+0000","last_change":"2023-11-16T16:47:26.382530+0000","last_active":"2023-11-16T16:47:44.855500+0000","last_peered":"2023-11-16T16:47:44.855500+0000","last_clean":"2023-11-16T16:47:44.855500+0000","last_became_active":"2023-11-16T16:47:26.382229+0000","last_became_peered":"2023-11-16T16:47:26.382229+0000","last_unstale":"2023-11-16T16:47:44.855500+0000","last_undegraded":"2023-11-16T16:47:44.855500+0000","last_fullsized":"2023-11-16T16:47:44.855500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:28:54.780902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473174+0000","last_change":"2023-11-16T16:47:45.473174+0000","last_active":"2023-11-16T16:47:45.473174+0000","last_peered":"2023-11-16T16:47:45.473174+0000","last_clean":"2023-11-16T16:47:45.473174+0000","last_became_active":"2023-11-16T16:47:45.472853+0000","last_became_peered":"2023-11-16T16:47:45.472853+0000","last_unstale":"2023-11-16T16:47:45.473174+0000","last_undegraded":"2023-11-16T16:47:45.473174+0000","last_fullsized":"2023-11-16T16:47:45.473174+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:57:05.934853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.859123+0000","last_change":"2023-11-16T16:47:26.375546+0000","last_active":"2023-11-16T16:47:44.859123+0000","last_peered":"2023-11-16T16:47:44.859123+0000","last_clean":"2023-11-16T16:47:44.859123+0000","last_became_active":"2023-11-16T16:47:26.375218+0000","last_became_peered":"2023-11-16T16:47:26.375218+0000","last_unstale":"2023-11-16T16:47:44.859123+0000","last_undegraded":"2023-11-16T16:47:44.859123+0000","last_fullsized":"2023-11-16T16:47:44.859123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:27:43.795280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.860204+0000","last_change":"2023-11-16T16:47:44.860204+0000","last_active":"2023-11-16T16:47:44.860204+0000","last_peered":"2023-11-16T16:47:44.860204+0000","last_clean":"2023-11-16T16:47:44.860204+0000","last_became_active":"2023-11-16T16:47:44.859473+0000","last_became_peered":"2023-11-16T16:47:44.859473+0000","last_unstale":"2023-11-16T16:47:44.860204+0000","last_undegraded":"2023-11-16T16:47:44.860204+0000","last_fullsized":"2023-11-16T16:47:44.860204+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:57:23.491160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857085+0000","last_change":"2023-11-16T16:47:26.379604+0000","last_active":"2023-11-16T16:47:44.857085+0000","last_peered":"2023-11-16T16:47:44.857085+0000","last_clean":"2023-11-16T16:47:44.857085+0000","last_became_active":"2023-11-16T16:47:26.379225+0000","last_became_peered":"2023-11-16T16:47:26.379225+0000","last_unstale":"2023-11-16T16:47:44.857085+0000","last_undegraded":"2023-11-16T16:47:44.857085+0000","last_fullsized":"2023-11-16T16:47:44.857085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:26:54.929229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472135+0000","last_change":"2023-11-16T16:47:26.367373+0000","last_active":"2023-11-16T16:47:45.472135+0000","last_peered":"2023-11-16T16:47:45.472135+0000","last_clean":"2023-11-16T16:47:45.472135+0000","last_became_active":"2023-11-16T16:47:26.366549+0000","last_became_peered":"2023-11-16T16:47:26.366549+0000","last_unstale":"2023-11-16T16:47:45.472135+0000","last_undegraded":"2023-11-16T16:47:45.472135+0000","last_fullsized":"2023-11-16T16:47:45.472135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:54:59.079284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854006+0000","last_change":"2023-11-16T16:47:26.376291+0000","last_active":"2023-11-16T16:47:44.854006+0000","last_peered":"2023-11-16T16:47:44.854006+0000","last_clean":"2023-11-16T16:47:44.854006+0000","last_became_active":"2023-11-16T16:47:26.374608+0000","last_became_peered":"2023-11-16T16:47:26.374608+0000","last_unstale":"2023-11-16T16:47:44.854006+0000","last_undegraded":"2023-11-16T16:47:44.854006+0000","last_fullsized":"2023-11-16T16:47:44.854006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:17:25.971629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855424+0000","last_change":"2023-11-16T16:47:26.382444+0000","last_active":"2023-11-16T16:47:44.855424+0000","last_peered":"2023-11-16T16:47:44.855424+0000","last_clean":"2023-11-16T16:47:44.855424+0000","last_became_active":"2023-11-16T16:47:26.382130+0000","last_became_peered":"2023-11-16T16:47:26.382130+0000","last_unstale":"2023-11-16T16:47:44.855424+0000","last_undegraded":"2023-11-16T16:47:44.855424+0000","last_fullsized":"2023-11-16T16:47:44.855424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:34.208412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854988+0000","last_change":"2023-11-16T16:47:30.838569+0000","last_active":"2023-11-16T16:47:44.854988+0000","last_peered":"2023-11-16T16:47:44.854988+0000","last_clean":"2023-11-16T16:47:44.854988+0000","last_became_active":"2023-11-16T16:47:30.838282+0000","last_became_peered":"2023-11-16T16:47:30.838282+0000","last_unstale":"2023-11-16T16:47:44.854988+0000","last_undegraded":"2023-11-16T16:47:44.854988+0000","last_fullsized":"2023-11-16T16:47:44.854988+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:06:09.584975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.907867+0000","last_change":"2023-11-16T16:47:25.820254+0000","last_active":"2023-11-16T16:47:51.907867+0000","last_peered":"2023-11-16T16:47:51.907867+0000","last_clean":"2023-11-16T16:47:51.907867+0000","last_became_active":"2023-11-16T16:47:25.819924+0000","last_became_peered":"2023-11-16T16:47:25.819924+0000","last_unstale":"2023-11-16T16:47:51.907867+0000","last_undegraded":"2023-11-16T16:47:51.907867+0000","last_fullsized":"2023-11-16T16:47:51.907867+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:50:37.572954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.413299+0000","last_change":"2023-11-16T16:47:26.374560+0000","last_active":"2023-11-16T16:47:45.413299+0000","last_peered":"2023-11-16T16:47:45.413299+0000","last_clean":"2023-11-16T16:47:45.413299+0000","last_became_active":"2023-11-16T16:47:26.373554+0000","last_became_peered":"2023-11-16T16:47:26.373554+0000","last_unstale":"2023-11-16T16:47:45.413299+0000","last_undegraded":"2023-11-16T16:47:45.413299+0000","last_fullsized":"2023-11-16T16:47:45.413299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:33:15.911693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854083+0000","last_change":"2023-11-16T16:47:30.826047+0000","last_active":"2023-11-16T16:47:44.854083+0000","last_peered":"2023-11-16T16:47:44.854083+0000","last_clean":"2023-11-16T16:47:44.854083+0000","last_became_active":"2023-11-16T16:47:30.825696+0000","last_became_peered":"2023-11-16T16:47:30.825696+0000","last_unstale":"2023-11-16T16:47:44.854083+0000","last_undegraded":"2023-11-16T16:47:44.854083+0000","last_fullsized":"2023-11-16T16:47:44.854083+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:31:58.343449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.413521+0000","last_change":"2023-11-16T16:47:25.830763+0000","last_active":"2023-11-16T16:47:45.413521+0000","last_peered":"2023-11-16T16:47:45.413521+0000","last_clean":"2023-11-16T16:47:45.413521+0000","last_became_active":"2023-11-16T16:47:25.830405+0000","last_became_peered":"2023-11-16T16:47:25.830405+0000","last_unstale":"2023-11-16T16:47:45.413521+0000","last_undegraded":"2023-11-16T16:47:45.413521+0000","last_fullsized":"2023-11-16T16:47:45.413521+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:39:02.087710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.859518+0000","last_change":"2023-11-16T16:47:26.385897+0000","last_active":"2023-11-16T16:47:44.859518+0000","last_peered":"2023-11-16T16:47:44.859518+0000","last_clean":"2023-11-16T16:47:44.859518+0000","last_became_active":"2023-11-16T16:47:26.382557+0000","last_became_peered":"2023-11-16T16:47:26.382557+0000","last_unstale":"2023-11-16T16:47:44.859518+0000","last_undegraded":"2023-11-16T16:47:44.859518+0000","last_fullsized":"2023-11-16T16:47:44.859518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:19:59.689853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854309+0000","last_change":"2023-11-16T16:47:30.828521+0000","last_active":"2023-11-16T16:47:44.854309+0000","last_peered":"2023-11-16T16:47:44.854309+0000","last_clean":"2023-11-16T16:47:44.854309+0000","last_became_active":"2023-11-16T16:47:30.828135+0000","last_became_peered":"2023-11-16T16:47:30.828135+0000","last_unstale":"2023-11-16T16:47:44.854309+0000","last_undegraded":"2023-11-16T16:47:44.854309+0000","last_fullsized":"2023-11-16T16:47:44.854309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:19:57.561985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857191+0000","last_change":"2023-11-16T16:47:25.819576+0000","last_active":"2023-11-16T16:47:44.857191+0000","last_peered":"2023-11-16T16:47:44.857191+0000","last_clean":"2023-11-16T16:47:44.857191+0000","last_became_active":"2023-11-16T16:47:25.818354+0000","last_became_peered":"2023-11-16T16:47:25.818354+0000","last_unstale":"2023-11-16T16:47:44.857191+0000","last_undegraded":"2023-11-16T16:47:44.857191+0000","last_fullsized":"2023-11-16T16:47:44.857191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:43:47.079077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857839+0000","last_change":"2023-11-16T16:47:26.364632+0000","last_active":"2023-11-16T16:47:44.857839+0000","last_peered":"2023-11-16T16:47:44.857839+0000","last_clean":"2023-11-16T16:47:44.857839+0000","last_became_active":"2023-11-16T16:47:26.364288+0000","last_became_peered":"2023-11-16T16:47:26.364288+0000","last_unstale":"2023-11-16T16:47:44.857839+0000","last_undegraded":"2023-11-16T16:47:44.857839+0000","last_fullsized":"2023-11-16T16:47:44.857839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:17:52.757571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854072+0000","last_change":"2023-11-16T16:47:30.828722+0000","last_active":"2023-11-16T16:47:44.854072+0000","last_peered":"2023-11-16T16:47:44.854072+0000","last_clean":"2023-11-16T16:47:44.854072+0000","last_became_active":"2023-11-16T16:47:30.828299+0000","last_became_peered":"2023-11-16T16:47:30.828299+0000","last_unstale":"2023-11-16T16:47:44.854072+0000","last_undegraded":"2023-11-16T16:47:44.854072+0000","last_fullsized":"2023-11-16T16:47:44.854072+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:33:38.194438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856600+0000","last_change":"2023-11-16T16:47:25.817030+0000","last_active":"2023-11-16T16:47:44.856600+0000","last_peered":"2023-11-16T16:47:44.856600+0000","last_clean":"2023-11-16T16:47:44.856600+0000","last_became_active":"2023-11-16T16:47:25.816710+0000","last_became_peered":"2023-11-16T16:47:25.816710+0000","last_unstale":"2023-11-16T16:47:44.856600+0000","last_undegraded":"2023-11-16T16:47:44.856600+0000","last_fullsized":"2023-11-16T16:47:44.856600+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:40:22.859236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473673+0000","last_change":"2023-11-16T16:47:26.371226+0000","last_active":"2023-11-16T16:47:45.473673+0000","last_peered":"2023-11-16T16:47:45.473673+0000","last_clean":"2023-11-16T16:47:45.473673+0000","last_became_active":"2023-11-16T16:47:26.370925+0000","last_became_peered":"2023-11-16T16:47:26.370925+0000","last_unstale":"2023-11-16T16:47:45.473673+0000","last_undegraded":"2023-11-16T16:47:45.473673+0000","last_fullsized":"2023-11-16T16:47:45.473673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:48:12.574242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855221+0000","last_change":"2023-11-16T16:47:30.821364+0000","last_active":"2023-11-16T16:47:44.855221+0000","last_peered":"2023-11-16T16:47:44.855221+0000","last_clean":"2023-11-16T16:47:44.855221+0000","last_became_active":"2023-11-16T16:47:30.820938+0000","last_became_peered":"2023-11-16T16:47:30.820938+0000","last_unstale":"2023-11-16T16:47:44.855221+0000","last_undegraded":"2023-11-16T16:47:44.855221+0000","last_fullsized":"2023-11-16T16:47:44.855221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:13:28.692462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'6","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:53.919052+0000","last_change":"2023-11-16T16:47:25.818612+0000","last_active":"2023-11-16T16:47:53.919052+0000","last_peered":"2023-11-16T16:47:53.919052+0000","last_clean":"2023-11-16T16:47:53.919052+0000","last_became_active":"2023-11-16T16:47:25.818272+0000","last_became_peered":"2023-11-16T16:47:25.818272+0000","last_unstale":"2023-11-16T16:47:53.919052+0000","last_undegraded":"2023-11-16T16:47:53.919052+0000","last_fullsized":"2023-11-16T16:47:53.919052+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:00:33.166271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857267+0000","last_change":"2023-11-16T16:47:26.372245+0000","last_active":"2023-11-16T16:47:44.857267+0000","last_peered":"2023-11-16T16:47:44.857267+0000","last_clean":"2023-11-16T16:47:44.857267+0000","last_became_active":"2023-11-16T16:47:26.371942+0000","last_became_peered":"2023-11-16T16:47:26.371942+0000","last_unstale":"2023-11-16T16:47:44.857267+0000","last_undegraded":"2023-11-16T16:47:44.857267+0000","last_fullsized":"2023-11-16T16:47:44.857267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:25:33.613660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855696+0000","last_change":"2023-11-16T16:47:30.839669+0000","last_active":"2023-11-16T16:47:44.855696+0000","last_peered":"2023-11-16T16:47:44.855696+0000","last_clean":"2023-11-16T16:47:44.855696+0000","last_became_active":"2023-11-16T16:47:30.839165+0000","last_became_peered":"2023-11-16T16:47:30.839165+0000","last_unstale":"2023-11-16T16:47:44.855696+0000","last_undegraded":"2023-11-16T16:47:44.855696+0000","last_fullsized":"2023-11-16T16:47:44.855696+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:26:12.940109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.905121+0000","last_change":"2023-11-16T16:47:25.819867+0000","last_active":"2023-11-16T16:47:51.905121+0000","last_peered":"2023-11-16T16:47:51.905121+0000","last_clean":"2023-11-16T16:47:51.905121+0000","last_became_active":"2023-11-16T16:47:25.819516+0000","last_became_peered":"2023-11-16T16:47:25.819516+0000","last_unstale":"2023-11-16T16:47:51.905121+0000","last_undegraded":"2023-11-16T16:47:51.905121+0000","last_fullsized":"2023-11-16T16:47:51.905121+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:23:15.640449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473687+0000","last_change":"2023-11-16T16:47:26.383679+0000","last_active":"2023-11-16T16:47:45.473687+0000","last_peered":"2023-11-16T16:47:45.473687+0000","last_clean":"2023-11-16T16:47:45.473687+0000","last_became_active":"2023-11-16T16:47:26.383370+0000","last_became_peered":"2023-11-16T16:47:26.383370+0000","last_unstale":"2023-11-16T16:47:45.473687+0000","last_undegraded":"2023-11-16T16:47:45.473687+0000","last_fullsized":"2023-11-16T16:47:45.473687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:56:40.615856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472127+0000","last_change":"2023-11-16T16:47:30.832845+0000","last_active":"2023-11-16T16:47:45.472127+0000","last_peered":"2023-11-16T16:47:45.472127+0000","last_clean":"2023-11-16T16:47:45.472127+0000","last_became_active":"2023-11-16T16:47:30.832364+0000","last_became_peered":"2023-11-16T16:47:30.832364+0000","last_unstale":"2023-11-16T16:47:45.472127+0000","last_undegraded":"2023-11-16T16:47:45.472127+0000","last_fullsized":"2023-11-16T16:47:45.472127+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:41:14.777870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856301+0000","last_change":"2023-11-16T16:47:25.817420+0000","last_active":"2023-11-16T16:47:44.856301+0000","last_peered":"2023-11-16T16:47:44.856301+0000","last_clean":"2023-11-16T16:47:44.856301+0000","last_became_active":"2023-11-16T16:47:25.816160+0000","last_became_peered":"2023-11-16T16:47:25.816160+0000","last_unstale":"2023-11-16T16:47:44.856301+0000","last_undegraded":"2023-11-16T16:47:44.856301+0000","last_fullsized":"2023-11-16T16:47:44.856301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:14:09.295340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.413125+0000","last_change":"2023-11-16T16:47:26.370729+0000","last_active":"2023-11-16T16:47:45.413125+0000","last_peered":"2023-11-16T16:47:45.413125+0000","last_clean":"2023-11-16T16:47:45.413125+0000","last_became_active":"2023-11-16T16:47:26.370435+0000","last_became_peered":"2023-11-16T16:47:26.370435+0000","last_unstale":"2023-11-16T16:47:45.413125+0000","last_undegraded":"2023-11-16T16:47:45.413125+0000","last_fullsized":"2023-11-16T16:47:45.413125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:37:45.259435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.861176+0000","last_change":"2023-11-16T16:47:30.832895+0000","last_active":"2023-11-16T16:47:44.861176+0000","last_peered":"2023-11-16T16:47:44.861176+0000","last_clean":"2023-11-16T16:47:44.861176+0000","last_became_active":"2023-11-16T16:47:30.832192+0000","last_became_peered":"2023-11-16T16:47:30.832192+0000","last_unstale":"2023-11-16T16:47:44.861176+0000","last_undegraded":"2023-11-16T16:47:44.861176+0000","last_fullsized":"2023-11-16T16:47:44.861176+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:30:54.337765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473152+0000","last_change":"2023-11-16T16:47:25.815673+0000","last_active":"2023-11-16T16:47:45.473152+0000","last_peered":"2023-11-16T16:47:45.473152+0000","last_clean":"2023-11-16T16:47:45.473152+0000","last_became_active":"2023-11-16T16:47:25.815362+0000","last_became_peered":"2023-11-16T16:47:25.815362+0000","last_unstale":"2023-11-16T16:47:45.473152+0000","last_undegraded":"2023-11-16T16:47:45.473152+0000","last_fullsized":"2023-11-16T16:47:45.473152+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:50:06.110785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.853033+0000","last_change":"2023-11-16T16:47:26.383483+0000","last_active":"2023-11-16T16:47:44.853033+0000","last_peered":"2023-11-16T16:47:44.853033+0000","last_clean":"2023-11-16T16:47:44.853033+0000","last_became_active":"2023-11-16T16:47:26.383177+0000","last_became_peered":"2023-11-16T16:47:26.383177+0000","last_unstale":"2023-11-16T16:47:44.853033+0000","last_undegraded":"2023-11-16T16:47:44.853033+0000","last_fullsized":"2023-11-16T16:47:44.853033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:27:38.101289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.413695+0000","last_change":"2023-11-16T16:47:30.829044+0000","last_active":"2023-11-16T16:47:45.413695+0000","last_peered":"2023-11-16T16:47:45.413695+0000","last_clean":"2023-11-16T16:47:45.413695+0000","last_became_active":"2023-11-16T16:47:30.827398+0000","last_became_peered":"2023-11-16T16:47:30.827398+0000","last_unstale":"2023-11-16T16:47:45.413695+0000","last_undegraded":"2023-11-16T16:47:45.413695+0000","last_fullsized":"2023-11-16T16:47:45.413695+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:54:57.131841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:52.912794+0000","last_change":"2023-11-16T16:47:25.817461+0000","last_active":"2023-11-16T16:47:52.912794+0000","last_peered":"2023-11-16T16:47:52.912794+0000","last_clean":"2023-11-16T16:47:52.912794+0000","last_became_active":"2023-11-16T16:47:25.817135+0000","last_became_peered":"2023-11-16T16:47:25.817135+0000","last_unstale":"2023-11-16T16:47:52.912794+0000","last_undegraded":"2023-11-16T16:47:52.912794+0000","last_fullsized":"2023-11-16T16:47:52.912794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:00:21.140226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858098+0000","last_change":"2023-11-16T16:47:26.380218+0000","last_active":"2023-11-16T16:47:44.858098+0000","last_peered":"2023-11-16T16:47:44.858098+0000","last_clean":"2023-11-16T16:47:44.858098+0000","last_became_active":"2023-11-16T16:47:26.379849+0000","last_became_peered":"2023-11-16T16:47:26.379849+0000","last_unstale":"2023-11-16T16:47:44.858098+0000","last_undegraded":"2023-11-16T16:47:44.858098+0000","last_fullsized":"2023-11-16T16:47:44.858098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:16:44.031031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.852960+0000","last_change":"2023-11-16T16:47:30.836464+0000","last_active":"2023-11-16T16:47:44.852960+0000","last_peered":"2023-11-16T16:47:44.852960+0000","last_clean":"2023-11-16T16:47:44.852960+0000","last_became_active":"2023-11-16T16:47:30.835185+0000","last_became_peered":"2023-11-16T16:47:30.835185+0000","last_unstale":"2023-11-16T16:47:44.852960+0000","last_undegraded":"2023-11-16T16:47:44.852960+0000","last_fullsized":"2023-11-16T16:47:44.852960+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:00:34.542077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.898666+0000","last_change":"2023-11-16T16:47:25.816583+0000","last_active":"2023-11-16T16:47:50.898666+0000","last_peered":"2023-11-16T16:47:50.898666+0000","last_clean":"2023-11-16T16:47:50.898666+0000","last_became_active":"2023-11-16T16:47:25.816286+0000","last_became_peered":"2023-11-16T16:47:25.816286+0000","last_unstale":"2023-11-16T16:47:50.898666+0000","last_undegraded":"2023-11-16T16:47:50.898666+0000","last_fullsized":"2023-11-16T16:47:50.898666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:55:47.335416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.860328+0000","last_change":"2023-11-16T16:47:26.379964+0000","last_active":"2023-11-16T16:47:44.860328+0000","last_peered":"2023-11-16T16:47:44.860328+0000","last_clean":"2023-11-16T16:47:44.860328+0000","last_became_active":"2023-11-16T16:47:26.376857+0000","last_became_peered":"2023-11-16T16:47:26.376857+0000","last_unstale":"2023-11-16T16:47:44.860328+0000","last_undegraded":"2023-11-16T16:47:44.860328+0000","last_fullsized":"2023-11-16T16:47:44.860328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:44:33.471523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854625+0000","last_change":"2023-11-16T16:47:30.832572+0000","last_active":"2023-11-16T16:47:44.854625+0000","last_peered":"2023-11-16T16:47:44.854625+0000","last_clean":"2023-11-16T16:47:44.854625+0000","last_became_active":"2023-11-16T16:47:30.832138+0000","last_became_peered":"2023-11-16T16:47:30.832138+0000","last_unstale":"2023-11-16T16:47:44.854625+0000","last_undegraded":"2023-11-16T16:47:44.854625+0000","last_fullsized":"2023-11-16T16:47:44.854625+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:25:20.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.900690+0000","last_change":"2023-11-16T16:47:25.826848+0000","last_active":"2023-11-16T16:47:50.900690+0000","last_peered":"2023-11-16T16:47:50.900690+0000","last_clean":"2023-11-16T16:47:50.900690+0000","last_became_active":"2023-11-16T16:47:25.826532+0000","last_became_peered":"2023-11-16T16:47:25.826532+0000","last_unstale":"2023-11-16T16:47:50.900690+0000","last_undegraded":"2023-11-16T16:47:50.900690+0000","last_fullsized":"2023-11-16T16:47:50.900690+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:34:36.051545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.863649+0000","last_change":"2023-11-16T16:47:44.863649+0000","last_active":"2023-11-16T16:47:44.863649+0000","last_peered":"2023-11-16T16:47:44.863649+0000","last_clean":"2023-11-16T16:47:44.863649+0000","last_became_active":"2023-11-16T16:47:44.863350+0000","last_became_peered":"2023-11-16T16:47:44.863350+0000","last_unstale":"2023-11-16T16:47:44.863649+0000","last_undegraded":"2023-11-16T16:47:44.863649+0000","last_fullsized":"2023-11-16T16:47:44.863649+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:45:32.381225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.853019+0000","last_change":"2023-11-16T16:47:30.829074+0000","last_active":"2023-11-16T16:47:44.853019+0000","last_peered":"2023-11-16T16:47:44.853019+0000","last_clean":"2023-11-16T16:47:44.853019+0000","last_became_active":"2023-11-16T16:47:30.828688+0000","last_became_peered":"2023-11-16T16:47:30.828688+0000","last_unstale":"2023-11-16T16:47:44.853019+0000","last_undegraded":"2023-11-16T16:47:44.853019+0000","last_fullsized":"2023-11-16T16:47:44.853019+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:56:56.319865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'18","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:53.917768+0000","last_change":"2023-11-16T16:47:25.816469+0000","last_active":"2023-11-16T16:47:53.917768+0000","last_peered":"2023-11-16T16:47:53.917768+0000","last_clean":"2023-11-16T16:47:53.917768+0000","last_became_active":"2023-11-16T16:47:25.816115+0000","last_became_peered":"2023-11-16T16:47:25.816115+0000","last_unstale":"2023-11-16T16:47:53.917768+0000","last_undegraded":"2023-11-16T16:47:53.917768+0000","last_fullsized":"2023-11-16T16:47:53.917768+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:33:32.434023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21175,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":24,"num_write":19,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856961+0000","last_change":"2023-11-16T16:47:26.373382+0000","last_active":"2023-11-16T16:47:44.856961+0000","last_peered":"2023-11-16T16:47:44.856961+0000","last_clean":"2023-11-16T16:47:44.856961+0000","last_became_active":"2023-11-16T16:47:26.372951+0000","last_became_peered":"2023-11-16T16:47:26.372951+0000","last_unstale":"2023-11-16T16:47:44.856961+0000","last_undegraded":"2023-11-16T16:47:44.856961+0000","last_fullsized":"2023-11-16T16:47:44.856961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:29:19.977662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473826+0000","last_change":"2023-11-16T16:47:30.833890+0000","last_active":"2023-11-16T16:47:45.473826+0000","last_peered":"2023-11-16T16:47:45.473826+0000","last_clean":"2023-11-16T16:47:45.473826+0000","last_became_active":"2023-11-16T16:47:30.833489+0000","last_became_peered":"2023-11-16T16:47:30.833489+0000","last_unstale":"2023-11-16T16:47:45.473826+0000","last_undegraded":"2023-11-16T16:47:45.473826+0000","last_fullsized":"2023-11-16T16:47:45.473826+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:21:14.794417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.905467+0000","last_change":"2023-11-16T16:47:25.815950+0000","last_active":"2023-11-16T16:47:51.905467+0000","last_peered":"2023-11-16T16:47:51.905467+0000","last_clean":"2023-11-16T16:47:51.905467+0000","last_became_active":"2023-11-16T16:47:25.815657+0000","last_became_peered":"2023-11-16T16:47:25.815657+0000","last_unstale":"2023-11-16T16:47:51.905467+0000","last_undegraded":"2023-11-16T16:47:51.905467+0000","last_fullsized":"2023-11-16T16:47:51.905467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:31:06.425915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856308+0000","last_change":"2023-11-16T16:47:26.378190+0000","last_active":"2023-11-16T16:47:44.856308+0000","last_peered":"2023-11-16T16:47:44.856308+0000","last_clean":"2023-11-16T16:47:44.856308+0000","last_became_active":"2023-11-16T16:47:26.374581+0000","last_became_peered":"2023-11-16T16:47:26.374581+0000","last_unstale":"2023-11-16T16:47:44.856308+0000","last_undegraded":"2023-11-16T16:47:44.856308+0000","last_fullsized":"2023-11-16T16:47:44.856308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:22:26.437085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.471899+0000","last_change":"2023-11-16T16:47:30.826637+0000","last_active":"2023-11-16T16:47:45.471899+0000","last_peered":"2023-11-16T16:47:45.471899+0000","last_clean":"2023-11-16T16:47:45.471899+0000","last_became_active":"2023-11-16T16:47:30.826156+0000","last_became_peered":"2023-11-16T16:47:30.826156+0000","last_unstale":"2023-11-16T16:47:45.471899+0000","last_undegraded":"2023-11-16T16:47:45.471899+0000","last_fullsized":"2023-11-16T16:47:45.471899+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:01:29.047562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.901236+0000","last_change":"2023-11-16T16:47:25.822749+0000","last_active":"2023-11-16T16:47:50.901236+0000","last_peered":"2023-11-16T16:47:50.901236+0000","last_clean":"2023-11-16T16:47:50.901236+0000","last_became_active":"2023-11-16T16:47:25.822441+0000","last_became_peered":"2023-11-16T16:47:25.822441+0000","last_unstale":"2023-11-16T16:47:50.901236+0000","last_undegraded":"2023-11-16T16:47:50.901236+0000","last_fullsized":"2023-11-16T16:47:50.901236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:42:27.484194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858444+0000","last_change":"2023-11-16T16:47:26.375131+0000","last_active":"2023-11-16T16:47:44.858444+0000","last_peered":"2023-11-16T16:47:44.858444+0000","last_clean":"2023-11-16T16:47:44.858444+0000","last_became_active":"2023-11-16T16:47:26.374740+0000","last_became_peered":"2023-11-16T16:47:26.374740+0000","last_unstale":"2023-11-16T16:47:44.858444+0000","last_undegraded":"2023-11-16T16:47:44.858444+0000","last_fullsized":"2023-11-16T16:47:44.858444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:58:23.761900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.853864+0000","last_change":"2023-11-16T16:47:30.836806+0000","last_active":"2023-11-16T16:47:44.853864+0000","last_peered":"2023-11-16T16:47:44.853864+0000","last_clean":"2023-11-16T16:47:44.853864+0000","last_became_active":"2023-11-16T16:47:30.835299+0000","last_became_peered":"2023-11-16T16:47:30.835299+0000","last_unstale":"2023-11-16T16:47:44.853864+0000","last_undegraded":"2023-11-16T16:47:44.853864+0000","last_fullsized":"2023-11-16T16:47:44.853864+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:53:26.843235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:53.919284+0000","last_change":"2023-11-16T16:47:25.816793+0000","last_active":"2023-11-16T16:47:53.919284+0000","last_peered":"2023-11-16T16:47:53.919284+0000","last_clean":"2023-11-16T16:47:53.919284+0000","last_became_active":"2023-11-16T16:47:25.816433+0000","last_became_peered":"2023-11-16T16:47:25.816433+0000","last_unstale":"2023-11-16T16:47:53.919284+0000","last_undegraded":"2023-11-16T16:47:53.919284+0000","last_fullsized":"2023-11-16T16:47:53.919284+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:30:24.755449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858132+0000","last_change":"2023-11-16T16:47:26.375383+0000","last_active":"2023-11-16T16:47:44.858132+0000","last_peered":"2023-11-16T16:47:44.858132+0000","last_clean":"2023-11-16T16:47:44.858132+0000","last_became_active":"2023-11-16T16:47:26.375026+0000","last_became_peered":"2023-11-16T16:47:26.375026+0000","last_unstale":"2023-11-16T16:47:44.858132+0000","last_undegraded":"2023-11-16T16:47:44.858132+0000","last_fullsized":"2023-11-16T16:47:44.858132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:07:01.547781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.475115+0000","last_change":"2023-11-16T16:47:45.475115+0000","last_active":"2023-11-16T16:47:45.475115+0000","last_peered":"2023-11-16T16:47:45.475115+0000","last_clean":"2023-11-16T16:47:45.475115+0000","last_became_active":"2023-11-16T16:47:45.474475+0000","last_became_peered":"2023-11-16T16:47:45.474475+0000","last_unstale":"2023-11-16T16:47:45.475115+0000","last_undegraded":"2023-11-16T16:47:45.475115+0000","last_fullsized":"2023-11-16T16:47:45.475115+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:14:57.456154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.898156+0000","last_change":"2023-11-16T16:47:25.826911+0000","last_active":"2023-11-16T16:47:50.898156+0000","last_peered":"2023-11-16T16:47:50.898156+0000","last_clean":"2023-11-16T16:47:50.898156+0000","last_became_active":"2023-11-16T16:47:25.826605+0000","last_became_peered":"2023-11-16T16:47:25.826605+0000","last_unstale":"2023-11-16T16:47:50.898156+0000","last_undegraded":"2023-11-16T16:47:50.898156+0000","last_fullsized":"2023-11-16T16:47:50.898156+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:43:53.441571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472956+0000","last_change":"2023-11-16T16:47:26.376072+0000","last_active":"2023-11-16T16:47:45.472956+0000","last_peered":"2023-11-16T16:47:45.472956+0000","last_clean":"2023-11-16T16:47:45.472956+0000","last_became_active":"2023-11-16T16:47:26.375585+0000","last_became_peered":"2023-11-16T16:47:26.375585+0000","last_unstale":"2023-11-16T16:47:45.472956+0000","last_undegraded":"2023-11-16T16:47:45.472956+0000","last_fullsized":"2023-11-16T16:47:45.472956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:30:44.970814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472838+0000","last_change":"2023-11-16T16:47:30.834558+0000","last_active":"2023-11-16T16:47:45.472838+0000","last_peered":"2023-11-16T16:47:45.472838+0000","last_clean":"2023-11-16T16:47:45.472838+0000","last_became_active":"2023-11-16T16:47:30.833989+0000","last_became_peered":"2023-11-16T16:47:30.833989+0000","last_unstale":"2023-11-16T16:47:45.472838+0000","last_undegraded":"2023-11-16T16:47:45.472838+0000","last_fullsized":"2023-11-16T16:47:45.472838+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:54:42.983565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857593+0000","last_change":"2023-11-16T16:47:25.817016+0000","last_active":"2023-11-16T16:47:44.857593+0000","last_peered":"2023-11-16T16:47:44.857593+0000","last_clean":"2023-11-16T16:47:44.857593+0000","last_became_active":"2023-11-16T16:47:25.816752+0000","last_became_peered":"2023-11-16T16:47:25.816752+0000","last_unstale":"2023-11-16T16:47:44.857593+0000","last_undegraded":"2023-11-16T16:47:44.857593+0000","last_fullsized":"2023-11-16T16:47:44.857593+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:40:44.076735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472022+0000","last_change":"2023-11-16T16:47:26.374947+0000","last_active":"2023-11-16T16:47:45.472022+0000","last_peered":"2023-11-16T16:47:45.472022+0000","last_clean":"2023-11-16T16:47:45.472022+0000","last_became_active":"2023-11-16T16:47:26.374533+0000","last_became_peered":"2023-11-16T16:47:26.374533+0000","last_unstale":"2023-11-16T16:47:45.472022+0000","last_undegraded":"2023-11-16T16:47:45.472022+0000","last_fullsized":"2023-11-16T16:47:45.472022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:58:16.020607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856695+0000","last_change":"2023-11-16T16:47:30.830506+0000","last_active":"2023-11-16T16:47:44.856695+0000","last_peered":"2023-11-16T16:47:44.856695+0000","last_clean":"2023-11-16T16:47:44.856695+0000","last_became_active":"2023-11-16T16:47:30.830086+0000","last_became_peered":"2023-11-16T16:47:30.830086+0000","last_unstale":"2023-11-16T16:47:44.856695+0000","last_undegraded":"2023-11-16T16:47:44.856695+0000","last_fullsized":"2023-11-16T16:47:44.856695+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:50:36.980481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.853657+0000","last_change":"2023-11-16T16:47:25.830104+0000","last_active":"2023-11-16T16:47:44.853657+0000","last_peered":"2023-11-16T16:47:44.853657+0000","last_clean":"2023-11-16T16:47:44.853657+0000","last_became_active":"2023-11-16T16:47:25.829232+0000","last_became_peered":"2023-11-16T16:47:25.829232+0000","last_unstale":"2023-11-16T16:47:44.853657+0000","last_undegraded":"2023-11-16T16:47:44.853657+0000","last_fullsized":"2023-11-16T16:47:44.853657+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:29:15.724324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473108+0000","last_change":"2023-11-16T16:47:26.368918+0000","last_active":"2023-11-16T16:47:45.473108+0000","last_peered":"2023-11-16T16:47:45.473108+0000","last_clean":"2023-11-16T16:47:45.473108+0000","last_became_active":"2023-11-16T16:47:26.368592+0000","last_became_peered":"2023-11-16T16:47:26.368592+0000","last_unstale":"2023-11-16T16:47:45.473108+0000","last_undegraded":"2023-11-16T16:47:45.473108+0000","last_fullsized":"2023-11-16T16:47:45.473108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:09:36.266152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857755+0000","last_change":"2023-11-16T16:47:30.830558+0000","last_active":"2023-11-16T16:47:44.857755+0000","last_peered":"2023-11-16T16:47:44.857755+0000","last_clean":"2023-11-16T16:47:44.857755+0000","last_became_active":"2023-11-16T16:47:30.829777+0000","last_became_peered":"2023-11-16T16:47:30.829777+0000","last_unstale":"2023-11-16T16:47:44.857755+0000","last_undegraded":"2023-11-16T16:47:44.857755+0000","last_fullsized":"2023-11-16T16:47:44.857755+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:15:47.182273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858408+0000","last_change":"2023-11-16T16:47:25.820315+0000","last_active":"2023-11-16T16:47:44.858408+0000","last_peered":"2023-11-16T16:47:44.858408+0000","last_clean":"2023-11-16T16:47:44.858408+0000","last_became_active":"2023-11-16T16:47:25.820032+0000","last_became_peered":"2023-11-16T16:47:25.820032+0000","last_unstale":"2023-11-16T16:47:44.858408+0000","last_undegraded":"2023-11-16T16:47:44.858408+0000","last_fullsized":"2023-11-16T16:47:44.858408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:36:13.332902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472754+0000","last_change":"2023-11-16T16:47:26.379841+0000","last_active":"2023-11-16T16:47:45.472754+0000","last_peered":"2023-11-16T16:47:45.472754+0000","last_clean":"2023-11-16T16:47:45.472754+0000","last_became_active":"2023-11-16T16:47:26.377401+0000","last_became_peered":"2023-11-16T16:47:26.377401+0000","last_unstale":"2023-11-16T16:47:45.472754+0000","last_undegraded":"2023-11-16T16:47:45.472754+0000","last_fullsized":"2023-11-16T16:47:45.472754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:21:55.430094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473595+0000","last_change":"2023-11-16T16:47:45.473595+0000","last_active":"2023-11-16T16:47:45.473595+0000","last_peered":"2023-11-16T16:47:45.473595+0000","last_clean":"2023-11-16T16:47:45.473595+0000","last_became_active":"2023-11-16T16:47:45.473100+0000","last_became_peered":"2023-11-16T16:47:45.473100+0000","last_unstale":"2023-11-16T16:47:45.473595+0000","last_undegraded":"2023-11-16T16:47:45.473595+0000","last_fullsized":"2023-11-16T16:47:45.473595+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:17:45.050424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473340+0000","last_change":"2023-11-16T16:47:26.364112+0000","last_active":"2023-11-16T16:47:45.473340+0000","last_peered":"2023-11-16T16:47:45.473340+0000","last_clean":"2023-11-16T16:47:45.473340+0000","last_became_active":"2023-11-16T16:47:26.363767+0000","last_became_peered":"2023-11-16T16:47:26.363767+0000","last_unstale":"2023-11-16T16:47:45.473340+0000","last_undegraded":"2023-11-16T16:47:45.473340+0000","last_fullsized":"2023-11-16T16:47:45.473340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:01:40.809234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856188+0000","last_change":"2023-11-16T16:47:25.830012+0000","last_active":"2023-11-16T16:47:44.856188+0000","last_peered":"2023-11-16T16:47:44.856188+0000","last_clean":"2023-11-16T16:47:44.856188+0000","last_became_active":"2023-11-16T16:47:25.829474+0000","last_became_peered":"2023-11-16T16:47:25.829474+0000","last_unstale":"2023-11-16T16:47:44.856188+0000","last_undegraded":"2023-11-16T16:47:44.856188+0000","last_fullsized":"2023-11-16T16:47:44.856188+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:01:21.296554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858885+0000","last_change":"2023-11-16T16:47:30.832724+0000","last_active":"2023-11-16T16:47:44.858885+0000","last_peered":"2023-11-16T16:47:44.858885+0000","last_clean":"2023-11-16T16:47:44.858885+0000","last_became_active":"2023-11-16T16:47:30.832226+0000","last_became_peered":"2023-11-16T16:47:30.832226+0000","last_unstale":"2023-11-16T16:47:44.858885+0000","last_undegraded":"2023-11-16T16:47:44.858885+0000","last_fullsized":"2023-11-16T16:47:44.858885+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:11:15.234460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.476312+0000","last_change":"2023-11-16T16:47:45.474859+0000","last_active":"2023-11-16T16:47:45.476312+0000","last_peered":"2023-11-16T16:47:45.476312+0000","last_clean":"2023-11-16T16:47:45.476312+0000","last_became_active":"2023-11-16T16:47:45.474439+0000","last_became_peered":"2023-11-16T16:47:45.474439+0000","last_unstale":"2023-11-16T16:47:45.476312+0000","last_undegraded":"2023-11-16T16:47:45.476312+0000","last_fullsized":"2023-11-16T16:47:45.476312+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:05:04.346537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:52.910430+0000","last_change":"2023-11-16T16:47:25.813894+0000","last_active":"2023-11-16T16:47:52.910430+0000","last_peered":"2023-11-16T16:47:52.910430+0000","last_clean":"2023-11-16T16:47:52.910430+0000","last_became_active":"2023-11-16T16:47:25.813526+0000","last_became_peered":"2023-11-16T16:47:25.813526+0000","last_unstale":"2023-11-16T16:47:52.910430+0000","last_undegraded":"2023-11-16T16:47:52.910430+0000","last_fullsized":"2023-11-16T16:47:52.910430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:26:19.570859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":359,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857425+0000","last_change":"2023-11-16T16:46:18.016268+0000","last_active":"2023-11-16T16:47:44.857425+0000","last_peered":"2023-11-16T16:47:44.857425+0000","last_clean":"2023-11-16T16:47:44.857425+0000","last_became_active":"2023-11-16T16:46:18.015772+0000","last_became_peered":"2023-11-16T16:46:18.015772+0000","last_unstale":"2023-11-16T16:47:44.857425+0000","last_undegraded":"2023-11-16T16:47:44.857425+0000","last_fullsized":"2023-11-16T16:47:44.857425+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:43:04.489460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:43:04.489460+0000","last_clean_scrub_stamp":"2023-11-16T16:43:04.489460+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:22:57.779720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473045+0000","last_change":"2023-11-16T16:47:30.830574+0000","last_active":"2023-11-16T16:47:45.473045+0000","last_peered":"2023-11-16T16:47:45.473045+0000","last_clean":"2023-11-16T16:47:45.473045+0000","last_became_active":"2023-11-16T16:47:30.830040+0000","last_became_peered":"2023-11-16T16:47:30.830040+0000","last_unstale":"2023-11-16T16:47:45.473045+0000","last_undegraded":"2023-11-16T16:47:45.473045+0000","last_fullsized":"2023-11-16T16:47:45.473045+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:29:55.893689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856017+0000","last_change":"2023-11-16T16:47:26.378278+0000","last_active":"2023-11-16T16:47:44.856017+0000","last_peered":"2023-11-16T16:47:44.856017+0000","last_clean":"2023-11-16T16:47:44.856017+0000","last_became_active":"2023-11-16T16:47:26.377139+0000","last_became_peered":"2023-11-16T16:47:26.377139+0000","last_unstale":"2023-11-16T16:47:44.856017+0000","last_undegraded":"2023-11-16T16:47:44.856017+0000","last_fullsized":"2023-11-16T16:47:44.856017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:43:35.952129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.906018+0000","last_change":"2023-11-16T16:47:25.815714+0000","last_active":"2023-11-16T16:47:51.906018+0000","last_peered":"2023-11-16T16:47:51.906018+0000","last_clean":"2023-11-16T16:47:51.906018+0000","last_became_active":"2023-11-16T16:47:25.815452+0000","last_became_peered":"2023-11-16T16:47:25.815452+0000","last_unstale":"2023-11-16T16:47:51.906018+0000","last_undegraded":"2023-11-16T16:47:51.906018+0000","last_fullsized":"2023-11-16T16:47:51.906018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:33:18.235656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472075+0000","last_change":"2023-11-16T16:47:30.826690+0000","last_active":"2023-11-16T16:47:45.472075+0000","last_peered":"2023-11-16T16:47:45.472075+0000","last_clean":"2023-11-16T16:47:45.472075+0000","last_became_active":"2023-11-16T16:47:30.826216+0000","last_became_peered":"2023-11-16T16:47:30.826216+0000","last_unstale":"2023-11-16T16:47:45.472075+0000","last_undegraded":"2023-11-16T16:47:45.472075+0000","last_fullsized":"2023-11-16T16:47:45.472075+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:59:47.136636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857062+0000","last_change":"2023-11-16T16:47:25.826960+0000","last_active":"2023-11-16T16:47:44.857062+0000","last_peered":"2023-11-16T16:47:44.857062+0000","last_clean":"2023-11-16T16:47:44.857062+0000","last_became_active":"2023-11-16T16:47:25.826668+0000","last_became_peered":"2023-11-16T16:47:25.826668+0000","last_unstale":"2023-11-16T16:47:44.857062+0000","last_undegraded":"2023-11-16T16:47:44.857062+0000","last_fullsized":"2023-11-16T16:47:44.857062+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:45:36.995013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856905+0000","last_change":"2023-11-16T16:47:26.370684+0000","last_active":"2023-11-16T16:47:44.856905+0000","last_peered":"2023-11-16T16:47:44.856905+0000","last_clean":"2023-11-16T16:47:44.856905+0000","last_became_active":"2023-11-16T16:47:26.370323+0000","last_became_peered":"2023-11-16T16:47:26.370323+0000","last_unstale":"2023-11-16T16:47:44.856905+0000","last_undegraded":"2023-11-16T16:47:44.856905+0000","last_fullsized":"2023-11-16T16:47:44.856905+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:33:01.058990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.412979+0000","last_change":"2023-11-16T16:47:30.837921+0000","last_active":"2023-11-16T16:47:45.412979+0000","last_peered":"2023-11-16T16:47:45.412979+0000","last_clean":"2023-11-16T16:47:45.412979+0000","last_became_active":"2023-11-16T16:47:30.831491+0000","last_became_peered":"2023-11-16T16:47:30.831491+0000","last_unstale":"2023-11-16T16:47:45.412979+0000","last_undegraded":"2023-11-16T16:47:45.412979+0000","last_fullsized":"2023-11-16T16:47:45.412979+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:02:07.321263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.907422+0000","last_change":"2023-11-16T16:47:25.816416+0000","last_active":"2023-11-16T16:47:51.907422+0000","last_peered":"2023-11-16T16:47:51.907422+0000","last_clean":"2023-11-16T16:47:51.907422+0000","last_became_active":"2023-11-16T16:47:25.815875+0000","last_became_peered":"2023-11-16T16:47:25.815875+0000","last_unstale":"2023-11-16T16:47:51.907422+0000","last_undegraded":"2023-11-16T16:47:51.907422+0000","last_fullsized":"2023-11-16T16:47:51.907422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:34:15.271617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.413301+0000","last_change":"2023-11-16T16:47:26.386831+0000","last_active":"2023-11-16T16:47:45.413301+0000","last_peered":"2023-11-16T16:47:45.413301+0000","last_clean":"2023-11-16T16:47:45.413301+0000","last_became_active":"2023-11-16T16:47:26.386518+0000","last_became_peered":"2023-11-16T16:47:26.386518+0000","last_unstale":"2023-11-16T16:47:45.413301+0000","last_undegraded":"2023-11-16T16:47:45.413301+0000","last_fullsized":"2023-11-16T16:47:45.413301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:28:40.491752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854593+0000","last_change":"2023-11-16T16:47:30.823733+0000","last_active":"2023-11-16T16:47:44.854593+0000","last_peered":"2023-11-16T16:47:44.854593+0000","last_clean":"2023-11-16T16:47:44.854593+0000","last_became_active":"2023-11-16T16:47:30.823293+0000","last_became_peered":"2023-11-16T16:47:30.823293+0000","last_unstale":"2023-11-16T16:47:44.854593+0000","last_undegraded":"2023-11-16T16:47:44.854593+0000","last_fullsized":"2023-11-16T16:47:44.854593+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:30:09.995445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.899469+0000","last_change":"2023-11-16T16:47:25.818137+0000","last_active":"2023-11-16T16:47:50.899469+0000","last_peered":"2023-11-16T16:47:50.899469+0000","last_clean":"2023-11-16T16:47:50.899469+0000","last_became_active":"2023-11-16T16:47:25.817661+0000","last_became_peered":"2023-11-16T16:47:25.817661+0000","last_unstale":"2023-11-16T16:47:50.899469+0000","last_undegraded":"2023-11-16T16:47:50.899469+0000","last_fullsized":"2023-11-16T16:47:50.899469+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:17:33.467718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.415106+0000","last_change":"2023-11-16T16:47:45.415106+0000","last_active":"2023-11-16T16:47:45.415106+0000","last_peered":"2023-11-16T16:47:45.415106+0000","last_clean":"2023-11-16T16:47:45.415106+0000","last_became_active":"2023-11-16T16:47:45.414554+0000","last_became_peered":"2023-11-16T16:47:45.414554+0000","last_unstale":"2023-11-16T16:47:45.415106+0000","last_undegraded":"2023-11-16T16:47:45.415106+0000","last_fullsized":"2023-11-16T16:47:45.415106+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:20:08.005726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,8],"acting":[0,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854667+0000","last_change":"2023-11-16T16:47:30.820944+0000","last_active":"2023-11-16T16:47:44.854667+0000","last_peered":"2023-11-16T16:47:44.854667+0000","last_clean":"2023-11-16T16:47:44.854667+0000","last_became_active":"2023-11-16T16:47:30.820514+0000","last_became_peered":"2023-11-16T16:47:30.820514+0000","last_unstale":"2023-11-16T16:47:44.854667+0000","last_undegraded":"2023-11-16T16:47:44.854667+0000","last_fullsized":"2023-11-16T16:47:44.854667+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:18:55.421306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.899895+0000","last_change":"2023-11-16T16:47:25.826960+0000","last_active":"2023-11-16T16:47:50.899895+0000","last_peered":"2023-11-16T16:47:50.899895+0000","last_clean":"2023-11-16T16:47:50.899895+0000","last_became_active":"2023-11-16T16:47:25.826671+0000","last_became_peered":"2023-11-16T16:47:25.826671+0000","last_unstale":"2023-11-16T16:47:50.899895+0000","last_undegraded":"2023-11-16T16:47:50.899895+0000","last_fullsized":"2023-11-16T16:47:50.899895+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:41:34.201037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856299+0000","last_change":"2023-11-16T16:47:26.364697+0000","last_active":"2023-11-16T16:47:44.856299+0000","last_peered":"2023-11-16T16:47:44.856299+0000","last_clean":"2023-11-16T16:47:44.856299+0000","last_became_active":"2023-11-16T16:47:26.364418+0000","last_became_peered":"2023-11-16T16:47:26.364418+0000","last_unstale":"2023-11-16T16:47:44.856299+0000","last_undegraded":"2023-11-16T16:47:44.856299+0000","last_fullsized":"2023-11-16T16:47:44.856299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:31:32.598699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.475963+0000","last_change":"2023-11-16T16:47:45.474571+0000","last_active":"2023-11-16T16:47:45.475963+0000","last_peered":"2023-11-16T16:47:45.475963+0000","last_clean":"2023-11-16T16:47:45.475963+0000","last_became_active":"2023-11-16T16:47:45.474093+0000","last_became_peered":"2023-11-16T16:47:45.474093+0000","last_unstale":"2023-11-16T16:47:45.475963+0000","last_undegraded":"2023-11-16T16:47:45.475963+0000","last_fullsized":"2023-11-16T16:47:45.475963+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:24:09.557557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.899977+0000","last_change":"2023-11-16T16:47:25.826208+0000","last_active":"2023-11-16T16:47:50.899977+0000","last_peered":"2023-11-16T16:47:50.899977+0000","last_clean":"2023-11-16T16:47:50.899977+0000","last_became_active":"2023-11-16T16:47:25.825853+0000","last_became_peered":"2023-11-16T16:47:25.825853+0000","last_unstale":"2023-11-16T16:47:50.899977+0000","last_undegraded":"2023-11-16T16:47:50.899977+0000","last_fullsized":"2023-11-16T16:47:50.899977+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:02:40.593175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.471677+0000","last_change":"2023-11-16T16:47:26.361786+0000","last_active":"2023-11-16T16:47:45.471677+0000","last_peered":"2023-11-16T16:47:45.471677+0000","last_clean":"2023-11-16T16:47:45.471677+0000","last_became_active":"2023-11-16T16:47:26.361410+0000","last_became_peered":"2023-11-16T16:47:26.361410+0000","last_unstale":"2023-11-16T16:47:45.471677+0000","last_undegraded":"2023-11-16T16:47:45.471677+0000","last_fullsized":"2023-11-16T16:47:45.471677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:14:02.213856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.474814+0000","last_change":"2023-11-16T16:47:45.474123+0000","last_active":"2023-11-16T16:47:45.474814+0000","last_peered":"2023-11-16T16:47:45.474814+0000","last_clean":"2023-11-16T16:47:45.474814+0000","last_became_active":"2023-11-16T16:47:45.473630+0000","last_became_peered":"2023-11-16T16:47:45.473630+0000","last_unstale":"2023-11-16T16:47:45.474814+0000","last_undegraded":"2023-11-16T16:47:45.474814+0000","last_fullsized":"2023-11-16T16:47:45.474814+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:39:24.719048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855051+0000","last_change":"2023-11-16T16:47:25.816793+0000","last_active":"2023-11-16T16:47:44.855051+0000","last_peered":"2023-11-16T16:47:44.855051+0000","last_clean":"2023-11-16T16:47:44.855051+0000","last_became_active":"2023-11-16T16:47:25.815136+0000","last_became_peered":"2023-11-16T16:47:25.815136+0000","last_unstale":"2023-11-16T16:47:44.855051+0000","last_undegraded":"2023-11-16T16:47:44.855051+0000","last_fullsized":"2023-11-16T16:47:44.855051+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:12:25.041301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.412827+0000","last_change":"2023-11-16T16:47:26.379945+0000","last_active":"2023-11-16T16:47:45.412827+0000","last_peered":"2023-11-16T16:47:45.412827+0000","last_clean":"2023-11-16T16:47:45.412827+0000","last_became_active":"2023-11-16T16:47:26.376552+0000","last_became_peered":"2023-11-16T16:47:26.376552+0000","last_unstale":"2023-11-16T16:47:45.412827+0000","last_undegraded":"2023-11-16T16:47:45.412827+0000","last_fullsized":"2023-11-16T16:47:45.412827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:44:18.198024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854429+0000","last_change":"2023-11-16T16:47:30.826281+0000","last_active":"2023-11-16T16:47:44.854429+0000","last_peered":"2023-11-16T16:47:44.854429+0000","last_clean":"2023-11-16T16:47:44.854429+0000","last_became_active":"2023-11-16T16:47:30.825920+0000","last_became_peered":"2023-11-16T16:47:30.825920+0000","last_unstale":"2023-11-16T16:47:44.854429+0000","last_undegraded":"2023-11-16T16:47:44.854429+0000","last_fullsized":"2023-11-16T16:47:44.854429+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:23:25.700645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.899633+0000","last_change":"2023-11-16T16:47:25.815879+0000","last_active":"2023-11-16T16:47:50.899633+0000","last_peered":"2023-11-16T16:47:50.899633+0000","last_clean":"2023-11-16T16:47:50.899633+0000","last_became_active":"2023-11-16T16:47:25.815557+0000","last_became_peered":"2023-11-16T16:47:25.815557+0000","last_unstale":"2023-11-16T16:47:50.899633+0000","last_undegraded":"2023-11-16T16:47:50.899633+0000","last_fullsized":"2023-11-16T16:47:50.899633+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:04:39.245684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858518+0000","last_change":"2023-11-16T16:47:26.365013+0000","last_active":"2023-11-16T16:47:44.858518+0000","last_peered":"2023-11-16T16:47:44.858518+0000","last_clean":"2023-11-16T16:47:44.858518+0000","last_became_active":"2023-11-16T16:47:26.364710+0000","last_became_peered":"2023-11-16T16:47:26.364710+0000","last_unstale":"2023-11-16T16:47:44.858518+0000","last_undegraded":"2023-11-16T16:47:44.858518+0000","last_fullsized":"2023-11-16T16:47:44.858518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:57:21.045225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857432+0000","last_change":"2023-11-16T16:47:30.826880+0000","last_active":"2023-11-16T16:47:44.857432+0000","last_peered":"2023-11-16T16:47:44.857432+0000","last_clean":"2023-11-16T16:47:44.857432+0000","last_became_active":"2023-11-16T16:47:30.826547+0000","last_became_peered":"2023-11-16T16:47:30.826547+0000","last_unstale":"2023-11-16T16:47:44.857432+0000","last_undegraded":"2023-11-16T16:47:44.857432+0000","last_fullsized":"2023-11-16T16:47:44.857432+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:34:34.448480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.908865+0000","last_change":"2023-11-16T16:47:25.825246+0000","last_active":"2023-11-16T16:47:51.908865+0000","last_peered":"2023-11-16T16:47:51.908865+0000","last_clean":"2023-11-16T16:47:51.908865+0000","last_became_active":"2023-11-16T16:47:25.824929+0000","last_became_peered":"2023-11-16T16:47:25.824929+0000","last_unstale":"2023-11-16T16:47:51.908865+0000","last_undegraded":"2023-11-16T16:47:51.908865+0000","last_fullsized":"2023-11-16T16:47:51.908865+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:15:26.168130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.859304+0000","last_change":"2023-11-16T16:47:26.372266+0000","last_active":"2023-11-16T16:47:44.859304+0000","last_peered":"2023-11-16T16:47:44.859304+0000","last_clean":"2023-11-16T16:47:44.859304+0000","last_became_active":"2023-11-16T16:47:26.371857+0000","last_became_peered":"2023-11-16T16:47:26.371857+0000","last_unstale":"2023-11-16T16:47:44.859304+0000","last_undegraded":"2023-11-16T16:47:44.859304+0000","last_fullsized":"2023-11-16T16:47:44.859304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:22:33.979224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.859613+0000","last_change":"2023-11-16T16:47:30.832187+0000","last_active":"2023-11-16T16:47:44.859613+0000","last_peered":"2023-11-16T16:47:44.859613+0000","last_clean":"2023-11-16T16:47:44.859613+0000","last_became_active":"2023-11-16T16:47:30.828115+0000","last_became_peered":"2023-11-16T16:47:30.828115+0000","last_unstale":"2023-11-16T16:47:44.859613+0000","last_undegraded":"2023-11-16T16:47:44.859613+0000","last_fullsized":"2023-11-16T16:47:44.859613+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:05:02.381726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473519+0000","last_change":"2023-11-16T16:47:25.822706+0000","last_active":"2023-11-16T16:47:45.473519+0000","last_peered":"2023-11-16T16:47:45.473519+0000","last_clean":"2023-11-16T16:47:45.473519+0000","last_became_active":"2023-11-16T16:47:25.822400+0000","last_became_peered":"2023-11-16T16:47:25.822400+0000","last_unstale":"2023-11-16T16:47:45.473519+0000","last_undegraded":"2023-11-16T16:47:45.473519+0000","last_fullsized":"2023-11-16T16:47:45.473519+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:32:15.408720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.474751+0000","last_change":"2023-11-16T16:47:45.474751+0000","last_active":"2023-11-16T16:47:45.474751+0000","last_peered":"2023-11-16T16:47:45.474751+0000","last_clean":"2023-11-16T16:47:45.474751+0000","last_became_active":"2023-11-16T16:47:45.474415+0000","last_became_peered":"2023-11-16T16:47:45.474415+0000","last_unstale":"2023-11-16T16:47:45.474751+0000","last_undegraded":"2023-11-16T16:47:45.474751+0000","last_fullsized":"2023-11-16T16:47:45.474751+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:56:14.927411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472751+0000","last_change":"2023-11-16T16:47:30.822549+0000","last_active":"2023-11-16T16:47:45.472751+0000","last_peered":"2023-11-16T16:47:45.472751+0000","last_clean":"2023-11-16T16:47:45.472751+0000","last_became_active":"2023-11-16T16:47:30.822150+0000","last_became_peered":"2023-11-16T16:47:30.822150+0000","last_unstale":"2023-11-16T16:47:45.472751+0000","last_undegraded":"2023-11-16T16:47:45.472751+0000","last_fullsized":"2023-11-16T16:47:45.472751+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:17:45.983962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857636+0000","last_change":"2023-11-16T16:47:25.826947+0000","last_active":"2023-11-16T16:47:44.857636+0000","last_peered":"2023-11-16T16:47:44.857636+0000","last_clean":"2023-11-16T16:47:44.857636+0000","last_became_active":"2023-11-16T16:47:25.826587+0000","last_became_peered":"2023-11-16T16:47:25.826587+0000","last_unstale":"2023-11-16T16:47:44.857636+0000","last_undegraded":"2023-11-16T16:47:44.857636+0000","last_fullsized":"2023-11-16T16:47:44.857636+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:26:31.810188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858841+0000","last_change":"2023-11-16T16:47:26.373914+0000","last_active":"2023-11-16T16:47:44.858841+0000","last_peered":"2023-11-16T16:47:44.858841+0000","last_clean":"2023-11-16T16:47:44.858841+0000","last_became_active":"2023-11-16T16:47:26.373594+0000","last_became_peered":"2023-11-16T16:47:26.373594+0000","last_unstale":"2023-11-16T16:47:44.858841+0000","last_undegraded":"2023-11-16T16:47:44.858841+0000","last_fullsized":"2023-11-16T16:47:44.858841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:18:55.608486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472165+0000","last_change":"2023-11-16T16:47:26.368237+0000","last_active":"2023-11-16T16:47:45.472165+0000","last_peered":"2023-11-16T16:47:45.472165+0000","last_clean":"2023-11-16T16:47:45.472165+0000","last_became_active":"2023-11-16T16:47:26.367957+0000","last_became_peered":"2023-11-16T16:47:26.367957+0000","last_unstale":"2023-11-16T16:47:45.472165+0000","last_undegraded":"2023-11-16T16:47:45.472165+0000","last_fullsized":"2023-11-16T16:47:45.472165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:33:19.976017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854982+0000","last_change":"2023-11-16T16:47:26.375399+0000","last_active":"2023-11-16T16:47:44.854982+0000","last_peered":"2023-11-16T16:47:44.854982+0000","last_clean":"2023-11-16T16:47:44.854982+0000","last_became_active":"2023-11-16T16:47:26.375064+0000","last_became_peered":"2023-11-16T16:47:26.375064+0000","last_unstale":"2023-11-16T16:47:44.854982+0000","last_undegraded":"2023-11-16T16:47:44.854982+0000","last_fullsized":"2023-11-16T16:47:44.854982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:10:23.687848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854908+0000","last_change":"2023-11-16T16:47:26.377080+0000","last_active":"2023-11-16T16:47:44.854908+0000","last_peered":"2023-11-16T16:47:44.854908+0000","last_clean":"2023-11-16T16:47:44.854908+0000","last_became_active":"2023-11-16T16:47:26.376759+0000","last_became_peered":"2023-11-16T16:47:26.376759+0000","last_unstale":"2023-11-16T16:47:44.854908+0000","last_undegraded":"2023-11-16T16:47:44.854908+0000","last_fullsized":"2023-11-16T16:47:44.854908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:51:18.518523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.862174+0000","last_change":"2023-11-16T16:47:26.380619+0000","last_active":"2023-11-16T16:47:44.862174+0000","last_peered":"2023-11-16T16:47:44.862174+0000","last_clean":"2023-11-16T16:47:44.862174+0000","last_became_active":"2023-11-16T16:47:26.378668+0000","last_became_peered":"2023-11-16T16:47:26.378668+0000","last_unstale":"2023-11-16T16:47:44.862174+0000","last_undegraded":"2023-11-16T16:47:44.862174+0000","last_fullsized":"2023-11-16T16:47:44.862174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:59:30.902022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856785+0000","last_change":"2023-11-16T16:47:26.377482+0000","last_active":"2023-11-16T16:47:44.856785+0000","last_peered":"2023-11-16T16:47:44.856785+0000","last_clean":"2023-11-16T16:47:44.856785+0000","last_became_active":"2023-11-16T16:47:26.377077+0000","last_became_peered":"2023-11-16T16:47:26.377077+0000","last_unstale":"2023-11-16T16:47:44.856785+0000","last_undegraded":"2023-11-16T16:47:44.856785+0000","last_fullsized":"2023-11-16T16:47:44.856785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:19:35.960276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472542+0000","last_change":"2023-11-16T16:47:26.368193+0000","last_active":"2023-11-16T16:47:45.472542+0000","last_peered":"2023-11-16T16:47:45.472542+0000","last_clean":"2023-11-16T16:47:45.472542+0000","last_became_active":"2023-11-16T16:47:26.367499+0000","last_became_peered":"2023-11-16T16:47:26.367499+0000","last_unstale":"2023-11-16T16:47:45.472542+0000","last_undegraded":"2023-11-16T16:47:45.472542+0000","last_fullsized":"2023-11-16T16:47:45.472542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:09:17.429110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855675+0000","last_change":"2023-11-16T16:47:26.367340+0000","last_active":"2023-11-16T16:47:44.855675+0000","last_peered":"2023-11-16T16:47:44.855675+0000","last_clean":"2023-11-16T16:47:44.855675+0000","last_became_active":"2023-11-16T16:47:26.367046+0000","last_became_peered":"2023-11-16T16:47:26.367046+0000","last_unstale":"2023-11-16T16:47:44.855675+0000","last_undegraded":"2023-11-16T16:47:44.855675+0000","last_fullsized":"2023-11-16T16:47:44.855675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:14:30.539229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.413517+0000","last_change":"2023-11-16T16:47:26.386174+0000","last_active":"2023-11-16T16:47:45.413517+0000","last_peered":"2023-11-16T16:47:45.413517+0000","last_clean":"2023-11-16T16:47:45.413517+0000","last_became_active":"2023-11-16T16:47:26.385496+0000","last_became_peered":"2023-11-16T16:47:26.385496+0000","last_unstale":"2023-11-16T16:47:45.413517+0000","last_undegraded":"2023-11-16T16:47:45.413517+0000","last_fullsized":"2023-11-16T16:47:45.413517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:52:02.965352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473021+0000","last_change":"2023-11-16T16:47:26.368224+0000","last_active":"2023-11-16T16:47:45.473021+0000","last_peered":"2023-11-16T16:47:45.473021+0000","last_clean":"2023-11-16T16:47:45.473021+0000","last_became_active":"2023-11-16T16:47:26.367909+0000","last_became_peered":"2023-11-16T16:47:26.367909+0000","last_unstale":"2023-11-16T16:47:45.473021+0000","last_undegraded":"2023-11-16T16:47:45.473021+0000","last_fullsized":"2023-11-16T16:47:45.473021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:52:55.966566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.412846+0000","last_change":"2023-11-16T16:47:26.381224+0000","last_active":"2023-11-16T16:47:45.412846+0000","last_peered":"2023-11-16T16:47:45.412846+0000","last_clean":"2023-11-16T16:47:45.412846+0000","last_became_active":"2023-11-16T16:47:26.376586+0000","last_became_peered":"2023-11-16T16:47:26.376586+0000","last_unstale":"2023-11-16T16:47:45.412846+0000","last_undegraded":"2023-11-16T16:47:45.412846+0000","last_fullsized":"2023-11-16T16:47:45.412846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:36:59.850981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858440+0000","last_change":"2023-11-16T16:47:26.385895+0000","last_active":"2023-11-16T16:47:44.858440+0000","last_peered":"2023-11-16T16:47:44.858440+0000","last_clean":"2023-11-16T16:47:44.858440+0000","last_became_active":"2023-11-16T16:47:26.385580+0000","last_became_peered":"2023-11-16T16:47:26.385580+0000","last_unstale":"2023-11-16T16:47:44.858440+0000","last_undegraded":"2023-11-16T16:47:44.858440+0000","last_fullsized":"2023-11-16T16:47:44.858440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:03:39.204040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856629+0000","last_change":"2023-11-16T16:47:26.373785+0000","last_active":"2023-11-16T16:47:44.856629+0000","last_peered":"2023-11-16T16:47:44.856629+0000","last_clean":"2023-11-16T16:47:44.856629+0000","last_became_active":"2023-11-16T16:47:26.373425+0000","last_became_peered":"2023-11-16T16:47:26.373425+0000","last_unstale":"2023-11-16T16:47:44.856629+0000","last_undegraded":"2023-11-16T16:47:44.856629+0000","last_fullsized":"2023-11-16T16:47:44.856629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:16:02.226398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472287+0000","last_change":"2023-11-16T16:47:26.374722+0000","last_active":"2023-11-16T16:47:45.472287+0000","last_peered":"2023-11-16T16:47:45.472287+0000","last_clean":"2023-11-16T16:47:45.472287+0000","last_became_active":"2023-11-16T16:47:26.374326+0000","last_became_peered":"2023-11-16T16:47:26.374326+0000","last_unstale":"2023-11-16T16:47:45.472287+0000","last_undegraded":"2023-11-16T16:47:45.472287+0000","last_fullsized":"2023-11-16T16:47:45.472287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:52:18.596419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":249745,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":25267,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":28,"num_read_kb":41,"num_write":121,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":74727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106,"ondisk_log_size":106,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383697,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6156,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738996,"statfs":{"total":95995035648,"available":95988731904,"internally_reserved":0,"allocated":987136,"data_stored":495639,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.56999999999999995}]},{"osd":1,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.68700000000000006}]},{"osd":2,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.75700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":4,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.76800000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.345,"5min":1.345,"15min":1.345},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.69899999999999995}]},{"osd":6,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.91500000000000004}]},{"osd":7,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.878},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.84699999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":9,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.90400000000000003}]},{"osd":10,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.86199999999999999}]}]},{"osd":10,"up_from":72,"seq":309237645334,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6520,"kb_used_data":1072,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738632,"statfs":{"total":95995035648,"available":95988359168,"internally_reserved":0,"allocated":1097728,"data_stored":1108771,"data_compressed":53789,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":1,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.39600000000000002}]},{"osd":2,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.68100000000000005}]},{"osd":3,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.65500000000000003}]},{"osd":4,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.45500000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.71699999999999997}]},{"osd":6,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.69599999999999995}]},{"osd":7,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.46999999999999997}]},{"osd":8,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.78100000000000003}]},{"osd":9,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.67000000000000004}]},{"osd":11,"last update":"Thu Nov 16 16:47:38 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.79400000000000004}]}]},{"osd":9,"up_from":67,"seq":287762808859,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6368,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738784,"statfs":{"total":95995035648,"available":95988514816,"internally_reserved":0,"allocated":1007616,"data_stored":496457,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.45500000000000002}]},{"osd":1,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.72699999999999998}]},{"osd":2,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.401,"5min":1.401,"15min":1.401},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.71599999999999997}]},{"osd":3,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.59699999999999998}]},{"osd":4,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.77400000000000002}]},{"osd":6,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.629}]},{"osd":7,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.624},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.85099999999999998}]},{"osd":8,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.87}]},{"osd":10,"last update":"Thu Nov 16 16:47:17 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.78800000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]}]},{"osd":8,"up_from":59,"seq":253403070495,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6484,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738668,"statfs":{"total":95995035648,"available":95988396032,"internally_reserved":0,"allocated":995328,"data_stored":495934,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.373,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70199999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.76800000000000002,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.47099999999999997}]},{"osd":1,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58999999999999997,"15min":0.59099999999999997},"min":{"1min":0.40400000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.83199999999999996,"5min":1.089,"15min":1.089},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.71699999999999997}]},{"osd":2,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.47499999999999998,"15min":0.47899999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73499999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.47199999999999998,"15min":0.47299999999999998},"min":{"1min":0.32500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.497}]},{"osd":3,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.626,"15min":0.63700000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73899999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.66900000000000004,"15min":0.68200000000000005},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.81000000000000005,"5min":1.115,"15min":1.115},"last":0.66900000000000004}]},{"osd":4,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66000000000000003,"15min":0.66800000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76000000000000001,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.53700000000000003}]},{"osd":5,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.59599999999999997,"15min":0.60599999999999998},"min":{"1min":0.33000000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.83199999999999996,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.378,"5min":0.37,"15min":0.37},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56899999999999995}]},{"osd":6,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.73699999999999999,"15min":0.751},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.80400000000000005,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.625},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.76600000000000001,"15min":0.78200000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.92300000000000004,"5min":1.093,"15min":1.093},"last":0.76000000000000001}]},{"osd":7,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.72199999999999998,"15min":0.72799999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.88600000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69299999999999995,"15min":0.69399999999999995},"min":{"1min":0.46200000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.90200000000000002,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.73599999999999999}]},{"osd":9,"last update":"Thu Nov 16 16:46:54 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":0.75}]},{"osd":10,"last update":"Thu Nov 16 16:47:17 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.77700000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]}]},{"osd":7,"up_from":51,"seq":219043332131,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6604,"kb_used_data":1028,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738548,"statfs":{"total":95995035648,"available":95988273152,"internally_reserved":0,"allocated":1052672,"data_stored":1087703,"data_compressed":53789,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56200000000000006,"15min":0.55500000000000005},"min":{"1min":0.34899999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.56799999999999995,"15min":0.55600000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.48699999999999999}]},{"osd":1,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.27400000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.61399999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.46500000000000002,"15min":0.45800000000000002},"min":{"1min":0.316,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.42999999999999999}]},{"osd":2,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57799999999999996,"15min":0.57399999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.496}]},{"osd":3,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.62,"15min":0.61899999999999999},"min":{"1min":0.435,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.59799999999999998,"15min":0.58799999999999997},"min":{"1min":0.48299999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.54100000000000004}]},{"osd":4,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.505,"15min":0.496},"min":{"1min":0.36199999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.51300000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69399999999999995,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.45700000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.80400000000000005,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.62,"15min":0.61299999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.61699999999999999}]},{"osd":6,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.73599999999999999,"15min":0.70399999999999996},"min":{"1min":0.59799999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":4.8840000000000003,"5min":4.8840000000000003,"15min":4.8840000000000003},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.68700000000000006,"15min":0.68000000000000005},"min":{"1min":0.55300000000000005,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.63800000000000001}]},{"osd":8,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.85599999999999998}]},{"osd":9,"last update":"Thu Nov 16 16:46:54 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.68300000000000005}]},{"osd":10,"last update":"Thu Nov 16 16:47:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.71799999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]}]},{"osd":6,"up_from":46,"seq":197568495656,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6476,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738676,"statfs":{"total":95995035648,"available":95988404224,"internally_reserved":0,"allocated":987136,"data_stored":495639,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.49099999999999999,"15min":0.46700000000000003},"min":{"1min":0.312,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.71799999999999997,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55400000000000005,"15min":0.53400000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.81899999999999995,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.58599999999999997}]},{"osd":1,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.69499999999999995,"15min":0.66000000000000003},"min":{"1min":0.50600000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.032,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.433,"5min":0.378,"15min":0.378},"max":{"1min":0.90000000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.69399999999999995}]},{"osd":2,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.72899999999999998,"15min":0.70999999999999996},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0589999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.74199999999999999,"15min":0.71999999999999997},"min":{"1min":0.56100000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.008,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.91000000000000003}]},{"osd":3,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.68799999999999994,"15min":0.66800000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60599999999999998,"15min":0.60199999999999998},"min":{"1min":0.36399999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.90600000000000003,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.61299999999999999}]},{"osd":4,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.73799999999999999},"min":{"1min":0.47799999999999998,"5min":0.44,"15min":0.44},"max":{"1min":1,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.748},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.79600000000000004,"15min":0.78100000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.1819999999999999,"5min":1.3,"15min":1.3},"last":0.88200000000000001}]},{"osd":5,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.73199999999999998,"15min":0.73099999999999998},"min":{"1min":0.50800000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.147,"5min":1.206,"15min":1.206},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80600000000000005,"15min":0.78800000000000003},"min":{"1min":0.58299999999999996,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.139,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.89400000000000002}]},{"osd":7,"last update":"Thu Nov 16 16:47:11 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.71999999999999997,"15min":0.70599999999999996},"min":{"1min":0.53500000000000003,"5min":0.499,"15min":0.499},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.81999999999999995,"15min":0.80400000000000005},"min":{"1min":0.59499999999999997,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.21,"5min":1.246,"15min":1.246},"last":0.84999999999999998}]},{"osd":8,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91500000000000004,"15min":0.91600000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.234,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.61599999999999999,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.268,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.96599999999999997}]},{"osd":9,"last update":"Thu Nov 16 16:46:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.95299999999999996}]},{"osd":10,"last update":"Thu Nov 16 16:47:16 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.018,"5min":1.018,"15min":1.018},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":1.006}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7384,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737768,"statfs":{"total":95995035648,"available":95987474432,"internally_reserved":0,"allocated":999424,"data_stored":499433,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:50 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.53000000000000003,"15min":0.51600000000000001},"min":{"1min":0.34699999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.71099999999999997,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.318},{"interface":"front","average":{"1min":0.495,"5min":0.52300000000000002,"15min":0.498},"min":{"1min":0.35199999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.68400000000000005,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.35199999999999998}]},{"osd":2,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.52600000000000002,"15min":0.51100000000000001},"min":{"1min":0.38100000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.42199999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.5229999999999999,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":0.49399999999999999}]},{"osd":3,"last update":"Thu Nov 16 16:47:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.58399999999999996,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72799999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.61299999999999999,"15min":0.60899999999999999},"min":{"1min":0.44,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.79700000000000004,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.58499999999999996}]},{"osd":4,"last update":"Thu Nov 16 16:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.41999999999999998,"15min":0.40699999999999997},"min":{"1min":0.32700000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.498,"5min":0.45600000000000002,"15min":0.433},"min":{"1min":0.30599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.66100000000000003}]},{"osd":5,"last update":"Thu Nov 16 16:47:20 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.58099999999999996,"15min":0.56000000000000005},"min":{"1min":0.46899999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.60899999999999999,"15min":0.58099999999999996},"min":{"1min":0.52500000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.61599999999999999}]},{"osd":6,"last update":"Thu Nov 16 16:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66800000000000004,"15min":0.67000000000000004},"min":{"1min":0.45700000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.94499999999999995,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.68899999999999995},"min":{"1min":0.47099999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.91300000000000003,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.69399999999999995}]},{"osd":7,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81100000000000005,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.38800000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":0.56899999999999995}]},{"osd":8,"last update":"Thu Nov 16 16:47:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.64700000000000002,"15min":0.63600000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.69299999999999995,"15min":0.68999999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.94099999999999995,"5min":1,"15min":1},"last":0.79000000000000004}]},{"osd":9,"last update":"Thu Nov 16 16:46:54 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.651,"5min":1.651,"15min":1.651},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.68200000000000005}]},{"osd":10,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.67000000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7376,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737776,"statfs":{"total":95995035648,"available":95987482624,"internally_reserved":0,"allocated":991232,"data_stored":495729,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 16:46:56 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.57399999999999995,"15min":0.52600000000000002},"min":{"1min":0.46600000000000003,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.57799999999999996,"15min":0.53300000000000003},"min":{"1min":0.443,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.91200000000000003}]},{"osd":2,"last update":"Thu Nov 16 16:47:11 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.60399999999999998,"15min":0.58599999999999997},"min":{"1min":0.42899999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.628,"15min":0.58599999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.3479999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":0.51400000000000001}]},{"osd":3,"last update":"Thu Nov 16 16:46:31 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.53200000000000003,"15min":0.499},"min":{"1min":0.35499999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.58799999999999997,"15min":0.56599999999999995},"min":{"1min":0.39200000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.75700000000000001}]},{"osd":4,"last update":"Thu Nov 16 16:46:58 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.65200000000000002,"15min":0.61699999999999999},"min":{"1min":0.45800000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.1709999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.621},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.65500000000000003,"15min":0.63300000000000001},"min":{"1min":0.40600000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.87}]},{"osd":5,"last update":"Thu Nov 16 16:47:19 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.66100000000000003,"15min":0.64800000000000002},"min":{"1min":0.43099999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.71499999999999997,"15min":0.69499999999999995},"min":{"1min":0.53100000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":0.66400000000000003}]},{"osd":6,"last update":"Thu Nov 16 16:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.69199999999999995,"15min":0.67400000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73399999999999999,"15min":0.70899999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.64200000000000002}]},{"osd":7,"last update":"Thu Nov 16 16:47:08 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.77600000000000002,"15min":0.749},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.78300000000000003,"15min":0.75600000000000001},"min":{"1min":0.57999999999999996,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":2.1859999999999999,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":0.92800000000000005}]},{"osd":8,"last update":"Thu Nov 16 16:47:29 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.80700000000000005,"15min":0.81100000000000005},"min":{"1min":0.51700000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.145,"5min":2.145,"15min":2.145},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.871,"5min":0.82599999999999996,"15min":0.81799999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.125,"5min":2.125,"15min":2.125},"last":0.80300000000000005}]},{"osd":9,"last update":"Thu Nov 16 16:46:56 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.83499999999999996}]},{"osd":10,"last update":"Thu Nov 16 16:47:15 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.3159999999999998,"5min":2.3159999999999998,"15min":2.3159999999999998},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.96499999999999997}]},{"osd":11,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":2.2189999999999999,"5min":2.2189999999999999,"15min":2.2189999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":0.97999999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7484,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737668,"statfs":{"total":95995035648,"available":95987372032,"internally_reserved":0,"allocated":1036288,"data_stored":517564,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.48499999999999999,"15min":0.46400000000000002},"min":{"1min":0.314,"5min":0.249,"15min":0.249},"max":{"1min":0.80100000000000005,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.49399999999999999,"15min":0.48099999999999998},"min":{"1min":0.35299999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77400000000000002,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.45600000000000002}]},{"osd":1,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51000000000000001,"15min":0.5},"min":{"1min":0.36499999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67800000000000005,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.52100000000000002,"15min":0.503},"min":{"1min":0.41499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70399999999999996,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.39500000000000002}]},{"osd":3,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58199999999999996,"15min":0.56000000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.84599999999999997,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.59399999999999997,"15min":0.57399999999999995},"min":{"1min":0.34300000000000003,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.85499999999999998,"5min":1.016,"15min":1.016},"last":0.625}]},{"osd":4,"last update":"Thu Nov 16 16:47:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.53200000000000003,"15min":0.51800000000000002},"min":{"1min":0.442,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73899999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55000000000000004,"15min":0.54200000000000004},"min":{"1min":0.38400000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.70899999999999996,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.59799999999999998}]},{"osd":5,"last update":"Thu Nov 16 16:47:17 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.41499999999999998,"15min":0.40600000000000003},"min":{"1min":0.28599999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.372},{"interface":"front","average":{"1min":0.501,"5min":0.47899999999999998,"15min":0.46600000000000003},"min":{"1min":0.33400000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.64400000000000002}]},{"osd":6,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.63800000000000001,"15min":0.61599999999999999},"min":{"1min":0.56599999999999995,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.91100000000000003,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.64400000000000002,"15min":0.63300000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.88500000000000001,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.56000000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:47:05 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77200000000000002,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.503},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.66700000000000004,"15min":0.66900000000000004},"min":{"1min":0.51200000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.74099999999999999}]},{"osd":8,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.89800000000000002,"5min":0.997,"15min":0.997},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.72899999999999998}]},{"osd":9,"last update":"Thu Nov 16 16:46:52 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.67400000000000004}]},{"osd":10,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.83199999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6740,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738412,"statfs":{"total":95995035648,"available":95988133888,"internally_reserved":0,"allocated":995328,"data_stored":495743,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.433,"15min":0.40300000000000002},"min":{"1min":0.33900000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.62},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.436,"15min":0.41399999999999998},"min":{"1min":0.38300000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.65700000000000003}]},{"osd":1,"last update":"Thu Nov 16 16:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.63400000000000001,"15min":0.60599999999999998},"min":{"1min":0.41899999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.64600000000000002,"15min":0.60799999999999998},"min":{"1min":0.46500000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.92400000000000004,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":1.028}]},{"osd":2,"last update":"Thu Nov 16 16:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.625},"min":{"1min":0.42699999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89800000000000002,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.63800000000000001,"15min":0.61499999999999999},"min":{"1min":0.45400000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.81699999999999995,"5min":0.871,"15min":0.871},"last":0.78300000000000003}]},{"osd":4,"last update":"Thu Nov 16 16:47:04 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.70299999999999996,"15min":0.70999999999999996},"min":{"1min":0.58599999999999997,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.74199999999999999,"15min":0.73999999999999999},"min":{"1min":0.64400000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.98399999999999999}]},{"osd":5,"last update":"Thu Nov 16 16:47:22 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.70799999999999996,"15min":0.70699999999999996},"min":{"1min":0.57999999999999996,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.88600000000000001,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":1.1060000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.70899999999999996,"15min":0.71699999999999997},"min":{"1min":0.53100000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.879}]},{"osd":6,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61199999999999999,"15min":0.60999999999999999},"min":{"1min":0.42699999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.495,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":1.155}]},{"osd":7,"last update":"Thu Nov 16 16:47:11 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.747,"15min":0.73099999999999998},"min":{"1min":0.60799999999999998,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1,"5min":1,"15min":1},"last":1.1930000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.71899999999999997,"15min":0.71599999999999997},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.90900000000000003}]},{"osd":8,"last update":"Thu Nov 16 16:46:26 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.81299999999999994}]},{"osd":9,"last update":"Thu Nov 16 16:46:53 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":1.2210000000000001}]},{"osd":10,"last update":"Thu Nov 16 16:47:17 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":1.2450000000000001},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":1.008}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]}]},{"osd":4,"up_from":33,"seq":141733920818,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6616,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738536,"statfs":{"total":95995035648,"available":95988260864,"internally_reserved":0,"allocated":999424,"data_stored":496625,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.504,"15min":0.49099999999999999},"min":{"1min":0.41299999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.35299999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.41699999999999998}]},{"osd":1,"last update":"Thu Nov 16 16:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.37,"15min":0.35399999999999998},"min":{"1min":0.245,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.39600000000000002,"15min":0.38400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.71299999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.38800000000000001}]},{"osd":2,"last update":"Thu Nov 16 16:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55000000000000004,"15min":0.54400000000000004},"min":{"1min":0.38400000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.68799999999999994,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53200000000000003,"15min":0.52700000000000002},"min":{"1min":0.38400000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.70599999999999996}]},{"osd":3,"last update":"Thu Nov 16 16:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.56000000000000005,"15min":0.54200000000000004},"min":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.62},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.56799999999999995,"15min":0.55400000000000005},"min":{"1min":0.372,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.44800000000000001}]},{"osd":5,"last update":"Thu Nov 16 16:47:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.56299999999999994,"15min":0.54300000000000004},"min":{"1min":0.438,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78900000000000003,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.56799999999999995,"15min":0.54700000000000004},"min":{"1min":0.434,"5min":0.375,"15min":0.375},"max":{"1min":0.78000000000000003,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.55800000000000005}]},{"osd":6,"last update":"Thu Nov 16 16:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61699999999999999,"15min":0.60699999999999998},"min":{"1min":0.47899999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.84399999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66200000000000003,"15min":0.65100000000000002},"min":{"1min":0.49399999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.88500000000000001,"5min":0.995,"15min":0.995},"last":0.53500000000000003}]},{"osd":7,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.52000000000000002,"15min":0.51000000000000001},"min":{"1min":0.30499999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.76000000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.60599999999999998,"15min":0.59699999999999998},"min":{"1min":0.46400000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.50800000000000001}]},{"osd":8,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.51600000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.86899999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68000000000000005,"15min":0.67800000000000005},"min":{"1min":0.50600000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.68500000000000005}]},{"osd":9,"last update":"Thu Nov 16 16:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.747},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.64100000000000001}]},{"osd":10,"last update":"Thu Nov 16 16:47:16 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.69199999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6744,"kb_used_data":1040,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738408,"statfs":{"total":95995035648,"available":95988129792,"internally_reserved":0,"allocated":1064960,"data_stored":1091590,"data_compressed":53789,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:21 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.52400000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.84399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52100000000000002,"15min":0.51700000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Nov 16 16:47:21 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.50600000000000001,"15min":0.48399999999999999},"min":{"1min":0.377,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80900000000000005,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.51300000000000001,"15min":0.49399999999999999},"min":{"1min":0.38200000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.73599999999999999,"5min":0.749,"15min":0.749},"last":0.79000000000000004}]},{"osd":2,"last update":"Thu Nov 16 16:47:21 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40300000000000002,"15min":0.39300000000000002},"min":{"1min":0.30499999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.52500000000000002,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.5},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.41499999999999998,"15min":0.40100000000000002},"min":{"1min":0.29299999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.81799999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.53600000000000003}]},{"osd":3,"last update":"Thu Nov 16 16:47:21 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.40400000000000003,"5min":0.371,"15min":0.371},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.629},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.56200000000000006,"15min":0.55300000000000005},"min":{"1min":0.44400000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.58299999999999996}]},{"osd":4,"last update":"Thu Nov 16 16:47:21 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57399999999999995,"15min":0.55100000000000005},"min":{"1min":0.432,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58799999999999997,"15min":0.56399999999999995},"min":{"1min":0.45700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.81599999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.82599999999999996}]},{"osd":6,"last update":"Thu Nov 16 16:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.626,"15min":0.60999999999999999},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.82699999999999996,"5min":1.052,"15min":1.052},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.63100000000000001,"15min":0.61699999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.86899999999999999,"5min":1.117,"15min":1.117},"last":0.65300000000000002}]},{"osd":7,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.58799999999999997,"15min":0.58499999999999996},"min":{"1min":0.40500000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.747}]},{"osd":8,"last update":"Thu Nov 16 16:47:29 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.61499999999999999,"15min":0.61399999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.84399999999999997,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.84999999999999998}]},{"osd":9,"last update":"Thu Nov 16 16:47:52 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.73999999999999999,"15min":0.73799999999999999},"min":{"1min":0.53100000000000003,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.94499999999999995,"5min":1.145,"15min":1.145},"last":0.878},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.74399999999999999,"15min":0.745},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.89900000000000002,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.80800000000000005}]},{"osd":10,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.86299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34505,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34505,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":21175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":22867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":23095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T16:47:56.388 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-16T16:47:56.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.620+0000 7fc5181ff700 1 -- 172.21.15.2:0/550121082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc510100290 msgr2=0x7fc5101006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:56.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.620+0000 7fc5181ff700 1 --2- 172.21.15.2:0/550121082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc510100290 0x7fc5101006a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc50c009990 tx=0x7fc50c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.620+0000 7fc5181ff700 1 -- 172.21.15.2:0/550121082 shutdown_connections 2023-11-16T16:47:56.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.620+0000 7fc5181ff700 1 --2- 172.21.15.2:0/550121082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc510101e10 0x7fc5101086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.620+0000 7fc5181ff700 1 --2- 172.21.15.2:0/550121082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc510100290 0x7fc5101006a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.620+0000 7fc5181ff700 1 --2- 172.21.15.2:0/550121082 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc510101480 0x7fc5101018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.620+0000 7fc5181ff700 1 -- 172.21.15.2:0/550121082 >> 172.21.15.2:0/550121082 conn(0x7fc5100fb870 msgr2=0x7fc5100fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:56.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.620+0000 7fc5181ff700 1 -- 172.21.15.2:0/550121082 shutdown_connections 2023-11-16T16:47:56.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.620+0000 7fc5181ff700 1 -- 172.21.15.2:0/550121082 wait complete. 2023-11-16T16:47:56.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.621+0000 7fc5181ff700 1 Processor -- start 2023-11-16T16:47:56.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.621+0000 7fc5181ff700 1 -- start start 2023-11-16T16:47:56.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.621+0000 7fc5181ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc510100290 0x7fc5100748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:56.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.621+0000 7fc5181ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc510101480 0x7fc510072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:56.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.622+0000 7fc5181ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc510101e10 0x7fc510073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:56.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.622+0000 7fc5181ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc510073bc0 con 0x7fc510101e10 2023-11-16T16:47:56.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.622+0000 7fc5181ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc510073d30 con 0x7fc510100290 2023-11-16T16:47:56.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.622+0000 7fc5181ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5101a0070 con 0x7fc510101480 2023-11-16T16:47:56.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.622+0000 7fc515f9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc510100290 0x7fc5100748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:56.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.622+0000 7fc515f9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc510100290 0x7fc5100748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:38760/0 (socket says 172.21.15.2:38760) 2023-11-16T16:47:56.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.622+0000 7fc515f9b700 1 -- 172.21.15.2:0/3506548071 learned_addr learned my addr 172.21.15.2:0/3506548071 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:56.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.622+0000 7fc51679c700 1 --2- 172.21.15.2:0/3506548071 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc510101e10 0x7fc510073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:56.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.622+0000 7fc51579a700 1 --2- 172.21.15.2:0/3506548071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc510101480 0x7fc510072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:56.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.622+0000 7fc515f9b700 1 -- 172.21.15.2:0/3506548071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc510101480 msgr2=0x7fc510072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:56.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.622+0000 7fc515f9b700 1 --2- 172.21.15.2:0/3506548071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc510101480 0x7fc510072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.622+0000 7fc515f9b700 1 -- 172.21.15.2:0/3506548071 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc510101e10 msgr2=0x7fc510073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:56.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.622+0000 7fc515f9b700 1 --2- 172.21.15.2:0/3506548071 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc510101e10 0x7fc510073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.622+0000 7fc515f9b700 1 -- 172.21.15.2:0/3506548071 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc50c0095f0 con 0x7fc510100290 2023-11-16T16:47:56.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.622+0000 7fc51679c700 1 --2- 172.21.15.2:0/3506548071 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc510101e10 0x7fc510073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:47:56.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.622+0000 7fc515f9b700 1 --2- 172.21.15.2:0/3506548071 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc510100290 0x7fc5100748a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00b620 tx=0x7fc50c005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:56.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.623+0000 7fc506ffd700 1 -- 172.21.15.2:0/3506548071 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c01d070 con 0x7fc510100290 2023-11-16T16:47:56.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.623+0000 7fc5181ff700 1 -- 172.21.15.2:0/3506548071 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc51010bc40 con 0x7fc510100290 2023-11-16T16:47:56.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.623+0000 7fc506ffd700 1 -- 172.21.15.2:0/3506548071 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc50c021410 con 0x7fc510100290 2023-11-16T16:47:56.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.623+0000 7fc506ffd700 1 -- 172.21.15.2:0/3506548071 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c017650 con 0x7fc510100290 2023-11-16T16:47:56.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.623+0000 7fc5181ff700 1 -- 172.21.15.2:0/3506548071 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc51010bab0 con 0x7fc510100290 2023-11-16T16:47:56.631 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.624+0000 7fc5181ff700 1 -- 172.21.15.2:0/3506548071 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc510107870 con 0x7fc510100290 2023-11-16T16:47:56.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.626+0000 7fc506ffd700 1 -- 172.21.15.2:0/3506548071 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fc50c021580 con 0x7fc510100290 2023-11-16T16:47:56.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.627+0000 7fc506ffd700 1 --2- 172.21.15.2:0/3506548071 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc4fc0a11e0 0x7fc4fc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:56.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.627+0000 7fc506ffd700 1 -- 172.21.15.2:0/3506548071 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc50c0ce920 con 0x7fc510100290 2023-11-16T16:47:56.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.628+0000 7fc51579a700 1 --2- 172.21.15.2:0/3506548071 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc4fc0a11e0 0x7fc4fc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:56.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.628+0000 7fc51579a700 1 --2- 172.21.15.2:0/3506548071 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc4fc0a11e0 0x7fc4fc0a3690 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc500009c60 tx=0x7fc500009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:56.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.629+0000 7fc506ffd700 1 -- 172.21.15.2:0/3506548071 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc50c097c30 con 0x7fc510100290 2023-11-16T16:47:56.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.833+0000 7fc5181ff700 1 -- 172.21.15.2:0/3506548071 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc510060f30 con 0x7fc4fc0a11e0 2023-11-16T16:47:56.848 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.846+0000 7fc506ffd700 1 -- 172.21.15.2:0/3506548071 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422659 (secure 0 0 0) 0x7fc510060f30 con 0x7fc4fc0a11e0 2023-11-16T16:47:56.856 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:47:56.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.852+0000 7fc5181ff700 1 -- 172.21.15.2:0/3506548071 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc4fc0a11e0 msgr2=0x7fc4fc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:56.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.852+0000 7fc5181ff700 1 --2- 172.21.15.2:0/3506548071 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc4fc0a11e0 0x7fc4fc0a3690 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc500009c60 tx=0x7fc500009260 comp rx=0 tx=0).stop 2023-11-16T16:47:56.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.852+0000 7fc5181ff700 1 -- 172.21.15.2:0/3506548071 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc510100290 msgr2=0x7fc5100748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:56.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.852+0000 7fc5181ff700 1 --2- 172.21.15.2:0/3506548071 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc510100290 0x7fc5100748a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00b620 tx=0x7fc50c005740 comp rx=0 tx=0).stop 2023-11-16T16:47:56.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.852+0000 7fc5181ff700 1 -- 172.21.15.2:0/3506548071 shutdown_connections 2023-11-16T16:47:56.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.852+0000 7fc5181ff700 1 --2- 172.21.15.2:0/3506548071 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc4fc0a11e0 0x7fc4fc0a3690 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.852+0000 7fc5181ff700 1 --2- 172.21.15.2:0/3506548071 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc510101e10 0x7fc510073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.852+0000 7fc5181ff700 1 --2- 172.21.15.2:0/3506548071 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc510100290 0x7fc5100748a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.852+0000 7fc5181ff700 1 --2- 172.21.15.2:0/3506548071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc510101480 0x7fc510072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:56.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.852+0000 7fc5181ff700 1 -- 172.21.15.2:0/3506548071 >> 172.21.15.2:0/3506548071 conn(0x7fc5100fb870 msgr2=0x7fc5101049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:56.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.852+0000 7fc5181ff700 1 -- 172.21.15.2:0/3506548071 shutdown_connections 2023-11-16T16:47:56.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:56.852+0000 7fc5181ff700 1 -- 172.21.15.2:0/3506548071 wait complete. 2023-11-16T16:47:56.864 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-16T16:47:56.873 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2023-11-16T16:47:55.793582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":615881,"num_objects":55,"num_object_clones":0,"num_object_copies":167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":164,"num_read_kb":160,"num_write":304,"num_write_kb":2995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":245,"ondisk_log_size":245,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":80952,"kb_used_data":11928,"kb_used_omap":0,"kb_used_meta":68928,"kb_avail":1124860872,"statfs":{"total":1151940427776,"available":1151857532928,"internally_reserved":0,"allocated":12214272,"data_stored":7776827,"data_compressed":320145,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":0,"internal_metadata":70582272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21062,"num_objects":31,"num_object_clones":0,"num_object_copies":95,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":28,"num_read_kb":41,"num_write":75,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.941846"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472424+0000","last_change":"2023-11-16T16:47:26.367598+0000","last_active":"2023-11-16T16:47:45.472424+0000","last_peered":"2023-11-16T16:47:45.472424+0000","last_clean":"2023-11-16T16:47:45.472424+0000","last_became_active":"2023-11-16T16:47:26.366233+0000","last_became_peered":"2023-11-16T16:47:26.366233+0000","last_unstale":"2023-11-16T16:47:45.472424+0000","last_undegraded":"2023-11-16T16:47:45.472424+0000","last_fullsized":"2023-11-16T16:47:45.472424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:27:47.751645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.861310+0000","last_change":"2023-11-16T16:47:26.371984+0000","last_active":"2023-11-16T16:47:44.861310+0000","last_peered":"2023-11-16T16:47:44.861310+0000","last_clean":"2023-11-16T16:47:44.861310+0000","last_became_active":"2023-11-16T16:47:26.371628+0000","last_became_peered":"2023-11-16T16:47:26.371628+0000","last_unstale":"2023-11-16T16:47:44.861310+0000","last_undegraded":"2023-11-16T16:47:44.861310+0000","last_fullsized":"2023-11-16T16:47:44.861310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:07:37.389666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.860205+0000","last_change":"2023-11-16T16:47:26.374109+0000","last_active":"2023-11-16T16:47:44.860205+0000","last_peered":"2023-11-16T16:47:44.860205+0000","last_clean":"2023-11-16T16:47:44.860205+0000","last_became_active":"2023-11-16T16:47:26.373759+0000","last_became_peered":"2023-11-16T16:47:26.373759+0000","last_unstale":"2023-11-16T16:47:44.860205+0000","last_undegraded":"2023-11-16T16:47:44.860205+0000","last_fullsized":"2023-11-16T16:47:44.860205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:38:53.188578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855484+0000","last_change":"2023-11-16T16:47:26.378404+0000","last_active":"2023-11-16T16:47:44.855484+0000","last_peered":"2023-11-16T16:47:44.855484+0000","last_clean":"2023-11-16T16:47:44.855484+0000","last_became_active":"2023-11-16T16:47:26.378067+0000","last_became_peered":"2023-11-16T16:47:26.378067+0000","last_unstale":"2023-11-16T16:47:44.855484+0000","last_undegraded":"2023-11-16T16:47:44.855484+0000","last_fullsized":"2023-11-16T16:47:44.855484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:58:13.078678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856409+0000","last_change":"2023-11-16T16:47:26.370066+0000","last_active":"2023-11-16T16:47:44.856409+0000","last_peered":"2023-11-16T16:47:44.856409+0000","last_clean":"2023-11-16T16:47:44.856409+0000","last_became_active":"2023-11-16T16:47:26.369789+0000","last_became_peered":"2023-11-16T16:47:26.369789+0000","last_unstale":"2023-11-16T16:47:44.856409+0000","last_undegraded":"2023-11-16T16:47:44.856409+0000","last_fullsized":"2023-11-16T16:47:44.856409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:33:38.139992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472727+0000","last_change":"2023-11-16T16:47:26.379403+0000","last_active":"2023-11-16T16:47:45.472727+0000","last_peered":"2023-11-16T16:47:45.472727+0000","last_clean":"2023-11-16T16:47:45.472727+0000","last_became_active":"2023-11-16T16:47:26.379086+0000","last_became_peered":"2023-11-16T16:47:26.379086+0000","last_unstale":"2023-11-16T16:47:45.472727+0000","last_undegraded":"2023-11-16T16:47:45.472727+0000","last_fullsized":"2023-11-16T16:47:45.472727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:20:51.896082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857929+0000","last_change":"2023-11-16T16:47:26.377555+0000","last_active":"2023-11-16T16:47:44.857929+0000","last_peered":"2023-11-16T16:47:44.857929+0000","last_clean":"2023-11-16T16:47:44.857929+0000","last_became_active":"2023-11-16T16:47:26.377274+0000","last_became_peered":"2023-11-16T16:47:26.377274+0000","last_unstale":"2023-11-16T16:47:44.857929+0000","last_undegraded":"2023-11-16T16:47:44.857929+0000","last_fullsized":"2023-11-16T16:47:44.857929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:40:11.254785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.412630+0000","last_change":"2023-11-16T16:47:26.380842+0000","last_active":"2023-11-16T16:47:45.412630+0000","last_peered":"2023-11-16T16:47:45.412630+0000","last_clean":"2023-11-16T16:47:45.412630+0000","last_became_active":"2023-11-16T16:47:26.380527+0000","last_became_peered":"2023-11-16T16:47:26.380527+0000","last_unstale":"2023-11-16T16:47:45.412630+0000","last_undegraded":"2023-11-16T16:47:45.412630+0000","last_fullsized":"2023-11-16T16:47:45.412630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:56:19.663855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858099+0000","last_change":"2023-11-16T16:47:26.371416+0000","last_active":"2023-11-16T16:47:44.858099+0000","last_peered":"2023-11-16T16:47:44.858099+0000","last_clean":"2023-11-16T16:47:44.858099+0000","last_became_active":"2023-11-16T16:47:26.371096+0000","last_became_peered":"2023-11-16T16:47:26.371096+0000","last_unstale":"2023-11-16T16:47:44.858099+0000","last_undegraded":"2023-11-16T16:47:44.858099+0000","last_fullsized":"2023-11-16T16:47:44.858099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:35:21.284705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473525+0000","last_change":"2023-11-16T16:47:26.380828+0000","last_active":"2023-11-16T16:47:45.473525+0000","last_peered":"2023-11-16T16:47:45.473525+0000","last_clean":"2023-11-16T16:47:45.473525+0000","last_became_active":"2023-11-16T16:47:26.377495+0000","last_became_peered":"2023-11-16T16:47:26.377495+0000","last_unstale":"2023-11-16T16:47:45.473525+0000","last_undegraded":"2023-11-16T16:47:45.473525+0000","last_fullsized":"2023-11-16T16:47:45.473525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:58:53.606878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.859142+0000","last_change":"2023-11-16T16:47:26.374127+0000","last_active":"2023-11-16T16:47:44.859142+0000","last_peered":"2023-11-16T16:47:44.859142+0000","last_clean":"2023-11-16T16:47:44.859142+0000","last_became_active":"2023-11-16T16:47:26.373794+0000","last_became_peered":"2023-11-16T16:47:26.373794+0000","last_unstale":"2023-11-16T16:47:44.859142+0000","last_undegraded":"2023-11-16T16:47:44.859142+0000","last_fullsized":"2023-11-16T16:47:44.859142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:39:54.776347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855500+0000","last_change":"2023-11-16T16:47:26.382530+0000","last_active":"2023-11-16T16:47:44.855500+0000","last_peered":"2023-11-16T16:47:44.855500+0000","last_clean":"2023-11-16T16:47:44.855500+0000","last_became_active":"2023-11-16T16:47:26.382229+0000","last_became_peered":"2023-11-16T16:47:26.382229+0000","last_unstale":"2023-11-16T16:47:44.855500+0000","last_undegraded":"2023-11-16T16:47:44.855500+0000","last_fullsized":"2023-11-16T16:47:44.855500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:28:54.780902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473174+0000","last_change":"2023-11-16T16:47:45.473174+0000","last_active":"2023-11-16T16:47:45.473174+0000","last_peered":"2023-11-16T16:47:45.473174+0000","last_clean":"2023-11-16T16:47:45.473174+0000","last_became_active":"2023-11-16T16:47:45.472853+0000","last_became_peered":"2023-11-16T16:47:45.472853+0000","last_unstale":"2023-11-16T16:47:45.473174+0000","last_undegraded":"2023-11-16T16:47:45.473174+0000","last_fullsized":"2023-11-16T16:47:45.473174+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:57:05.934853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.859123+0000","last_change":"2023-11-16T16:47:26.375546+0000","last_active":"2023-11-16T16:47:44.859123+0000","last_peered":"2023-11-16T16:47:44.859123+0000","last_clean":"2023-11-16T16:47:44.859123+0000","last_became_active":"2023-11-16T16:47:26.375218+0000","last_became_peered":"2023-11-16T16:47:26.375218+0000","last_unstale":"2023-11-16T16:47:44.859123+0000","last_undegraded":"2023-11-16T16:47:44.859123+0000","last_fullsized":"2023-11-16T16:47:44.859123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:27:43.795280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.860204+0000","last_change":"2023-11-16T16:47:44.860204+0000","last_active":"2023-11-16T16:47:44.860204+0000","last_peered":"2023-11-16T16:47:44.860204+0000","last_clean":"2023-11-16T16:47:44.860204+0000","last_became_active":"2023-11-16T16:47:44.859473+0000","last_became_peered":"2023-11-16T16:47:44.859473+0000","last_unstale":"2023-11-16T16:47:44.860204+0000","last_undegraded":"2023-11-16T16:47:44.860204+0000","last_fullsized":"2023-11-16T16:47:44.860204+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:57:23.491160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857085+0000","last_change":"2023-11-16T16:47:26.379604+0000","last_active":"2023-11-16T16:47:44.857085+0000","last_peered":"2023-11-16T16:47:44.857085+0000","last_clean":"2023-11-16T16:47:44.857085+0000","last_became_active":"2023-11-16T16:47:26.379225+0000","last_became_peered":"2023-11-16T16:47:26.379225+0000","last_unstale":"2023-11-16T16:47:44.857085+0000","last_undegraded":"2023-11-16T16:47:44.857085+0000","last_fullsized":"2023-11-16T16:47:44.857085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:26:54.929229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472135+0000","last_change":"2023-11-16T16:47:26.367373+0000","last_active":"2023-11-16T16:47:45.472135+0000","last_peered":"2023-11-16T16:47:45.472135+0000","last_clean":"2023-11-16T16:47:45.472135+0000","last_became_active":"2023-11-16T16:47:26.366549+0000","last_became_peered":"2023-11-16T16:47:26.366549+0000","last_unstale":"2023-11-16T16:47:45.472135+0000","last_undegraded":"2023-11-16T16:47:45.472135+0000","last_fullsized":"2023-11-16T16:47:45.472135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:54:59.079284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854006+0000","last_change":"2023-11-16T16:47:26.376291+0000","last_active":"2023-11-16T16:47:44.854006+0000","last_peered":"2023-11-16T16:47:44.854006+0000","last_clean":"2023-11-16T16:47:44.854006+0000","last_became_active":"2023-11-16T16:47:26.374608+0000","last_became_peered":"2023-11-16T16:47:26.374608+0000","last_unstale":"2023-11-16T16:47:44.854006+0000","last_undegraded":"2023-11-16T16:47:44.854006+0000","last_fullsized":"2023-11-16T16:47:44.854006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:17:25.971629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855424+0000","last_change":"2023-11-16T16:47:26.382444+0000","last_active":"2023-11-16T16:47:44.855424+0000","last_peered":"2023-11-16T16:47:44.855424+0000","last_clean":"2023-11-16T16:47:44.855424+0000","last_became_active":"2023-11-16T16:47:26.382130+0000","last_became_peered":"2023-11-16T16:47:26.382130+0000","last_unstale":"2023-11-16T16:47:44.855424+0000","last_undegraded":"2023-11-16T16:47:44.855424+0000","last_fullsized":"2023-11-16T16:47:44.855424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:34.208412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854988+0000","last_change":"2023-11-16T16:47:30.838569+0000","last_active":"2023-11-16T16:47:44.854988+0000","last_peered":"2023-11-16T16:47:44.854988+0000","last_clean":"2023-11-16T16:47:44.854988+0000","last_became_active":"2023-11-16T16:47:30.838282+0000","last_became_peered":"2023-11-16T16:47:30.838282+0000","last_unstale":"2023-11-16T16:47:44.854988+0000","last_undegraded":"2023-11-16T16:47:44.854988+0000","last_fullsized":"2023-11-16T16:47:44.854988+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:06:09.584975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.907867+0000","last_change":"2023-11-16T16:47:25.820254+0000","last_active":"2023-11-16T16:47:51.907867+0000","last_peered":"2023-11-16T16:47:51.907867+0000","last_clean":"2023-11-16T16:47:51.907867+0000","last_became_active":"2023-11-16T16:47:25.819924+0000","last_became_peered":"2023-11-16T16:47:25.819924+0000","last_unstale":"2023-11-16T16:47:51.907867+0000","last_undegraded":"2023-11-16T16:47:51.907867+0000","last_fullsized":"2023-11-16T16:47:51.907867+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:50:37.572954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.413299+0000","last_change":"2023-11-16T16:47:26.374560+0000","last_active":"2023-11-16T16:47:45.413299+0000","last_peered":"2023-11-16T16:47:45.413299+0000","last_clean":"2023-11-16T16:47:45.413299+0000","last_became_active":"2023-11-16T16:47:26.373554+0000","last_became_peered":"2023-11-16T16:47:26.373554+0000","last_unstale":"2023-11-16T16:47:45.413299+0000","last_undegraded":"2023-11-16T16:47:45.413299+0000","last_fullsized":"2023-11-16T16:47:45.413299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:33:15.911693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854083+0000","last_change":"2023-11-16T16:47:30.826047+0000","last_active":"2023-11-16T16:47:44.854083+0000","last_peered":"2023-11-16T16:47:44.854083+0000","last_clean":"2023-11-16T16:47:44.854083+0000","last_became_active":"2023-11-16T16:47:30.825696+0000","last_became_peered":"2023-11-16T16:47:30.825696+0000","last_unstale":"2023-11-16T16:47:44.854083+0000","last_undegraded":"2023-11-16T16:47:44.854083+0000","last_fullsized":"2023-11-16T16:47:44.854083+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:31:58.343449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.413521+0000","last_change":"2023-11-16T16:47:25.830763+0000","last_active":"2023-11-16T16:47:45.413521+0000","last_peered":"2023-11-16T16:47:45.413521+0000","last_clean":"2023-11-16T16:47:45.413521+0000","last_became_active":"2023-11-16T16:47:25.830405+0000","last_became_peered":"2023-11-16T16:47:25.830405+0000","last_unstale":"2023-11-16T16:47:45.413521+0000","last_undegraded":"2023-11-16T16:47:45.413521+0000","last_fullsized":"2023-11-16T16:47:45.413521+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:39:02.087710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.859518+0000","last_change":"2023-11-16T16:47:26.385897+0000","last_active":"2023-11-16T16:47:44.859518+0000","last_peered":"2023-11-16T16:47:44.859518+0000","last_clean":"2023-11-16T16:47:44.859518+0000","last_became_active":"2023-11-16T16:47:26.382557+0000","last_became_peered":"2023-11-16T16:47:26.382557+0000","last_unstale":"2023-11-16T16:47:44.859518+0000","last_undegraded":"2023-11-16T16:47:44.859518+0000","last_fullsized":"2023-11-16T16:47:44.859518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:19:59.689853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854309+0000","last_change":"2023-11-16T16:47:30.828521+0000","last_active":"2023-11-16T16:47:44.854309+0000","last_peered":"2023-11-16T16:47:44.854309+0000","last_clean":"2023-11-16T16:47:44.854309+0000","last_became_active":"2023-11-16T16:47:30.828135+0000","last_became_peered":"2023-11-16T16:47:30.828135+0000","last_unstale":"2023-11-16T16:47:44.854309+0000","last_undegraded":"2023-11-16T16:47:44.854309+0000","last_fullsized":"2023-11-16T16:47:44.854309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:19:57.561985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857191+0000","last_change":"2023-11-16T16:47:25.819576+0000","last_active":"2023-11-16T16:47:44.857191+0000","last_peered":"2023-11-16T16:47:44.857191+0000","last_clean":"2023-11-16T16:47:44.857191+0000","last_became_active":"2023-11-16T16:47:25.818354+0000","last_became_peered":"2023-11-16T16:47:25.818354+0000","last_unstale":"2023-11-16T16:47:44.857191+0000","last_undegraded":"2023-11-16T16:47:44.857191+0000","last_fullsized":"2023-11-16T16:47:44.857191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:43:47.079077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857839+0000","last_change":"2023-11-16T16:47:26.364632+0000","last_active":"2023-11-16T16:47:44.857839+0000","last_peered":"2023-11-16T16:47:44.857839+0000","last_clean":"2023-11-16T16:47:44.857839+0000","last_became_active":"2023-11-16T16:47:26.364288+0000","last_became_peered":"2023-11-16T16:47:26.364288+0000","last_unstale":"2023-11-16T16:47:44.857839+0000","last_undegraded":"2023-11-16T16:47:44.857839+0000","last_fullsized":"2023-11-16T16:47:44.857839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:17:52.757571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854072+0000","last_change":"2023-11-16T16:47:30.828722+0000","last_active":"2023-11-16T16:47:44.854072+0000","last_peered":"2023-11-16T16:47:44.854072+0000","last_clean":"2023-11-16T16:47:44.854072+0000","last_became_active":"2023-11-16T16:47:30.828299+0000","last_became_peered":"2023-11-16T16:47:30.828299+0000","last_unstale":"2023-11-16T16:47:44.854072+0000","last_undegraded":"2023-11-16T16:47:44.854072+0000","last_fullsized":"2023-11-16T16:47:44.854072+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:33:38.194438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856600+0000","last_change":"2023-11-16T16:47:25.817030+0000","last_active":"2023-11-16T16:47:44.856600+0000","last_peered":"2023-11-16T16:47:44.856600+0000","last_clean":"2023-11-16T16:47:44.856600+0000","last_became_active":"2023-11-16T16:47:25.816710+0000","last_became_peered":"2023-11-16T16:47:25.816710+0000","last_unstale":"2023-11-16T16:47:44.856600+0000","last_undegraded":"2023-11-16T16:47:44.856600+0000","last_fullsized":"2023-11-16T16:47:44.856600+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:40:22.859236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473673+0000","last_change":"2023-11-16T16:47:26.371226+0000","last_active":"2023-11-16T16:47:45.473673+0000","last_peered":"2023-11-16T16:47:45.473673+0000","last_clean":"2023-11-16T16:47:45.473673+0000","last_became_active":"2023-11-16T16:47:26.370925+0000","last_became_peered":"2023-11-16T16:47:26.370925+0000","last_unstale":"2023-11-16T16:47:45.473673+0000","last_undegraded":"2023-11-16T16:47:45.473673+0000","last_fullsized":"2023-11-16T16:47:45.473673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:48:12.574242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855221+0000","last_change":"2023-11-16T16:47:30.821364+0000","last_active":"2023-11-16T16:47:44.855221+0000","last_peered":"2023-11-16T16:47:44.855221+0000","last_clean":"2023-11-16T16:47:44.855221+0000","last_became_active":"2023-11-16T16:47:30.820938+0000","last_became_peered":"2023-11-16T16:47:30.820938+0000","last_unstale":"2023-11-16T16:47:44.855221+0000","last_undegraded":"2023-11-16T16:47:44.855221+0000","last_fullsized":"2023-11-16T16:47:44.855221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:13:28.692462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'6","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:53.919052+0000","last_change":"2023-11-16T16:47:25.818612+0000","last_active":"2023-11-16T16:47:53.919052+0000","last_peered":"2023-11-16T16:47:53.919052+0000","last_clean":"2023-11-16T16:47:53.919052+0000","last_became_active":"2023-11-16T16:47:25.818272+0000","last_became_peered":"2023-11-16T16:47:25.818272+0000","last_unstale":"2023-11-16T16:47:53.919052+0000","last_undegraded":"2023-11-16T16:47:53.919052+0000","last_fullsized":"2023-11-16T16:47:53.919052+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:00:33.166271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857267+0000","last_change":"2023-11-16T16:47:26.372245+0000","last_active":"2023-11-16T16:47:44.857267+0000","last_peered":"2023-11-16T16:47:44.857267+0000","last_clean":"2023-11-16T16:47:44.857267+0000","last_became_active":"2023-11-16T16:47:26.371942+0000","last_became_peered":"2023-11-16T16:47:26.371942+0000","last_unstale":"2023-11-16T16:47:44.857267+0000","last_undegraded":"2023-11-16T16:47:44.857267+0000","last_fullsized":"2023-11-16T16:47:44.857267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:25:33.613660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855696+0000","last_change":"2023-11-16T16:47:30.839669+0000","last_active":"2023-11-16T16:47:44.855696+0000","last_peered":"2023-11-16T16:47:44.855696+0000","last_clean":"2023-11-16T16:47:44.855696+0000","last_became_active":"2023-11-16T16:47:30.839165+0000","last_became_peered":"2023-11-16T16:47:30.839165+0000","last_unstale":"2023-11-16T16:47:44.855696+0000","last_undegraded":"2023-11-16T16:47:44.855696+0000","last_fullsized":"2023-11-16T16:47:44.855696+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:26:12.940109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.905121+0000","last_change":"2023-11-16T16:47:25.819867+0000","last_active":"2023-11-16T16:47:51.905121+0000","last_peered":"2023-11-16T16:47:51.905121+0000","last_clean":"2023-11-16T16:47:51.905121+0000","last_became_active":"2023-11-16T16:47:25.819516+0000","last_became_peered":"2023-11-16T16:47:25.819516+0000","last_unstale":"2023-11-16T16:47:51.905121+0000","last_undegraded":"2023-11-16T16:47:51.905121+0000","last_fullsized":"2023-11-16T16:47:51.905121+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:23:15.640449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473687+0000","last_change":"2023-11-16T16:47:26.383679+0000","last_active":"2023-11-16T16:47:45.473687+0000","last_peered":"2023-11-16T16:47:45.473687+0000","last_clean":"2023-11-16T16:47:45.473687+0000","last_became_active":"2023-11-16T16:47:26.383370+0000","last_became_peered":"2023-11-16T16:47:26.383370+0000","last_unstale":"2023-11-16T16:47:45.473687+0000","last_undegraded":"2023-11-16T16:47:45.473687+0000","last_fullsized":"2023-11-16T16:47:45.473687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:56:40.615856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472127+0000","last_change":"2023-11-16T16:47:30.832845+0000","last_active":"2023-11-16T16:47:45.472127+0000","last_peered":"2023-11-16T16:47:45.472127+0000","last_clean":"2023-11-16T16:47:45.472127+0000","last_became_active":"2023-11-16T16:47:30.832364+0000","last_became_peered":"2023-11-16T16:47:30.832364+0000","last_unstale":"2023-11-16T16:47:45.472127+0000","last_undegraded":"2023-11-16T16:47:45.472127+0000","last_fullsized":"2023-11-16T16:47:45.472127+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:41:14.777870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856301+0000","last_change":"2023-11-16T16:47:25.817420+0000","last_active":"2023-11-16T16:47:44.856301+0000","last_peered":"2023-11-16T16:47:44.856301+0000","last_clean":"2023-11-16T16:47:44.856301+0000","last_became_active":"2023-11-16T16:47:25.816160+0000","last_became_peered":"2023-11-16T16:47:25.816160+0000","last_unstale":"2023-11-16T16:47:44.856301+0000","last_undegraded":"2023-11-16T16:47:44.856301+0000","last_fullsized":"2023-11-16T16:47:44.856301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:14:09.295340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.413125+0000","last_change":"2023-11-16T16:47:26.370729+0000","last_active":"2023-11-16T16:47:45.413125+0000","last_peered":"2023-11-16T16:47:45.413125+0000","last_clean":"2023-11-16T16:47:45.413125+0000","last_became_active":"2023-11-16T16:47:26.370435+0000","last_became_peered":"2023-11-16T16:47:26.370435+0000","last_unstale":"2023-11-16T16:47:45.413125+0000","last_undegraded":"2023-11-16T16:47:45.413125+0000","last_fullsized":"2023-11-16T16:47:45.413125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:37:45.259435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.861176+0000","last_change":"2023-11-16T16:47:30.832895+0000","last_active":"2023-11-16T16:47:44.861176+0000","last_peered":"2023-11-16T16:47:44.861176+0000","last_clean":"2023-11-16T16:47:44.861176+0000","last_became_active":"2023-11-16T16:47:30.832192+0000","last_became_peered":"2023-11-16T16:47:30.832192+0000","last_unstale":"2023-11-16T16:47:44.861176+0000","last_undegraded":"2023-11-16T16:47:44.861176+0000","last_fullsized":"2023-11-16T16:47:44.861176+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:30:54.337765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473152+0000","last_change":"2023-11-16T16:47:25.815673+0000","last_active":"2023-11-16T16:47:45.473152+0000","last_peered":"2023-11-16T16:47:45.473152+0000","last_clean":"2023-11-16T16:47:45.473152+0000","last_became_active":"2023-11-16T16:47:25.815362+0000","last_became_peered":"2023-11-16T16:47:25.815362+0000","last_unstale":"2023-11-16T16:47:45.473152+0000","last_undegraded":"2023-11-16T16:47:45.473152+0000","last_fullsized":"2023-11-16T16:47:45.473152+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:50:06.110785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.853033+0000","last_change":"2023-11-16T16:47:26.383483+0000","last_active":"2023-11-16T16:47:44.853033+0000","last_peered":"2023-11-16T16:47:44.853033+0000","last_clean":"2023-11-16T16:47:44.853033+0000","last_became_active":"2023-11-16T16:47:26.383177+0000","last_became_peered":"2023-11-16T16:47:26.383177+0000","last_unstale":"2023-11-16T16:47:44.853033+0000","last_undegraded":"2023-11-16T16:47:44.853033+0000","last_fullsized":"2023-11-16T16:47:44.853033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:27:38.101289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.413695+0000","last_change":"2023-11-16T16:47:30.829044+0000","last_active":"2023-11-16T16:47:45.413695+0000","last_peered":"2023-11-16T16:47:45.413695+0000","last_clean":"2023-11-16T16:47:45.413695+0000","last_became_active":"2023-11-16T16:47:30.827398+0000","last_became_peered":"2023-11-16T16:47:30.827398+0000","last_unstale":"2023-11-16T16:47:45.413695+0000","last_undegraded":"2023-11-16T16:47:45.413695+0000","last_fullsized":"2023-11-16T16:47:45.413695+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:54:57.131841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:52.912794+0000","last_change":"2023-11-16T16:47:25.817461+0000","last_active":"2023-11-16T16:47:52.912794+0000","last_peered":"2023-11-16T16:47:52.912794+0000","last_clean":"2023-11-16T16:47:52.912794+0000","last_became_active":"2023-11-16T16:47:25.817135+0000","last_became_peered":"2023-11-16T16:47:25.817135+0000","last_unstale":"2023-11-16T16:47:52.912794+0000","last_undegraded":"2023-11-16T16:47:52.912794+0000","last_fullsized":"2023-11-16T16:47:52.912794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:00:21.140226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858098+0000","last_change":"2023-11-16T16:47:26.380218+0000","last_active":"2023-11-16T16:47:44.858098+0000","last_peered":"2023-11-16T16:47:44.858098+0000","last_clean":"2023-11-16T16:47:44.858098+0000","last_became_active":"2023-11-16T16:47:26.379849+0000","last_became_peered":"2023-11-16T16:47:26.379849+0000","last_unstale":"2023-11-16T16:47:44.858098+0000","last_undegraded":"2023-11-16T16:47:44.858098+0000","last_fullsized":"2023-11-16T16:47:44.858098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:16:44.031031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.852960+0000","last_change":"2023-11-16T16:47:30.836464+0000","last_active":"2023-11-16T16:47:44.852960+0000","last_peered":"2023-11-16T16:47:44.852960+0000","last_clean":"2023-11-16T16:47:44.852960+0000","last_became_active":"2023-11-16T16:47:30.835185+0000","last_became_peered":"2023-11-16T16:47:30.835185+0000","last_unstale":"2023-11-16T16:47:44.852960+0000","last_undegraded":"2023-11-16T16:47:44.852960+0000","last_fullsized":"2023-11-16T16:47:44.852960+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:00:34.542077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.898666+0000","last_change":"2023-11-16T16:47:25.816583+0000","last_active":"2023-11-16T16:47:50.898666+0000","last_peered":"2023-11-16T16:47:50.898666+0000","last_clean":"2023-11-16T16:47:50.898666+0000","last_became_active":"2023-11-16T16:47:25.816286+0000","last_became_peered":"2023-11-16T16:47:25.816286+0000","last_unstale":"2023-11-16T16:47:50.898666+0000","last_undegraded":"2023-11-16T16:47:50.898666+0000","last_fullsized":"2023-11-16T16:47:50.898666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:55:47.335416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.860328+0000","last_change":"2023-11-16T16:47:26.379964+0000","last_active":"2023-11-16T16:47:44.860328+0000","last_peered":"2023-11-16T16:47:44.860328+0000","last_clean":"2023-11-16T16:47:44.860328+0000","last_became_active":"2023-11-16T16:47:26.376857+0000","last_became_peered":"2023-11-16T16:47:26.376857+0000","last_unstale":"2023-11-16T16:47:44.860328+0000","last_undegraded":"2023-11-16T16:47:44.860328+0000","last_fullsized":"2023-11-16T16:47:44.860328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:44:33.471523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854625+0000","last_change":"2023-11-16T16:47:30.832572+0000","last_active":"2023-11-16T16:47:44.854625+0000","last_peered":"2023-11-16T16:47:44.854625+0000","last_clean":"2023-11-16T16:47:44.854625+0000","last_became_active":"2023-11-16T16:47:30.832138+0000","last_became_peered":"2023-11-16T16:47:30.832138+0000","last_unstale":"2023-11-16T16:47:44.854625+0000","last_undegraded":"2023-11-16T16:47:44.854625+0000","last_fullsized":"2023-11-16T16:47:44.854625+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:25:20.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.900690+0000","last_change":"2023-11-16T16:47:25.826848+0000","last_active":"2023-11-16T16:47:50.900690+0000","last_peered":"2023-11-16T16:47:50.900690+0000","last_clean":"2023-11-16T16:47:50.900690+0000","last_became_active":"2023-11-16T16:47:25.826532+0000","last_became_peered":"2023-11-16T16:47:25.826532+0000","last_unstale":"2023-11-16T16:47:50.900690+0000","last_undegraded":"2023-11-16T16:47:50.900690+0000","last_fullsized":"2023-11-16T16:47:50.900690+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:34:36.051545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.863649+0000","last_change":"2023-11-16T16:47:44.863649+0000","last_active":"2023-11-16T16:47:44.863649+0000","last_peered":"2023-11-16T16:47:44.863649+0000","last_clean":"2023-11-16T16:47:44.863649+0000","last_became_active":"2023-11-16T16:47:44.863350+0000","last_became_peered":"2023-11-16T16:47:44.863350+0000","last_unstale":"2023-11-16T16:47:44.863649+0000","last_undegraded":"2023-11-16T16:47:44.863649+0000","last_fullsized":"2023-11-16T16:47:44.863649+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:45:32.381225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.853019+0000","last_change":"2023-11-16T16:47:30.829074+0000","last_active":"2023-11-16T16:47:44.853019+0000","last_peered":"2023-11-16T16:47:44.853019+0000","last_clean":"2023-11-16T16:47:44.853019+0000","last_became_active":"2023-11-16T16:47:30.828688+0000","last_became_peered":"2023-11-16T16:47:30.828688+0000","last_unstale":"2023-11-16T16:47:44.853019+0000","last_undegraded":"2023-11-16T16:47:44.853019+0000","last_fullsized":"2023-11-16T16:47:44.853019+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:56:56.319865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'18","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:53.917768+0000","last_change":"2023-11-16T16:47:25.816469+0000","last_active":"2023-11-16T16:47:53.917768+0000","last_peered":"2023-11-16T16:47:53.917768+0000","last_clean":"2023-11-16T16:47:53.917768+0000","last_became_active":"2023-11-16T16:47:25.816115+0000","last_became_peered":"2023-11-16T16:47:25.816115+0000","last_unstale":"2023-11-16T16:47:53.917768+0000","last_undegraded":"2023-11-16T16:47:53.917768+0000","last_fullsized":"2023-11-16T16:47:53.917768+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:33:32.434023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21175,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":24,"num_write":19,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856961+0000","last_change":"2023-11-16T16:47:26.373382+0000","last_active":"2023-11-16T16:47:44.856961+0000","last_peered":"2023-11-16T16:47:44.856961+0000","last_clean":"2023-11-16T16:47:44.856961+0000","last_became_active":"2023-11-16T16:47:26.372951+0000","last_became_peered":"2023-11-16T16:47:26.372951+0000","last_unstale":"2023-11-16T16:47:44.856961+0000","last_undegraded":"2023-11-16T16:47:44.856961+0000","last_fullsized":"2023-11-16T16:47:44.856961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:29:19.977662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473826+0000","last_change":"2023-11-16T16:47:30.833890+0000","last_active":"2023-11-16T16:47:45.473826+0000","last_peered":"2023-11-16T16:47:45.473826+0000","last_clean":"2023-11-16T16:47:45.473826+0000","last_became_active":"2023-11-16T16:47:30.833489+0000","last_became_peered":"2023-11-16T16:47:30.833489+0000","last_unstale":"2023-11-16T16:47:45.473826+0000","last_undegraded":"2023-11-16T16:47:45.473826+0000","last_fullsized":"2023-11-16T16:47:45.473826+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:21:14.794417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.905467+0000","last_change":"2023-11-16T16:47:25.815950+0000","last_active":"2023-11-16T16:47:51.905467+0000","last_peered":"2023-11-16T16:47:51.905467+0000","last_clean":"2023-11-16T16:47:51.905467+0000","last_became_active":"2023-11-16T16:47:25.815657+0000","last_became_peered":"2023-11-16T16:47:25.815657+0000","last_unstale":"2023-11-16T16:47:51.905467+0000","last_undegraded":"2023-11-16T16:47:51.905467+0000","last_fullsized":"2023-11-16T16:47:51.905467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:31:06.425915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856308+0000","last_change":"2023-11-16T16:47:26.378190+0000","last_active":"2023-11-16T16:47:44.856308+0000","last_peered":"2023-11-16T16:47:44.856308+0000","last_clean":"2023-11-16T16:47:44.856308+0000","last_became_active":"2023-11-16T16:47:26.374581+0000","last_became_peered":"2023-11-16T16:47:26.374581+0000","last_unstale":"2023-11-16T16:47:44.856308+0000","last_undegraded":"2023-11-16T16:47:44.856308+0000","last_fullsized":"2023-11-16T16:47:44.856308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:22:26.437085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.471899+0000","last_change":"2023-11-16T16:47:30.826637+0000","last_active":"2023-11-16T16:47:45.471899+0000","last_peered":"2023-11-16T16:47:45.471899+0000","last_clean":"2023-11-16T16:47:45.471899+0000","last_became_active":"2023-11-16T16:47:30.826156+0000","last_became_peered":"2023-11-16T16:47:30.826156+0000","last_unstale":"2023-11-16T16:47:45.471899+0000","last_undegraded":"2023-11-16T16:47:45.471899+0000","last_fullsized":"2023-11-16T16:47:45.471899+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:01:29.047562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.901236+0000","last_change":"2023-11-16T16:47:25.822749+0000","last_active":"2023-11-16T16:47:50.901236+0000","last_peered":"2023-11-16T16:47:50.901236+0000","last_clean":"2023-11-16T16:47:50.901236+0000","last_became_active":"2023-11-16T16:47:25.822441+0000","last_became_peered":"2023-11-16T16:47:25.822441+0000","last_unstale":"2023-11-16T16:47:50.901236+0000","last_undegraded":"2023-11-16T16:47:50.901236+0000","last_fullsized":"2023-11-16T16:47:50.901236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:42:27.484194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858444+0000","last_change":"2023-11-16T16:47:26.375131+0000","last_active":"2023-11-16T16:47:44.858444+0000","last_peered":"2023-11-16T16:47:44.858444+0000","last_clean":"2023-11-16T16:47:44.858444+0000","last_became_active":"2023-11-16T16:47:26.374740+0000","last_became_peered":"2023-11-16T16:47:26.374740+0000","last_unstale":"2023-11-16T16:47:44.858444+0000","last_undegraded":"2023-11-16T16:47:44.858444+0000","last_fullsized":"2023-11-16T16:47:44.858444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:58:23.761900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.853864+0000","last_change":"2023-11-16T16:47:30.836806+0000","last_active":"2023-11-16T16:47:44.853864+0000","last_peered":"2023-11-16T16:47:44.853864+0000","last_clean":"2023-11-16T16:47:44.853864+0000","last_became_active":"2023-11-16T16:47:30.835299+0000","last_became_peered":"2023-11-16T16:47:30.835299+0000","last_unstale":"2023-11-16T16:47:44.853864+0000","last_undegraded":"2023-11-16T16:47:44.853864+0000","last_fullsized":"2023-11-16T16:47:44.853864+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:53:26.843235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:53.919284+0000","last_change":"2023-11-16T16:47:25.816793+0000","last_active":"2023-11-16T16:47:53.919284+0000","last_peered":"2023-11-16T16:47:53.919284+0000","last_clean":"2023-11-16T16:47:53.919284+0000","last_became_active":"2023-11-16T16:47:25.816433+0000","last_became_peered":"2023-11-16T16:47:25.816433+0000","last_unstale":"2023-11-16T16:47:53.919284+0000","last_undegraded":"2023-11-16T16:47:53.919284+0000","last_fullsized":"2023-11-16T16:47:53.919284+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:30:24.755449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858132+0000","last_change":"2023-11-16T16:47:26.375383+0000","last_active":"2023-11-16T16:47:44.858132+0000","last_peered":"2023-11-16T16:47:44.858132+0000","last_clean":"2023-11-16T16:47:44.858132+0000","last_became_active":"2023-11-16T16:47:26.375026+0000","last_became_peered":"2023-11-16T16:47:26.375026+0000","last_unstale":"2023-11-16T16:47:44.858132+0000","last_undegraded":"2023-11-16T16:47:44.858132+0000","last_fullsized":"2023-11-16T16:47:44.858132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:07:01.547781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.475115+0000","last_change":"2023-11-16T16:47:45.475115+0000","last_active":"2023-11-16T16:47:45.475115+0000","last_peered":"2023-11-16T16:47:45.475115+0000","last_clean":"2023-11-16T16:47:45.475115+0000","last_became_active":"2023-11-16T16:47:45.474475+0000","last_became_peered":"2023-11-16T16:47:45.474475+0000","last_unstale":"2023-11-16T16:47:45.475115+0000","last_undegraded":"2023-11-16T16:47:45.475115+0000","last_fullsized":"2023-11-16T16:47:45.475115+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:14:57.456154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.898156+0000","last_change":"2023-11-16T16:47:25.826911+0000","last_active":"2023-11-16T16:47:50.898156+0000","last_peered":"2023-11-16T16:47:50.898156+0000","last_clean":"2023-11-16T16:47:50.898156+0000","last_became_active":"2023-11-16T16:47:25.826605+0000","last_became_peered":"2023-11-16T16:47:25.826605+0000","last_unstale":"2023-11-16T16:47:50.898156+0000","last_undegraded":"2023-11-16T16:47:50.898156+0000","last_fullsized":"2023-11-16T16:47:50.898156+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:43:53.441571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472956+0000","last_change":"2023-11-16T16:47:26.376072+0000","last_active":"2023-11-16T16:47:45.472956+0000","last_peered":"2023-11-16T16:47:45.472956+0000","last_clean":"2023-11-16T16:47:45.472956+0000","last_became_active":"2023-11-16T16:47:26.375585+0000","last_became_peered":"2023-11-16T16:47:26.375585+0000","last_unstale":"2023-11-16T16:47:45.472956+0000","last_undegraded":"2023-11-16T16:47:45.472956+0000","last_fullsized":"2023-11-16T16:47:45.472956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:30:44.970814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472838+0000","last_change":"2023-11-16T16:47:30.834558+0000","last_active":"2023-11-16T16:47:45.472838+0000","last_peered":"2023-11-16T16:47:45.472838+0000","last_clean":"2023-11-16T16:47:45.472838+0000","last_became_active":"2023-11-16T16:47:30.833989+0000","last_became_peered":"2023-11-16T16:47:30.833989+0000","last_unstale":"2023-11-16T16:47:45.472838+0000","last_undegraded":"2023-11-16T16:47:45.472838+0000","last_fullsized":"2023-11-16T16:47:45.472838+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:54:42.983565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857593+0000","last_change":"2023-11-16T16:47:25.817016+0000","last_active":"2023-11-16T16:47:44.857593+0000","last_peered":"2023-11-16T16:47:44.857593+0000","last_clean":"2023-11-16T16:47:44.857593+0000","last_became_active":"2023-11-16T16:47:25.816752+0000","last_became_peered":"2023-11-16T16:47:25.816752+0000","last_unstale":"2023-11-16T16:47:44.857593+0000","last_undegraded":"2023-11-16T16:47:44.857593+0000","last_fullsized":"2023-11-16T16:47:44.857593+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:40:44.076735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472022+0000","last_change":"2023-11-16T16:47:26.374947+0000","last_active":"2023-11-16T16:47:45.472022+0000","last_peered":"2023-11-16T16:47:45.472022+0000","last_clean":"2023-11-16T16:47:45.472022+0000","last_became_active":"2023-11-16T16:47:26.374533+0000","last_became_peered":"2023-11-16T16:47:26.374533+0000","last_unstale":"2023-11-16T16:47:45.472022+0000","last_undegraded":"2023-11-16T16:47:45.472022+0000","last_fullsized":"2023-11-16T16:47:45.472022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:58:16.020607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856695+0000","last_change":"2023-11-16T16:47:30.830506+0000","last_active":"2023-11-16T16:47:44.856695+0000","last_peered":"2023-11-16T16:47:44.856695+0000","last_clean":"2023-11-16T16:47:44.856695+0000","last_became_active":"2023-11-16T16:47:30.830086+0000","last_became_peered":"2023-11-16T16:47:30.830086+0000","last_unstale":"2023-11-16T16:47:44.856695+0000","last_undegraded":"2023-11-16T16:47:44.856695+0000","last_fullsized":"2023-11-16T16:47:44.856695+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:50:36.980481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.853657+0000","last_change":"2023-11-16T16:47:25.830104+0000","last_active":"2023-11-16T16:47:44.853657+0000","last_peered":"2023-11-16T16:47:44.853657+0000","last_clean":"2023-11-16T16:47:44.853657+0000","last_became_active":"2023-11-16T16:47:25.829232+0000","last_became_peered":"2023-11-16T16:47:25.829232+0000","last_unstale":"2023-11-16T16:47:44.853657+0000","last_undegraded":"2023-11-16T16:47:44.853657+0000","last_fullsized":"2023-11-16T16:47:44.853657+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:29:15.724324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473108+0000","last_change":"2023-11-16T16:47:26.368918+0000","last_active":"2023-11-16T16:47:45.473108+0000","last_peered":"2023-11-16T16:47:45.473108+0000","last_clean":"2023-11-16T16:47:45.473108+0000","last_became_active":"2023-11-16T16:47:26.368592+0000","last_became_peered":"2023-11-16T16:47:26.368592+0000","last_unstale":"2023-11-16T16:47:45.473108+0000","last_undegraded":"2023-11-16T16:47:45.473108+0000","last_fullsized":"2023-11-16T16:47:45.473108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:09:36.266152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857755+0000","last_change":"2023-11-16T16:47:30.830558+0000","last_active":"2023-11-16T16:47:44.857755+0000","last_peered":"2023-11-16T16:47:44.857755+0000","last_clean":"2023-11-16T16:47:44.857755+0000","last_became_active":"2023-11-16T16:47:30.829777+0000","last_became_peered":"2023-11-16T16:47:30.829777+0000","last_unstale":"2023-11-16T16:47:44.857755+0000","last_undegraded":"2023-11-16T16:47:44.857755+0000","last_fullsized":"2023-11-16T16:47:44.857755+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:15:47.182273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858408+0000","last_change":"2023-11-16T16:47:25.820315+0000","last_active":"2023-11-16T16:47:44.858408+0000","last_peered":"2023-11-16T16:47:44.858408+0000","last_clean":"2023-11-16T16:47:44.858408+0000","last_became_active":"2023-11-16T16:47:25.820032+0000","last_became_peered":"2023-11-16T16:47:25.820032+0000","last_unstale":"2023-11-16T16:47:44.858408+0000","last_undegraded":"2023-11-16T16:47:44.858408+0000","last_fullsized":"2023-11-16T16:47:44.858408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:36:13.332902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472754+0000","last_change":"2023-11-16T16:47:26.379841+0000","last_active":"2023-11-16T16:47:45.472754+0000","last_peered":"2023-11-16T16:47:45.472754+0000","last_clean":"2023-11-16T16:47:45.472754+0000","last_became_active":"2023-11-16T16:47:26.377401+0000","last_became_peered":"2023-11-16T16:47:26.377401+0000","last_unstale":"2023-11-16T16:47:45.472754+0000","last_undegraded":"2023-11-16T16:47:45.472754+0000","last_fullsized":"2023-11-16T16:47:45.472754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:21:55.430094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473595+0000","last_change":"2023-11-16T16:47:45.473595+0000","last_active":"2023-11-16T16:47:45.473595+0000","last_peered":"2023-11-16T16:47:45.473595+0000","last_clean":"2023-11-16T16:47:45.473595+0000","last_became_active":"2023-11-16T16:47:45.473100+0000","last_became_peered":"2023-11-16T16:47:45.473100+0000","last_unstale":"2023-11-16T16:47:45.473595+0000","last_undegraded":"2023-11-16T16:47:45.473595+0000","last_fullsized":"2023-11-16T16:47:45.473595+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:17:45.050424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473340+0000","last_change":"2023-11-16T16:47:26.364112+0000","last_active":"2023-11-16T16:47:45.473340+0000","last_peered":"2023-11-16T16:47:45.473340+0000","last_clean":"2023-11-16T16:47:45.473340+0000","last_became_active":"2023-11-16T16:47:26.363767+0000","last_became_peered":"2023-11-16T16:47:26.363767+0000","last_unstale":"2023-11-16T16:47:45.473340+0000","last_undegraded":"2023-11-16T16:47:45.473340+0000","last_fullsized":"2023-11-16T16:47:45.473340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:01:40.809234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856188+0000","last_change":"2023-11-16T16:47:25.830012+0000","last_active":"2023-11-16T16:47:44.856188+0000","last_peered":"2023-11-16T16:47:44.856188+0000","last_clean":"2023-11-16T16:47:44.856188+0000","last_became_active":"2023-11-16T16:47:25.829474+0000","last_became_peered":"2023-11-16T16:47:25.829474+0000","last_unstale":"2023-11-16T16:47:44.856188+0000","last_undegraded":"2023-11-16T16:47:44.856188+0000","last_fullsized":"2023-11-16T16:47:44.856188+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:01:21.296554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858885+0000","last_change":"2023-11-16T16:47:30.832724+0000","last_active":"2023-11-16T16:47:44.858885+0000","last_peered":"2023-11-16T16:47:44.858885+0000","last_clean":"2023-11-16T16:47:44.858885+0000","last_became_active":"2023-11-16T16:47:30.832226+0000","last_became_peered":"2023-11-16T16:47:30.832226+0000","last_unstale":"2023-11-16T16:47:44.858885+0000","last_undegraded":"2023-11-16T16:47:44.858885+0000","last_fullsized":"2023-11-16T16:47:44.858885+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:11:15.234460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.476312+0000","last_change":"2023-11-16T16:47:45.474859+0000","last_active":"2023-11-16T16:47:45.476312+0000","last_peered":"2023-11-16T16:47:45.476312+0000","last_clean":"2023-11-16T16:47:45.476312+0000","last_became_active":"2023-11-16T16:47:45.474439+0000","last_became_peered":"2023-11-16T16:47:45.474439+0000","last_unstale":"2023-11-16T16:47:45.476312+0000","last_undegraded":"2023-11-16T16:47:45.476312+0000","last_fullsized":"2023-11-16T16:47:45.476312+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:05:04.346537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:52.910430+0000","last_change":"2023-11-16T16:47:25.813894+0000","last_active":"2023-11-16T16:47:52.910430+0000","last_peered":"2023-11-16T16:47:52.910430+0000","last_clean":"2023-11-16T16:47:52.910430+0000","last_became_active":"2023-11-16T16:47:25.813526+0000","last_became_peered":"2023-11-16T16:47:25.813526+0000","last_unstale":"2023-11-16T16:47:52.910430+0000","last_undegraded":"2023-11-16T16:47:52.910430+0000","last_fullsized":"2023-11-16T16:47:52.910430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:26:19.570859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":359,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857425+0000","last_change":"2023-11-16T16:46:18.016268+0000","last_active":"2023-11-16T16:47:44.857425+0000","last_peered":"2023-11-16T16:47:44.857425+0000","last_clean":"2023-11-16T16:47:44.857425+0000","last_became_active":"2023-11-16T16:46:18.015772+0000","last_became_peered":"2023-11-16T16:46:18.015772+0000","last_unstale":"2023-11-16T16:47:44.857425+0000","last_undegraded":"2023-11-16T16:47:44.857425+0000","last_fullsized":"2023-11-16T16:47:44.857425+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:43:04.489460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:43:04.489460+0000","last_clean_scrub_stamp":"2023-11-16T16:43:04.489460+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:22:57.779720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473045+0000","last_change":"2023-11-16T16:47:30.830574+0000","last_active":"2023-11-16T16:47:45.473045+0000","last_peered":"2023-11-16T16:47:45.473045+0000","last_clean":"2023-11-16T16:47:45.473045+0000","last_became_active":"2023-11-16T16:47:30.830040+0000","last_became_peered":"2023-11-16T16:47:30.830040+0000","last_unstale":"2023-11-16T16:47:45.473045+0000","last_undegraded":"2023-11-16T16:47:45.473045+0000","last_fullsized":"2023-11-16T16:47:45.473045+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:29:55.893689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856017+0000","last_change":"2023-11-16T16:47:26.378278+0000","last_active":"2023-11-16T16:47:44.856017+0000","last_peered":"2023-11-16T16:47:44.856017+0000","last_clean":"2023-11-16T16:47:44.856017+0000","last_became_active":"2023-11-16T16:47:26.377139+0000","last_became_peered":"2023-11-16T16:47:26.377139+0000","last_unstale":"2023-11-16T16:47:44.856017+0000","last_undegraded":"2023-11-16T16:47:44.856017+0000","last_fullsized":"2023-11-16T16:47:44.856017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:43:35.952129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.906018+0000","last_change":"2023-11-16T16:47:25.815714+0000","last_active":"2023-11-16T16:47:51.906018+0000","last_peered":"2023-11-16T16:47:51.906018+0000","last_clean":"2023-11-16T16:47:51.906018+0000","last_became_active":"2023-11-16T16:47:25.815452+0000","last_became_peered":"2023-11-16T16:47:25.815452+0000","last_unstale":"2023-11-16T16:47:51.906018+0000","last_undegraded":"2023-11-16T16:47:51.906018+0000","last_fullsized":"2023-11-16T16:47:51.906018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:33:18.235656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472075+0000","last_change":"2023-11-16T16:47:30.826690+0000","last_active":"2023-11-16T16:47:45.472075+0000","last_peered":"2023-11-16T16:47:45.472075+0000","last_clean":"2023-11-16T16:47:45.472075+0000","last_became_active":"2023-11-16T16:47:30.826216+0000","last_became_peered":"2023-11-16T16:47:30.826216+0000","last_unstale":"2023-11-16T16:47:45.472075+0000","last_undegraded":"2023-11-16T16:47:45.472075+0000","last_fullsized":"2023-11-16T16:47:45.472075+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:59:47.136636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857062+0000","last_change":"2023-11-16T16:47:25.826960+0000","last_active":"2023-11-16T16:47:44.857062+0000","last_peered":"2023-11-16T16:47:44.857062+0000","last_clean":"2023-11-16T16:47:44.857062+0000","last_became_active":"2023-11-16T16:47:25.826668+0000","last_became_peered":"2023-11-16T16:47:25.826668+0000","last_unstale":"2023-11-16T16:47:44.857062+0000","last_undegraded":"2023-11-16T16:47:44.857062+0000","last_fullsized":"2023-11-16T16:47:44.857062+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:45:36.995013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856905+0000","last_change":"2023-11-16T16:47:26.370684+0000","last_active":"2023-11-16T16:47:44.856905+0000","last_peered":"2023-11-16T16:47:44.856905+0000","last_clean":"2023-11-16T16:47:44.856905+0000","last_became_active":"2023-11-16T16:47:26.370323+0000","last_became_peered":"2023-11-16T16:47:26.370323+0000","last_unstale":"2023-11-16T16:47:44.856905+0000","last_undegraded":"2023-11-16T16:47:44.856905+0000","last_fullsized":"2023-11-16T16:47:44.856905+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:33:01.058990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.412979+0000","last_change":"2023-11-16T16:47:30.837921+0000","last_active":"2023-11-16T16:47:45.412979+0000","last_peered":"2023-11-16T16:47:45.412979+0000","last_clean":"2023-11-16T16:47:45.412979+0000","last_became_active":"2023-11-16T16:47:30.831491+0000","last_became_peered":"2023-11-16T16:47:30.831491+0000","last_unstale":"2023-11-16T16:47:45.412979+0000","last_undegraded":"2023-11-16T16:47:45.412979+0000","last_fullsized":"2023-11-16T16:47:45.412979+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:02:07.321263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.907422+0000","last_change":"2023-11-16T16:47:25.816416+0000","last_active":"2023-11-16T16:47:51.907422+0000","last_peered":"2023-11-16T16:47:51.907422+0000","last_clean":"2023-11-16T16:47:51.907422+0000","last_became_active":"2023-11-16T16:47:25.815875+0000","last_became_peered":"2023-11-16T16:47:25.815875+0000","last_unstale":"2023-11-16T16:47:51.907422+0000","last_undegraded":"2023-11-16T16:47:51.907422+0000","last_fullsized":"2023-11-16T16:47:51.907422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:34:15.271617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.413301+0000","last_change":"2023-11-16T16:47:26.386831+0000","last_active":"2023-11-16T16:47:45.413301+0000","last_peered":"2023-11-16T16:47:45.413301+0000","last_clean":"2023-11-16T16:47:45.413301+0000","last_became_active":"2023-11-16T16:47:26.386518+0000","last_became_peered":"2023-11-16T16:47:26.386518+0000","last_unstale":"2023-11-16T16:47:45.413301+0000","last_undegraded":"2023-11-16T16:47:45.413301+0000","last_fullsized":"2023-11-16T16:47:45.413301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:28:40.491752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854593+0000","last_change":"2023-11-16T16:47:30.823733+0000","last_active":"2023-11-16T16:47:44.854593+0000","last_peered":"2023-11-16T16:47:44.854593+0000","last_clean":"2023-11-16T16:47:44.854593+0000","last_became_active":"2023-11-16T16:47:30.823293+0000","last_became_peered":"2023-11-16T16:47:30.823293+0000","last_unstale":"2023-11-16T16:47:44.854593+0000","last_undegraded":"2023-11-16T16:47:44.854593+0000","last_fullsized":"2023-11-16T16:47:44.854593+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:30:09.995445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.899469+0000","last_change":"2023-11-16T16:47:25.818137+0000","last_active":"2023-11-16T16:47:50.899469+0000","last_peered":"2023-11-16T16:47:50.899469+0000","last_clean":"2023-11-16T16:47:50.899469+0000","last_became_active":"2023-11-16T16:47:25.817661+0000","last_became_peered":"2023-11-16T16:47:25.817661+0000","last_unstale":"2023-11-16T16:47:50.899469+0000","last_undegraded":"2023-11-16T16:47:50.899469+0000","last_fullsized":"2023-11-16T16:47:50.899469+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:17:33.467718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.415106+0000","last_change":"2023-11-16T16:47:45.415106+0000","last_active":"2023-11-16T16:47:45.415106+0000","last_peered":"2023-11-16T16:47:45.415106+0000","last_clean":"2023-11-16T16:47:45.415106+0000","last_became_active":"2023-11-16T16:47:45.414554+0000","last_became_peered":"2023-11-16T16:47:45.414554+0000","last_unstale":"2023-11-16T16:47:45.415106+0000","last_undegraded":"2023-11-16T16:47:45.415106+0000","last_fullsized":"2023-11-16T16:47:45.415106+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:20:08.005726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,8],"acting":[0,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854667+0000","last_change":"2023-11-16T16:47:30.820944+0000","last_active":"2023-11-16T16:47:44.854667+0000","last_peered":"2023-11-16T16:47:44.854667+0000","last_clean":"2023-11-16T16:47:44.854667+0000","last_became_active":"2023-11-16T16:47:30.820514+0000","last_became_peered":"2023-11-16T16:47:30.820514+0000","last_unstale":"2023-11-16T16:47:44.854667+0000","last_undegraded":"2023-11-16T16:47:44.854667+0000","last_fullsized":"2023-11-16T16:47:44.854667+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:18:55.421306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.899895+0000","last_change":"2023-11-16T16:47:25.826960+0000","last_active":"2023-11-16T16:47:50.899895+0000","last_peered":"2023-11-16T16:47:50.899895+0000","last_clean":"2023-11-16T16:47:50.899895+0000","last_became_active":"2023-11-16T16:47:25.826671+0000","last_became_peered":"2023-11-16T16:47:25.826671+0000","last_unstale":"2023-11-16T16:47:50.899895+0000","last_undegraded":"2023-11-16T16:47:50.899895+0000","last_fullsized":"2023-11-16T16:47:50.899895+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:41:34.201037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856299+0000","last_change":"2023-11-16T16:47:26.364697+0000","last_active":"2023-11-16T16:47:44.856299+0000","last_peered":"2023-11-16T16:47:44.856299+0000","last_clean":"2023-11-16T16:47:44.856299+0000","last_became_active":"2023-11-16T16:47:26.364418+0000","last_became_peered":"2023-11-16T16:47:26.364418+0000","last_unstale":"2023-11-16T16:47:44.856299+0000","last_undegraded":"2023-11-16T16:47:44.856299+0000","last_fullsized":"2023-11-16T16:47:44.856299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:31:32.598699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.475963+0000","last_change":"2023-11-16T16:47:45.474571+0000","last_active":"2023-11-16T16:47:45.475963+0000","last_peered":"2023-11-16T16:47:45.475963+0000","last_clean":"2023-11-16T16:47:45.475963+0000","last_became_active":"2023-11-16T16:47:45.474093+0000","last_became_peered":"2023-11-16T16:47:45.474093+0000","last_unstale":"2023-11-16T16:47:45.475963+0000","last_undegraded":"2023-11-16T16:47:45.475963+0000","last_fullsized":"2023-11-16T16:47:45.475963+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:24:09.557557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.899977+0000","last_change":"2023-11-16T16:47:25.826208+0000","last_active":"2023-11-16T16:47:50.899977+0000","last_peered":"2023-11-16T16:47:50.899977+0000","last_clean":"2023-11-16T16:47:50.899977+0000","last_became_active":"2023-11-16T16:47:25.825853+0000","last_became_peered":"2023-11-16T16:47:25.825853+0000","last_unstale":"2023-11-16T16:47:50.899977+0000","last_undegraded":"2023-11-16T16:47:50.899977+0000","last_fullsized":"2023-11-16T16:47:50.899977+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:02:40.593175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.471677+0000","last_change":"2023-11-16T16:47:26.361786+0000","last_active":"2023-11-16T16:47:45.471677+0000","last_peered":"2023-11-16T16:47:45.471677+0000","last_clean":"2023-11-16T16:47:45.471677+0000","last_became_active":"2023-11-16T16:47:26.361410+0000","last_became_peered":"2023-11-16T16:47:26.361410+0000","last_unstale":"2023-11-16T16:47:45.471677+0000","last_undegraded":"2023-11-16T16:47:45.471677+0000","last_fullsized":"2023-11-16T16:47:45.471677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:14:02.213856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.474814+0000","last_change":"2023-11-16T16:47:45.474123+0000","last_active":"2023-11-16T16:47:45.474814+0000","last_peered":"2023-11-16T16:47:45.474814+0000","last_clean":"2023-11-16T16:47:45.474814+0000","last_became_active":"2023-11-16T16:47:45.473630+0000","last_became_peered":"2023-11-16T16:47:45.473630+0000","last_unstale":"2023-11-16T16:47:45.474814+0000","last_undegraded":"2023-11-16T16:47:45.474814+0000","last_fullsized":"2023-11-16T16:47:45.474814+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:39:24.719048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855051+0000","last_change":"2023-11-16T16:47:25.816793+0000","last_active":"2023-11-16T16:47:44.855051+0000","last_peered":"2023-11-16T16:47:44.855051+0000","last_clean":"2023-11-16T16:47:44.855051+0000","last_became_active":"2023-11-16T16:47:25.815136+0000","last_became_peered":"2023-11-16T16:47:25.815136+0000","last_unstale":"2023-11-16T16:47:44.855051+0000","last_undegraded":"2023-11-16T16:47:44.855051+0000","last_fullsized":"2023-11-16T16:47:44.855051+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:12:25.041301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.412827+0000","last_change":"2023-11-16T16:47:26.379945+0000","last_active":"2023-11-16T16:47:45.412827+0000","last_peered":"2023-11-16T16:47:45.412827+0000","last_clean":"2023-11-16T16:47:45.412827+0000","last_became_active":"2023-11-16T16:47:26.376552+0000","last_became_peered":"2023-11-16T16:47:26.376552+0000","last_unstale":"2023-11-16T16:47:45.412827+0000","last_undegraded":"2023-11-16T16:47:45.412827+0000","last_fullsized":"2023-11-16T16:47:45.412827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:44:18.198024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854429+0000","last_change":"2023-11-16T16:47:30.826281+0000","last_active":"2023-11-16T16:47:44.854429+0000","last_peered":"2023-11-16T16:47:44.854429+0000","last_clean":"2023-11-16T16:47:44.854429+0000","last_became_active":"2023-11-16T16:47:30.825920+0000","last_became_peered":"2023-11-16T16:47:30.825920+0000","last_unstale":"2023-11-16T16:47:44.854429+0000","last_undegraded":"2023-11-16T16:47:44.854429+0000","last_fullsized":"2023-11-16T16:47:44.854429+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:23:25.700645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.899633+0000","last_change":"2023-11-16T16:47:25.815879+0000","last_active":"2023-11-16T16:47:50.899633+0000","last_peered":"2023-11-16T16:47:50.899633+0000","last_clean":"2023-11-16T16:47:50.899633+0000","last_became_active":"2023-11-16T16:47:25.815557+0000","last_became_peered":"2023-11-16T16:47:25.815557+0000","last_unstale":"2023-11-16T16:47:50.899633+0000","last_undegraded":"2023-11-16T16:47:50.899633+0000","last_fullsized":"2023-11-16T16:47:50.899633+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:04:39.245684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858518+0000","last_change":"2023-11-16T16:47:26.365013+0000","last_active":"2023-11-16T16:47:44.858518+0000","last_peered":"2023-11-16T16:47:44.858518+0000","last_clean":"2023-11-16T16:47:44.858518+0000","last_became_active":"2023-11-16T16:47:26.364710+0000","last_became_peered":"2023-11-16T16:47:26.364710+0000","last_unstale":"2023-11-16T16:47:44.858518+0000","last_undegraded":"2023-11-16T16:47:44.858518+0000","last_fullsized":"2023-11-16T16:47:44.858518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:57:21.045225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857432+0000","last_change":"2023-11-16T16:47:30.826880+0000","last_active":"2023-11-16T16:47:44.857432+0000","last_peered":"2023-11-16T16:47:44.857432+0000","last_clean":"2023-11-16T16:47:44.857432+0000","last_became_active":"2023-11-16T16:47:30.826547+0000","last_became_peered":"2023-11-16T16:47:30.826547+0000","last_unstale":"2023-11-16T16:47:44.857432+0000","last_undegraded":"2023-11-16T16:47:44.857432+0000","last_fullsized":"2023-11-16T16:47:44.857432+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:34:34.448480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.908865+0000","last_change":"2023-11-16T16:47:25.825246+0000","last_active":"2023-11-16T16:47:51.908865+0000","last_peered":"2023-11-16T16:47:51.908865+0000","last_clean":"2023-11-16T16:47:51.908865+0000","last_became_active":"2023-11-16T16:47:25.824929+0000","last_became_peered":"2023-11-16T16:47:25.824929+0000","last_unstale":"2023-11-16T16:47:51.908865+0000","last_undegraded":"2023-11-16T16:47:51.908865+0000","last_fullsized":"2023-11-16T16:47:51.908865+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:15:26.168130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.859304+0000","last_change":"2023-11-16T16:47:26.372266+0000","last_active":"2023-11-16T16:47:44.859304+0000","last_peered":"2023-11-16T16:47:44.859304+0000","last_clean":"2023-11-16T16:47:44.859304+0000","last_became_active":"2023-11-16T16:47:26.371857+0000","last_became_peered":"2023-11-16T16:47:26.371857+0000","last_unstale":"2023-11-16T16:47:44.859304+0000","last_undegraded":"2023-11-16T16:47:44.859304+0000","last_fullsized":"2023-11-16T16:47:44.859304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:22:33.979224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.859613+0000","last_change":"2023-11-16T16:47:30.832187+0000","last_active":"2023-11-16T16:47:44.859613+0000","last_peered":"2023-11-16T16:47:44.859613+0000","last_clean":"2023-11-16T16:47:44.859613+0000","last_became_active":"2023-11-16T16:47:30.828115+0000","last_became_peered":"2023-11-16T16:47:30.828115+0000","last_unstale":"2023-11-16T16:47:44.859613+0000","last_undegraded":"2023-11-16T16:47:44.859613+0000","last_fullsized":"2023-11-16T16:47:44.859613+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:05:02.381726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473519+0000","last_change":"2023-11-16T16:47:25.822706+0000","last_active":"2023-11-16T16:47:45.473519+0000","last_peered":"2023-11-16T16:47:45.473519+0000","last_clean":"2023-11-16T16:47:45.473519+0000","last_became_active":"2023-11-16T16:47:25.822400+0000","last_became_peered":"2023-11-16T16:47:25.822400+0000","last_unstale":"2023-11-16T16:47:45.473519+0000","last_undegraded":"2023-11-16T16:47:45.473519+0000","last_fullsized":"2023-11-16T16:47:45.473519+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:32:15.408720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.474751+0000","last_change":"2023-11-16T16:47:45.474751+0000","last_active":"2023-11-16T16:47:45.474751+0000","last_peered":"2023-11-16T16:47:45.474751+0000","last_clean":"2023-11-16T16:47:45.474751+0000","last_became_active":"2023-11-16T16:47:45.474415+0000","last_became_peered":"2023-11-16T16:47:45.474415+0000","last_unstale":"2023-11-16T16:47:45.474751+0000","last_undegraded":"2023-11-16T16:47:45.474751+0000","last_fullsized":"2023-11-16T16:47:45.474751+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:56:14.927411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472751+0000","last_change":"2023-11-16T16:47:30.822549+0000","last_active":"2023-11-16T16:47:45.472751+0000","last_peered":"2023-11-16T16:47:45.472751+0000","last_clean":"2023-11-16T16:47:45.472751+0000","last_became_active":"2023-11-16T16:47:30.822150+0000","last_became_peered":"2023-11-16T16:47:30.822150+0000","last_unstale":"2023-11-16T16:47:45.472751+0000","last_undegraded":"2023-11-16T16:47:45.472751+0000","last_fullsized":"2023-11-16T16:47:45.472751+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:17:45.983962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857636+0000","last_change":"2023-11-16T16:47:25.826947+0000","last_active":"2023-11-16T16:47:44.857636+0000","last_peered":"2023-11-16T16:47:44.857636+0000","last_clean":"2023-11-16T16:47:44.857636+0000","last_became_active":"2023-11-16T16:47:25.826587+0000","last_became_peered":"2023-11-16T16:47:25.826587+0000","last_unstale":"2023-11-16T16:47:44.857636+0000","last_undegraded":"2023-11-16T16:47:44.857636+0000","last_fullsized":"2023-11-16T16:47:44.857636+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:26:31.810188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858841+0000","last_change":"2023-11-16T16:47:26.373914+0000","last_active":"2023-11-16T16:47:44.858841+0000","last_peered":"2023-11-16T16:47:44.858841+0000","last_clean":"2023-11-16T16:47:44.858841+0000","last_became_active":"2023-11-16T16:47:26.373594+0000","last_became_peered":"2023-11-16T16:47:26.373594+0000","last_unstale":"2023-11-16T16:47:44.858841+0000","last_undegraded":"2023-11-16T16:47:44.858841+0000","last_fullsized":"2023-11-16T16:47:44.858841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:18:55.608486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472165+0000","last_change":"2023-11-16T16:47:26.368237+0000","last_active":"2023-11-16T16:47:45.472165+0000","last_peered":"2023-11-16T16:47:45.472165+0000","last_clean":"2023-11-16T16:47:45.472165+0000","last_became_active":"2023-11-16T16:47:26.367957+0000","last_became_peered":"2023-11-16T16:47:26.367957+0000","last_unstale":"2023-11-16T16:47:45.472165+0000","last_undegraded":"2023-11-16T16:47:45.472165+0000","last_fullsized":"2023-11-16T16:47:45.472165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:33:19.976017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854982+0000","last_change":"2023-11-16T16:47:26.375399+0000","last_active":"2023-11-16T16:47:44.854982+0000","last_peered":"2023-11-16T16:47:44.854982+0000","last_clean":"2023-11-16T16:47:44.854982+0000","last_became_active":"2023-11-16T16:47:26.375064+0000","last_became_peered":"2023-11-16T16:47:26.375064+0000","last_unstale":"2023-11-16T16:47:44.854982+0000","last_undegraded":"2023-11-16T16:47:44.854982+0000","last_fullsized":"2023-11-16T16:47:44.854982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:10:23.687848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854908+0000","last_change":"2023-11-16T16:47:26.377080+0000","last_active":"2023-11-16T16:47:44.854908+0000","last_peered":"2023-11-16T16:47:44.854908+0000","last_clean":"2023-11-16T16:47:44.854908+0000","last_became_active":"2023-11-16T16:47:26.376759+0000","last_became_peered":"2023-11-16T16:47:26.376759+0000","last_unstale":"2023-11-16T16:47:44.854908+0000","last_undegraded":"2023-11-16T16:47:44.854908+0000","last_fullsized":"2023-11-16T16:47:44.854908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:51:18.518523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.862174+0000","last_change":"2023-11-16T16:47:26.380619+0000","last_active":"2023-11-16T16:47:44.862174+0000","last_peered":"2023-11-16T16:47:44.862174+0000","last_clean":"2023-11-16T16:47:44.862174+0000","last_became_active":"2023-11-16T16:47:26.378668+0000","last_became_peered":"2023-11-16T16:47:26.378668+0000","last_unstale":"2023-11-16T16:47:44.862174+0000","last_undegraded":"2023-11-16T16:47:44.862174+0000","last_fullsized":"2023-11-16T16:47:44.862174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:59:30.902022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856785+0000","last_change":"2023-11-16T16:47:26.377482+0000","last_active":"2023-11-16T16:47:44.856785+0000","last_peered":"2023-11-16T16:47:44.856785+0000","last_clean":"2023-11-16T16:47:44.856785+0000","last_became_active":"2023-11-16T16:47:26.377077+0000","last_became_peered":"2023-11-16T16:47:26.377077+0000","last_unstale":"2023-11-16T16:47:44.856785+0000","last_undegraded":"2023-11-16T16:47:44.856785+0000","last_fullsized":"2023-11-16T16:47:44.856785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:19:35.960276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472542+0000","last_change":"2023-11-16T16:47:26.368193+0000","last_active":"2023-11-16T16:47:45.472542+0000","last_peered":"2023-11-16T16:47:45.472542+0000","last_clean":"2023-11-16T16:47:45.472542+0000","last_became_active":"2023-11-16T16:47:26.367499+0000","last_became_peered":"2023-11-16T16:47:26.367499+0000","last_unstale":"2023-11-16T16:47:45.472542+0000","last_undegraded":"2023-11-16T16:47:45.472542+0000","last_fullsized":"2023-11-16T16:47:45.472542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:09:17.429110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855675+0000","last_change":"2023-11-16T16:47:26.367340+0000","last_active":"2023-11-16T16:47:44.855675+0000","last_peered":"2023-11-16T16:47:44.855675+0000","last_clean":"2023-11-16T16:47:44.855675+0000","last_became_active":"2023-11-16T16:47:26.367046+0000","last_became_peered":"2023-11-16T16:47:26.367046+0000","last_unstale":"2023-11-16T16:47:44.855675+0000","last_undegraded":"2023-11-16T16:47:44.855675+0000","last_fullsized":"2023-11-16T16:47:44.855675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:14:30.539229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.413517+0000","last_change":"2023-11-16T16:47:26.386174+0000","last_active":"2023-11-16T16:47:45.413517+0000","last_peered":"2023-11-16T16:47:45.413517+0000","last_clean":"2023-11-16T16:47:45.413517+0000","last_became_active":"2023-11-16T16:47:26.385496+0000","last_became_peered":"2023-11-16T16:47:26.385496+0000","last_unstale":"2023-11-16T16:47:45.413517+0000","last_undegraded":"2023-11-16T16:47:45.413517+0000","last_fullsized":"2023-11-16T16:47:45.413517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:52:02.965352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473021+0000","last_change":"2023-11-16T16:47:26.368224+0000","last_active":"2023-11-16T16:47:45.473021+0000","last_peered":"2023-11-16T16:47:45.473021+0000","last_clean":"2023-11-16T16:47:45.473021+0000","last_became_active":"2023-11-16T16:47:26.367909+0000","last_became_peered":"2023-11-16T16:47:26.367909+0000","last_unstale":"2023-11-16T16:47:45.473021+0000","last_undegraded":"2023-11-16T16:47:45.473021+0000","last_fullsized":"2023-11-16T16:47:45.473021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:52:55.966566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.412846+0000","last_change":"2023-11-16T16:47:26.381224+0000","last_active":"2023-11-16T16:47:45.412846+0000","last_peered":"2023-11-16T16:47:45.412846+0000","last_clean":"2023-11-16T16:47:45.412846+0000","last_became_active":"2023-11-16T16:47:26.376586+0000","last_became_peered":"2023-11-16T16:47:26.376586+0000","last_unstale":"2023-11-16T16:47:45.412846+0000","last_undegraded":"2023-11-16T16:47:45.412846+0000","last_fullsized":"2023-11-16T16:47:45.412846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:36:59.850981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858440+0000","last_change":"2023-11-16T16:47:26.385895+0000","last_active":"2023-11-16T16:47:44.858440+0000","last_peered":"2023-11-16T16:47:44.858440+0000","last_clean":"2023-11-16T16:47:44.858440+0000","last_became_active":"2023-11-16T16:47:26.385580+0000","last_became_peered":"2023-11-16T16:47:26.385580+0000","last_unstale":"2023-11-16T16:47:44.858440+0000","last_undegraded":"2023-11-16T16:47:44.858440+0000","last_fullsized":"2023-11-16T16:47:44.858440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:03:39.204040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856629+0000","last_change":"2023-11-16T16:47:26.373785+0000","last_active":"2023-11-16T16:47:44.856629+0000","last_peered":"2023-11-16T16:47:44.856629+0000","last_clean":"2023-11-16T16:47:44.856629+0000","last_became_active":"2023-11-16T16:47:26.373425+0000","last_became_peered":"2023-11-16T16:47:26.373425+0000","last_unstale":"2023-11-16T16:47:44.856629+0000","last_undegraded":"2023-11-16T16:47:44.856629+0000","last_fullsized":"2023-11-16T16:47:44.856629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:16:02.226398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472287+0000","last_change":"2023-11-16T16:47:26.374722+0000","last_active":"2023-11-16T16:47:45.472287+0000","last_peered":"2023-11-16T16:47:45.472287+0000","last_clean":"2023-11-16T16:47:45.472287+0000","last_became_active":"2023-11-16T16:47:26.374326+0000","last_became_peered":"2023-11-16T16:47:26.374326+0000","last_unstale":"2023-11-16T16:47:45.472287+0000","last_undegraded":"2023-11-16T16:47:45.472287+0000","last_fullsized":"2023-11-16T16:47:45.472287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:52:18.596419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":249745,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":25267,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":28,"num_read_kb":41,"num_write":121,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":74727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106,"ondisk_log_size":106,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383697,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6156,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738996,"statfs":{"total":95995035648,"available":95988731904,"internally_reserved":0,"allocated":987136,"data_stored":495639,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.56999999999999995}]},{"osd":1,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.68700000000000006}]},{"osd":2,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.75700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":4,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.76800000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.345,"5min":1.345,"15min":1.345},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.69899999999999995}]},{"osd":6,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.91500000000000004}]},{"osd":7,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.878},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.84699999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":9,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.90400000000000003}]},{"osd":10,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.86199999999999999}]}]},{"osd":10,"up_from":72,"seq":309237645334,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6520,"kb_used_data":1072,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738632,"statfs":{"total":95995035648,"available":95988359168,"internally_reserved":0,"allocated":1097728,"data_stored":1108771,"data_compressed":53789,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":1,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.39600000000000002}]},{"osd":2,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.68100000000000005}]},{"osd":3,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.65500000000000003}]},{"osd":4,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.45500000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.71699999999999997}]},{"osd":6,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.69599999999999995}]},{"osd":7,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.46999999999999997}]},{"osd":8,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.78100000000000003}]},{"osd":9,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.67000000000000004}]},{"osd":11,"last update":"Thu Nov 16 16:47:38 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.79400000000000004}]}]},{"osd":9,"up_from":67,"seq":287762808859,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6368,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738784,"statfs":{"total":95995035648,"available":95988514816,"internally_reserved":0,"allocated":1007616,"data_stored":496457,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.45500000000000002}]},{"osd":1,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.72699999999999998}]},{"osd":2,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.401,"5min":1.401,"15min":1.401},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.71599999999999997}]},{"osd":3,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.59699999999999998}]},{"osd":4,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.77400000000000002}]},{"osd":6,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.629}]},{"osd":7,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.624},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.85099999999999998}]},{"osd":8,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.87}]},{"osd":10,"last update":"Thu Nov 16 16:47:17 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.78800000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]}]},{"osd":8,"up_from":59,"seq":253403070495,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6484,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738668,"statfs":{"total":95995035648,"available":95988396032,"internally_reserved":0,"allocated":995328,"data_stored":495934,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.373,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70199999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.76800000000000002,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.47099999999999997}]},{"osd":1,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58999999999999997,"15min":0.59099999999999997},"min":{"1min":0.40400000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.83199999999999996,"5min":1.089,"15min":1.089},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.71699999999999997}]},{"osd":2,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.47499999999999998,"15min":0.47899999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73499999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.47199999999999998,"15min":0.47299999999999998},"min":{"1min":0.32500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.497}]},{"osd":3,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.626,"15min":0.63700000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73899999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.66900000000000004,"15min":0.68200000000000005},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.81000000000000005,"5min":1.115,"15min":1.115},"last":0.66900000000000004}]},{"osd":4,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66000000000000003,"15min":0.66800000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76000000000000001,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.53700000000000003}]},{"osd":5,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.59599999999999997,"15min":0.60599999999999998},"min":{"1min":0.33000000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.83199999999999996,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.378,"5min":0.37,"15min":0.37},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56899999999999995}]},{"osd":6,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.73699999999999999,"15min":0.751},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.80400000000000005,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.625},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.76600000000000001,"15min":0.78200000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.92300000000000004,"5min":1.093,"15min":1.093},"last":0.76000000000000001}]},{"osd":7,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.72199999999999998,"15min":0.72799999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.88600000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69299999999999995,"15min":0.69399999999999995},"min":{"1min":0.46200000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.90200000000000002,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.73599999999999999}]},{"osd":9,"last update":"Thu Nov 16 16:46:54 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":0.75}]},{"osd":10,"last update":"Thu Nov 16 16:47:17 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.77700000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]}]},{"osd":7,"up_from":51,"seq":219043332131,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6604,"kb_used_data":1028,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738548,"statfs":{"total":95995035648,"available":95988273152,"internally_reserved":0,"allocated":1052672,"data_stored":1087703,"data_compressed":53789,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56200000000000006,"15min":0.55500000000000005},"min":{"1min":0.34899999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.56799999999999995,"15min":0.55600000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.48699999999999999}]},{"osd":1,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.27400000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.61399999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.46500000000000002,"15min":0.45800000000000002},"min":{"1min":0.316,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.42999999999999999}]},{"osd":2,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57799999999999996,"15min":0.57399999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.496}]},{"osd":3,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.62,"15min":0.61899999999999999},"min":{"1min":0.435,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.59799999999999998,"15min":0.58799999999999997},"min":{"1min":0.48299999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.54100000000000004}]},{"osd":4,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.505,"15min":0.496},"min":{"1min":0.36199999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.51300000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69399999999999995,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.45700000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.80400000000000005,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.62,"15min":0.61299999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.61699999999999999}]},{"osd":6,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.73599999999999999,"15min":0.70399999999999996},"min":{"1min":0.59799999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":4.8840000000000003,"5min":4.8840000000000003,"15min":4.8840000000000003},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.68700000000000006,"15min":0.68000000000000005},"min":{"1min":0.55300000000000005,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.63800000000000001}]},{"osd":8,"last update":"Thu Nov 16 16:47:26 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.85599999999999998}]},{"osd":9,"last update":"Thu Nov 16 16:46:54 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.68300000000000005}]},{"osd":10,"last update":"Thu Nov 16 16:47:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.71799999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]}]},{"osd":6,"up_from":46,"seq":197568495656,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6476,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738676,"statfs":{"total":95995035648,"available":95988404224,"internally_reserved":0,"allocated":987136,"data_stored":495639,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.49099999999999999,"15min":0.46700000000000003},"min":{"1min":0.312,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.71799999999999997,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55400000000000005,"15min":0.53400000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.81899999999999995,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.58599999999999997}]},{"osd":1,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.69499999999999995,"15min":0.66000000000000003},"min":{"1min":0.50600000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.032,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.433,"5min":0.378,"15min":0.378},"max":{"1min":0.90000000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.69399999999999995}]},{"osd":2,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.72899999999999998,"15min":0.70999999999999996},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0589999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.74199999999999999,"15min":0.71999999999999997},"min":{"1min":0.56100000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.008,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.91000000000000003}]},{"osd":3,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.68799999999999994,"15min":0.66800000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60599999999999998,"15min":0.60199999999999998},"min":{"1min":0.36399999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.90600000000000003,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.61299999999999999}]},{"osd":4,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.73799999999999999},"min":{"1min":0.47799999999999998,"5min":0.44,"15min":0.44},"max":{"1min":1,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.748},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.79600000000000004,"15min":0.78100000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.1819999999999999,"5min":1.3,"15min":1.3},"last":0.88200000000000001}]},{"osd":5,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.73199999999999998,"15min":0.73099999999999998},"min":{"1min":0.50800000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.147,"5min":1.206,"15min":1.206},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80600000000000005,"15min":0.78800000000000003},"min":{"1min":0.58299999999999996,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.139,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.89400000000000002}]},{"osd":7,"last update":"Thu Nov 16 16:47:11 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.71999999999999997,"15min":0.70599999999999996},"min":{"1min":0.53500000000000003,"5min":0.499,"15min":0.499},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.81999999999999995,"15min":0.80400000000000005},"min":{"1min":0.59499999999999997,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.21,"5min":1.246,"15min":1.246},"last":0.84999999999999998}]},{"osd":8,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91500000000000004,"15min":0.91600000000000004},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.234,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.61599999999999999,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.268,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.96599999999999997}]},{"osd":9,"last update":"Thu Nov 16 16:46:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.95299999999999996}]},{"osd":10,"last update":"Thu Nov 16 16:47:16 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.018,"5min":1.018,"15min":1.018},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":1.006}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7384,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737768,"statfs":{"total":95995035648,"available":95987474432,"internally_reserved":0,"allocated":999424,"data_stored":499433,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:50 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.53000000000000003,"15min":0.51600000000000001},"min":{"1min":0.34699999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.71099999999999997,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.318},{"interface":"front","average":{"1min":0.495,"5min":0.52300000000000002,"15min":0.498},"min":{"1min":0.35199999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.68400000000000005,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.35199999999999998}]},{"osd":2,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.52600000000000002,"15min":0.51100000000000001},"min":{"1min":0.38100000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.42199999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.5229999999999999,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":0.49399999999999999}]},{"osd":3,"last update":"Thu Nov 16 16:47:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.58399999999999996,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72799999999999998,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.61299999999999999,"15min":0.60899999999999999},"min":{"1min":0.44,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.79700000000000004,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.58499999999999996}]},{"osd":4,"last update":"Thu Nov 16 16:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.41999999999999998,"15min":0.40699999999999997},"min":{"1min":0.32700000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.498,"5min":0.45600000000000002,"15min":0.433},"min":{"1min":0.30599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.66100000000000003}]},{"osd":5,"last update":"Thu Nov 16 16:47:20 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.58099999999999996,"15min":0.56000000000000005},"min":{"1min":0.46899999999999997,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.60899999999999999,"15min":0.58099999999999996},"min":{"1min":0.52500000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.61599999999999999}]},{"osd":6,"last update":"Thu Nov 16 16:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66800000000000004,"15min":0.67000000000000004},"min":{"1min":0.45700000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.94499999999999995,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.68899999999999995},"min":{"1min":0.47099999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.91300000000000003,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.69399999999999995}]},{"osd":7,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81100000000000005,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.38800000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":0.56899999999999995}]},{"osd":8,"last update":"Thu Nov 16 16:47:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.64700000000000002,"15min":0.63600000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.69299999999999995,"15min":0.68999999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.94099999999999995,"5min":1,"15min":1},"last":0.79000000000000004}]},{"osd":9,"last update":"Thu Nov 16 16:46:54 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.651,"5min":1.651,"15min":1.651},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.68200000000000005}]},{"osd":10,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.67000000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7376,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737776,"statfs":{"total":95995035648,"available":95987482624,"internally_reserved":0,"allocated":991232,"data_stored":495729,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 16:46:56 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.57399999999999995,"15min":0.52600000000000002},"min":{"1min":0.46600000000000003,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.57799999999999996,"15min":0.53300000000000003},"min":{"1min":0.443,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.91200000000000003}]},{"osd":2,"last update":"Thu Nov 16 16:47:11 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.60399999999999998,"15min":0.58599999999999997},"min":{"1min":0.42899999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.628,"15min":0.58599999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.3479999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":0.51400000000000001}]},{"osd":3,"last update":"Thu Nov 16 16:46:31 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.53200000000000003,"15min":0.499},"min":{"1min":0.35499999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.58799999999999997,"15min":0.56599999999999995},"min":{"1min":0.39200000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.75700000000000001}]},{"osd":4,"last update":"Thu Nov 16 16:46:58 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.65200000000000002,"15min":0.61699999999999999},"min":{"1min":0.45800000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.1709999999999998,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.621},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.65500000000000003,"15min":0.63300000000000001},"min":{"1min":0.40600000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.87}]},{"osd":5,"last update":"Thu Nov 16 16:47:19 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.66100000000000003,"15min":0.64800000000000002},"min":{"1min":0.43099999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.71499999999999997,"15min":0.69499999999999995},"min":{"1min":0.53100000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":0.66400000000000003}]},{"osd":6,"last update":"Thu Nov 16 16:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.69199999999999995,"15min":0.67400000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73399999999999999,"15min":0.70899999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.64200000000000002}]},{"osd":7,"last update":"Thu Nov 16 16:47:08 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.77600000000000002,"15min":0.749},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.2810000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.78300000000000003,"15min":0.75600000000000001},"min":{"1min":0.57999999999999996,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":2.1859999999999999,"5min":2.1859999999999999,"15min":2.1859999999999999},"last":0.92800000000000005}]},{"osd":8,"last update":"Thu Nov 16 16:47:29 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.80700000000000005,"15min":0.81100000000000005},"min":{"1min":0.51700000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.145,"5min":2.145,"15min":2.145},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.871,"5min":0.82599999999999996,"15min":0.81799999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.125,"5min":2.125,"15min":2.125},"last":0.80300000000000005}]},{"osd":9,"last update":"Thu Nov 16 16:46:56 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.83499999999999996}]},{"osd":10,"last update":"Thu Nov 16 16:47:15 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.3159999999999998,"5min":2.3159999999999998,"15min":2.3159999999999998},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.96499999999999997}]},{"osd":11,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":2.2189999999999999,"5min":2.2189999999999999,"15min":2.2189999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":0.97999999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7484,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737668,"statfs":{"total":95995035648,"available":95987372032,"internally_reserved":0,"allocated":1036288,"data_stored":517564,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.48499999999999999,"15min":0.46400000000000002},"min":{"1min":0.314,"5min":0.249,"15min":0.249},"max":{"1min":0.80100000000000005,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.49399999999999999,"15min":0.48099999999999998},"min":{"1min":0.35299999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77400000000000002,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.45600000000000002}]},{"osd":1,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51000000000000001,"15min":0.5},"min":{"1min":0.36499999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67800000000000005,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.52100000000000002,"15min":0.503},"min":{"1min":0.41499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70399999999999996,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.39500000000000002}]},{"osd":3,"last update":"Thu Nov 16 16:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58199999999999996,"15min":0.56000000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.84599999999999997,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.59399999999999997,"15min":0.57399999999999995},"min":{"1min":0.34300000000000003,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.85499999999999998,"5min":1.016,"15min":1.016},"last":0.625}]},{"osd":4,"last update":"Thu Nov 16 16:47:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.53200000000000003,"15min":0.51800000000000002},"min":{"1min":0.442,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73899999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55000000000000004,"15min":0.54200000000000004},"min":{"1min":0.38400000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.70899999999999996,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.59799999999999998}]},{"osd":5,"last update":"Thu Nov 16 16:47:17 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.41499999999999998,"15min":0.40600000000000003},"min":{"1min":0.28599999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.372},{"interface":"front","average":{"1min":0.501,"5min":0.47899999999999998,"15min":0.46600000000000003},"min":{"1min":0.33400000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.64400000000000002}]},{"osd":6,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.63800000000000001,"15min":0.61599999999999999},"min":{"1min":0.56599999999999995,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.91100000000000003,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.64400000000000002,"15min":0.63300000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.88500000000000001,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.56000000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:47:05 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77200000000000002,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.503},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.66700000000000004,"15min":0.66900000000000004},"min":{"1min":0.51200000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.74099999999999999}]},{"osd":8,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.89800000000000002,"5min":0.997,"15min":0.997},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.72899999999999998}]},{"osd":9,"last update":"Thu Nov 16 16:46:52 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.67400000000000004}]},{"osd":10,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.83199999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6740,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738412,"statfs":{"total":95995035648,"available":95988133888,"internally_reserved":0,"allocated":995328,"data_stored":495743,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.433,"15min":0.40300000000000002},"min":{"1min":0.33900000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.62},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.436,"15min":0.41399999999999998},"min":{"1min":0.38300000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.65700000000000003}]},{"osd":1,"last update":"Thu Nov 16 16:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.63400000000000001,"15min":0.60599999999999998},"min":{"1min":0.41899999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.64600000000000002,"15min":0.60799999999999998},"min":{"1min":0.46500000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.92400000000000004,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":1.028}]},{"osd":2,"last update":"Thu Nov 16 16:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.625},"min":{"1min":0.42699999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89800000000000002,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.63800000000000001,"15min":0.61499999999999999},"min":{"1min":0.45400000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.81699999999999995,"5min":0.871,"15min":0.871},"last":0.78300000000000003}]},{"osd":4,"last update":"Thu Nov 16 16:47:04 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.70299999999999996,"15min":0.70999999999999996},"min":{"1min":0.58599999999999997,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.74199999999999999,"15min":0.73999999999999999},"min":{"1min":0.64400000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.98399999999999999}]},{"osd":5,"last update":"Thu Nov 16 16:47:22 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.70799999999999996,"15min":0.70699999999999996},"min":{"1min":0.57999999999999996,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.88600000000000001,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":1.1060000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.70899999999999996,"15min":0.71699999999999997},"min":{"1min":0.53100000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.879}]},{"osd":6,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61199999999999999,"15min":0.60999999999999999},"min":{"1min":0.42699999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.495,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":1.155}]},{"osd":7,"last update":"Thu Nov 16 16:47:11 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.747,"15min":0.73099999999999998},"min":{"1min":0.60799999999999998,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1,"5min":1,"15min":1},"last":1.1930000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.71899999999999997,"15min":0.71599999999999997},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.90900000000000003}]},{"osd":8,"last update":"Thu Nov 16 16:46:26 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.81299999999999994}]},{"osd":9,"last update":"Thu Nov 16 16:46:53 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":1.2210000000000001}]},{"osd":10,"last update":"Thu Nov 16 16:47:17 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":1.2450000000000001},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":1.008}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]}]},{"osd":4,"up_from":33,"seq":141733920818,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6616,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738536,"statfs":{"total":95995035648,"available":95988260864,"internally_reserved":0,"allocated":999424,"data_stored":496625,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.504,"15min":0.49099999999999999},"min":{"1min":0.41299999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.35299999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.41699999999999998}]},{"osd":1,"last update":"Thu Nov 16 16:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.37,"15min":0.35399999999999998},"min":{"1min":0.245,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.39600000000000002,"15min":0.38400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.71299999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.38800000000000001}]},{"osd":2,"last update":"Thu Nov 16 16:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55000000000000004,"15min":0.54400000000000004},"min":{"1min":0.38400000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.68799999999999994,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53200000000000003,"15min":0.52700000000000002},"min":{"1min":0.38400000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.70599999999999996}]},{"osd":3,"last update":"Thu Nov 16 16:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.56000000000000005,"15min":0.54200000000000004},"min":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.62},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.56799999999999995,"15min":0.55400000000000005},"min":{"1min":0.372,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.44800000000000001}]},{"osd":5,"last update":"Thu Nov 16 16:47:21 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.56299999999999994,"15min":0.54300000000000004},"min":{"1min":0.438,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78900000000000003,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.56799999999999995,"15min":0.54700000000000004},"min":{"1min":0.434,"5min":0.375,"15min":0.375},"max":{"1min":0.78000000000000003,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.55800000000000005}]},{"osd":6,"last update":"Thu Nov 16 16:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61699999999999999,"15min":0.60699999999999998},"min":{"1min":0.47899999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.84399999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66200000000000003,"15min":0.65100000000000002},"min":{"1min":0.49399999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.88500000000000001,"5min":0.995,"15min":0.995},"last":0.53500000000000003}]},{"osd":7,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.52000000000000002,"15min":0.51000000000000001},"min":{"1min":0.30499999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.76000000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.60599999999999998,"15min":0.59699999999999998},"min":{"1min":0.46400000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.50800000000000001}]},{"osd":8,"last update":"Thu Nov 16 16:47:28 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.51600000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.86899999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68000000000000005,"15min":0.67800000000000005},"min":{"1min":0.50600000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.68500000000000005}]},{"osd":9,"last update":"Thu Nov 16 16:46:55 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.747},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.64100000000000001}]},{"osd":10,"last update":"Thu Nov 16 16:47:16 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.69199999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6744,"kb_used_data":1040,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738408,"statfs":{"total":95995035648,"available":95988129792,"internally_reserved":0,"allocated":1064960,"data_stored":1091590,"data_compressed":53789,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:21 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.52400000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.84399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52100000000000002,"15min":0.51700000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Nov 16 16:47:21 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.50600000000000001,"15min":0.48399999999999999},"min":{"1min":0.377,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80900000000000005,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.51300000000000001,"15min":0.49399999999999999},"min":{"1min":0.38200000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.73599999999999999,"5min":0.749,"15min":0.749},"last":0.79000000000000004}]},{"osd":2,"last update":"Thu Nov 16 16:47:21 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40300000000000002,"15min":0.39300000000000002},"min":{"1min":0.30499999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.52500000000000002,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.5},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.41499999999999998,"15min":0.40100000000000002},"min":{"1min":0.29299999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.81799999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.53600000000000003}]},{"osd":3,"last update":"Thu Nov 16 16:47:21 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.55900000000000005,"15min":0.55100000000000005},"min":{"1min":0.40400000000000003,"5min":0.371,"15min":0.371},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.629},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.56200000000000006,"15min":0.55300000000000005},"min":{"1min":0.44400000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.58299999999999996}]},{"osd":4,"last update":"Thu Nov 16 16:47:21 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57399999999999995,"15min":0.55100000000000005},"min":{"1min":0.432,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58799999999999997,"15min":0.56399999999999995},"min":{"1min":0.45700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.81599999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.82599999999999996}]},{"osd":6,"last update":"Thu Nov 16 16:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.626,"15min":0.60999999999999999},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.82699999999999996,"5min":1.052,"15min":1.052},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.63100000000000001,"15min":0.61699999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.86899999999999999,"5min":1.117,"15min":1.117},"last":0.65300000000000002}]},{"osd":7,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.58799999999999997,"15min":0.58499999999999996},"min":{"1min":0.40500000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.747}]},{"osd":8,"last update":"Thu Nov 16 16:47:29 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.61499999999999999,"15min":0.61399999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.84399999999999997,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.84999999999999998}]},{"osd":9,"last update":"Thu Nov 16 16:47:52 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.73999999999999999,"15min":0.73799999999999999},"min":{"1min":0.53100000000000003,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.94499999999999995,"5min":1.145,"15min":1.145},"last":0.878},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.74399999999999999,"15min":0.745},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.89900000000000002,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.80800000000000005}]},{"osd":10,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.86299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34505,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34505,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":21175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":22867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":23095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T16:47:56.890 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-16T16:47:56.891 INFO:tasks.cephadm:Setting up client nodes... 2023-11-16T16:47:56.893 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-11-16T16:47:57.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:56 smithi002 ceph-mon[128712]: pgmap v297: 129 pgs: 129 active+clean; 601 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2023-11-16T16:47:57.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:56 smithi141 ceph-mon[131184]: pgmap v297: 129 pgs: 129 active+clean; 601 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2023-11-16T16:47:57.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:56 smithi155 ceph-mon[131607]: pgmap v297: 129 pgs: 129 active+clean; 601 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2023-11-16T16:47:57.776 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:47:58.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:57 smithi002 ceph-mon[128712]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:47:58.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:57 smithi002 ceph-mon[128712]: from='client.24878 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:47:58.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:57 smithi141 ceph-mon[131184]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:47:58.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:57 smithi141 ceph-mon[131184]: from='client.24878 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:47:58.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:57 smithi155 ceph-mon[131607]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:47:58.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:57 smithi155 ceph-mon[131607]: from='client.24878 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:47:59.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:47:58 smithi002 ceph-mon[128712]: pgmap v298: 129 pgs: 129 active+clean; 602 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 10 op/s 2023-11-16T16:47:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:47:58 smithi141 ceph-mon[131184]: pgmap v298: 129 pgs: 129 active+clean; 602 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 10 op/s 2023-11-16T16:47:59.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:47:58 smithi155 ceph-mon[131607]: pgmap v298: 129 pgs: 129 active+clean; 602 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 10 op/s 2023-11-16T16:47:59.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.450+0000 7f85c3c5f700 1 -- 172.21.15.2:0/1778525503 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85bc1014f0 msgr2=0x7f85bc101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:59.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.450+0000 7f85c3c5f700 1 --2- 172.21.15.2:0/1778525503 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85bc1014f0 0x7f85bc101940 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f85ac009990 tx=0x7f85ac009ca0 comp rx=0 tx=0).stop 2023-11-16T16:47:59.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.451+0000 7f85c3c5f700 1 -- 172.21.15.2:0/1778525503 shutdown_connections 2023-11-16T16:47:59.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.451+0000 7f85c3c5f700 1 --2- 172.21.15.2:0/1778525503 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85bc1002f0 0x7f85bc100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:59.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.451+0000 7f85c3c5f700 1 --2- 172.21.15.2:0/1778525503 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85bc101e80 0x7f85bc108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:59.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.451+0000 7f85c3c5f700 1 --2- 172.21.15.2:0/1778525503 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85bc1014f0 0x7f85bc101940 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:59.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.451+0000 7f85c3c5f700 1 -- 172.21.15.2:0/1778525503 >> 172.21.15.2:0/1778525503 conn(0x7f85bc0fb900 msgr2=0x7f85bc0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:59.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.451+0000 7f85c3c5f700 1 -- 172.21.15.2:0/1778525503 shutdown_connections 2023-11-16T16:47:59.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.451+0000 7f85c3c5f700 1 -- 172.21.15.2:0/1778525503 wait complete. 2023-11-16T16:47:59.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.452+0000 7f85c3c5f700 1 Processor -- start 2023-11-16T16:47:59.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.452+0000 7f85c3c5f700 1 -- start start 2023-11-16T16:47:59.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.452+0000 7f85c3c5f700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85bc1002f0 0x7f85bc196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:59.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.452+0000 7f85c3c5f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85bc1014f0 0x7f85bc1965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:59.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.452+0000 7f85c3c5f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85bc101e80 0x7f85bc19d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:59.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.452+0000 7f85c3c5f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc19dbf0 con 0x7f85bc101e80 2023-11-16T16:47:59.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.452+0000 7f85c3c5f700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc19dd30 con 0x7f85bc1002f0 2023-11-16T16:47:59.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.452+0000 7f85c3c5f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc19dec0 con 0x7f85bc1014f0 2023-11-16T16:47:59.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.453+0000 7f85c21fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85bc101e80 0x7f85bc19d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:59.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.453+0000 7f85c19fb700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85bc1002f0 0x7f85bc196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:59.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.453+0000 7f85c21fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85bc101e80 0x7f85bc19d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58892/0 (socket says 172.21.15.2:58892) 2023-11-16T16:47:59.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.453+0000 7f85c19fb700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85bc1002f0 0x7f85bc196060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:38782/0 (socket says 172.21.15.2:38782) 2023-11-16T16:47:59.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.453+0000 7f85c21fc700 1 -- 172.21.15.2:0/1951826673 learned_addr learned my addr 172.21.15.2:0/1951826673 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:47:59.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.453+0000 7f85c11fa700 1 --2- 172.21.15.2:0/1951826673 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85bc1014f0 0x7f85bc1965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:59.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.453+0000 7f85c21fc700 1 -- 172.21.15.2:0/1951826673 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85bc1014f0 msgr2=0x7f85bc1965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:59.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.453+0000 7f85c21fc700 1 --2- 172.21.15.2:0/1951826673 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85bc1014f0 0x7f85bc1965a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:59.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.453+0000 7f85c21fc700 1 -- 172.21.15.2:0/1951826673 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85bc1002f0 msgr2=0x7f85bc196060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:59.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.453+0000 7f85c21fc700 1 --2- 172.21.15.2:0/1951826673 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85bc1002f0 0x7f85bc196060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:59.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.453+0000 7f85c21fc700 1 -- 172.21.15.2:0/1951826673 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac0095f0 con 0x7f85bc101e80 2023-11-16T16:47:59.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.453+0000 7f85c21fc700 1 --2- 172.21.15.2:0/1951826673 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85bc101e80 0x7f85bc19d620 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f85b4007f10 tx=0x7f85b400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:59.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.454+0000 7f85b2ffd700 1 -- 172.21.15.2:0/1951826673 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b400dd20 con 0x7f85bc101e80 2023-11-16T16:47:59.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.454+0000 7f85b2ffd700 1 -- 172.21.15.2:0/1951826673 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f85b400f040 con 0x7f85bc101e80 2023-11-16T16:47:59.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.454+0000 7f85b2ffd700 1 -- 172.21.15.2:0/1951826673 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b40048a0 con 0x7f85bc101e80 2023-11-16T16:47:59.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.454+0000 7f85c3c5f700 1 -- 172.21.15.2:0/1951826673 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85bc10bc90 con 0x7f85bc101e80 2023-11-16T16:47:59.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.454+0000 7f85c3c5f700 1 -- 172.21.15.2:0/1951826673 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85bc10bb00 con 0x7f85bc101e80 2023-11-16T16:47:59.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.455+0000 7f85c3c5f700 1 -- 172.21.15.2:0/1951826673 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85bc1078f0 con 0x7f85bc101e80 2023-11-16T16:47:59.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.457+0000 7f85b2ffd700 1 -- 172.21.15.2:0/1951826673 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f85b4003a00 con 0x7f85bc101e80 2023-11-16T16:47:59.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.458+0000 7f85b2ffd700 1 --2- 172.21.15.2:0/1951826673 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f85a80a1260 0x7f85a80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:47:59.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.458+0000 7f85b2ffd700 1 -- 172.21.15.2:0/1951826673 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f85b40cc600 con 0x7f85bc101e80 2023-11-16T16:47:59.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.458+0000 7f85c19fb700 1 --2- 172.21.15.2:0/1951826673 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f85a80a1260 0x7f85a80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:47:59.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.459+0000 7f85c19fb700 1 --2- 172.21.15.2:0/1951826673 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f85a80a1260 0x7f85a80a3710 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f85bc101350 tx=0x7f85b8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:47:59.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.462+0000 7f85b2ffd700 1 -- 172.21.15.2:0/1951826673 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f85b404e730 con 0x7f85bc101e80 2023-11-16T16:47:59.753 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.751+0000 7f85c3c5f700 1 -- 172.21.15.2:0/1951826673 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f85bc02cfe0 con 0x7f85bc101e80 2023-11-16T16:47:59.757 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.755+0000 7f85b2ffd700 1 -- 172.21.15.2:0/1951826673 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f85b4095880 con 0x7f85bc101e80 2023-11-16T16:47:59.757 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2023-11-16T16:47:59.758 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQC/R1ZlKwzdLBAAHFjwgmleJsghlQdIecuI7w== 2023-11-16T16:47:59.760 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.758+0000 7f85c3c5f700 1 -- 172.21.15.2:0/1951826673 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f85a80a1260 msgr2=0x7f85a80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:59.760 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.758+0000 7f85c3c5f700 1 --2- 172.21.15.2:0/1951826673 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f85a80a1260 0x7f85a80a3710 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f85bc101350 tx=0x7f85b8009260 comp rx=0 tx=0).stop 2023-11-16T16:47:59.760 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.758+0000 7f85c3c5f700 1 -- 172.21.15.2:0/1951826673 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85bc101e80 msgr2=0x7f85bc19d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:47:59.760 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.758+0000 7f85c3c5f700 1 --2- 172.21.15.2:0/1951826673 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85bc101e80 0x7f85bc19d620 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f85b4007f10 tx=0x7f85b400d3b0 comp rx=0 tx=0).stop 2023-11-16T16:47:59.760 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.758+0000 7f85c3c5f700 1 -- 172.21.15.2:0/1951826673 shutdown_connections 2023-11-16T16:47:59.760 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.758+0000 7f85c3c5f700 1 --2- 172.21.15.2:0/1951826673 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f85a80a1260 0x7f85a80a3710 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:59.760 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.758+0000 7f85c3c5f700 1 --2- 172.21.15.2:0/1951826673 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85bc101e80 0x7f85bc19d620 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:59.760 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.758+0000 7f85c3c5f700 1 --2- 172.21.15.2:0/1951826673 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f85bc1002f0 0x7f85bc196060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:59.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.758+0000 7f85c3c5f700 1 --2- 172.21.15.2:0/1951826673 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85bc1014f0 0x7f85bc1965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:47:59.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.758+0000 7f85c3c5f700 1 -- 172.21.15.2:0/1951826673 >> 172.21.15.2:0/1951826673 conn(0x7f85bc0fb900 msgr2=0x7f85bc104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:47:59.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.758+0000 7f85c3c5f700 1 -- 172.21.15.2:0/1951826673 shutdown_connections 2023-11-16T16:47:59.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:47:59.758+0000 7f85c3c5f700 1 -- 172.21.15.2:0/1951826673 wait complete. 2023-11-16T16:48:00.281 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T16:48:00.281 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-11-16T16:48:00.281 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-16T16:48:00.386 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-16T16:48:00.387 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-11-16T16:48:00.387 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph mgr dump --format=json 2023-11-16T16:48:00.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:00 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1951826673' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-16T16:48:00.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:00 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1951826673' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-16T16:48:00.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:00 smithi002 ceph-mon[128712]: pgmap v299: 129 pgs: 129 active+clean; 624 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 6.1 KiB/s wr, 13 op/s 2023-11-16T16:48:00.764 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:00 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1951826673' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-16T16:48:00.764 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:00 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1951826673' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-16T16:48:00.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:00 smithi141 ceph-mon[131184]: pgmap v299: 129 pgs: 129 active+clean; 624 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 6.1 KiB/s wr, 13 op/s 2023-11-16T16:48:00.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:00 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1951826673' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-11-16T16:48:00.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:00 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1951826673' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-11-16T16:48:00.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:00 smithi155 ceph-mon[131607]: pgmap v299: 129 pgs: 129 active+clean; 624 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 6.1 KiB/s wr, 13 op/s 2023-11-16T16:48:01.287 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:01.902 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:01 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:48:02.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:01 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:48:02.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:01 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:48:02.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.945+0000 7fa048af9700 1 -- 172.21.15.2:0/339002279 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0441055e0 msgr2=0x7fa04410d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:02.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.945+0000 7fa048af9700 1 --2- 172.21.15.2:0/339002279 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0441055e0 0x7fa04410d160 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa03c00b0d0 tx=0x7fa03c00b3e0 comp rx=0 tx=0).stop 2023-11-16T16:48:02.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.945+0000 7fa048af9700 1 -- 172.21.15.2:0/339002279 shutdown_connections 2023-11-16T16:48:02.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.945+0000 7fa048af9700 1 --2- 172.21.15.2:0/339002279 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0441055e0 0x7fa04410d160 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:02.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.945+0000 7fa048af9700 1 --2- 172.21.15.2:0/339002279 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa044100540 0x7fa044100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:02.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.945+0000 7fa048af9700 1 --2- 172.21.15.2:0/339002279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa044100ee0 0x7fa044104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:02.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.945+0000 7fa048af9700 1 -- 172.21.15.2:0/339002279 >> 172.21.15.2:0/339002279 conn(0x7fa0440fbdc0 msgr2=0x7fa0440fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:02.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.945+0000 7fa048af9700 1 -- 172.21.15.2:0/339002279 shutdown_connections 2023-11-16T16:48:02.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.945+0000 7fa048af9700 1 -- 172.21.15.2:0/339002279 wait complete. 2023-11-16T16:48:02.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.946+0000 7fa048af9700 1 Processor -- start 2023-11-16T16:48:02.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.946+0000 7fa048af9700 1 -- start start 2023-11-16T16:48:02.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.947+0000 7fa048af9700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa044100540 0x7fa04419e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:02.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.947+0000 7fa048af9700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa044100ee0 0x7fa04419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:02.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.947+0000 7fa048af9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0441055e0 0x7fa044198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:02.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.947+0000 7fa048af9700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa044198d30 con 0x7fa0441055e0 2023-11-16T16:48:02.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.947+0000 7fa048af9700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa044198ea0 con 0x7fa044100ee0 2023-11-16T16:48:02.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.947+0000 7fa048af9700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa044199060 con 0x7fa044100540 2023-11-16T16:48:02.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.947+0000 7fa042d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0441055e0 0x7fa044198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:02.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.947+0000 7fa042d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0441055e0 0x7fa044198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58916/0 (socket says 172.21.15.2:58916) 2023-11-16T16:48:02.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.947+0000 7fa042d9d700 1 -- 172.21.15.2:0/3731113044 learned_addr learned my addr 172.21.15.2:0/3731113044 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:02.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.947+0000 7fa041d9b700 1 --2- 172.21.15.2:0/3731113044 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa044100ee0 0x7fa04419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:02.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.947+0000 7fa04259c700 1 --2- 172.21.15.2:0/3731113044 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa044100540 0x7fa04419e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:02.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.948+0000 7fa042d9d700 1 -- 172.21.15.2:0/3731113044 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa044100540 msgr2=0x7fa04419e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:02.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.948+0000 7fa042d9d700 1 --2- 172.21.15.2:0/3731113044 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa044100540 0x7fa04419e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:02.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.948+0000 7fa042d9d700 1 -- 172.21.15.2:0/3731113044 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa044100ee0 msgr2=0x7fa04419ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:02.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.948+0000 7fa042d9d700 1 --2- 172.21.15.2:0/3731113044 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa044100ee0 0x7fa04419ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:02.952 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.948+0000 7fa042d9d700 1 -- 172.21.15.2:0/3731113044 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa034009520 con 0x7fa0441055e0 2023-11-16T16:48:02.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.948+0000 7fa041d9b700 1 --2- 172.21.15.2:0/3731113044 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa044100ee0 0x7fa04419ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:02.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.948+0000 7fa042d9d700 1 --2- 172.21.15.2:0/3731113044 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0441055e0 0x7fa044198580 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa03c007820 tx=0x7fa03c007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:02.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.948+0000 7fa0337fe700 1 -- 172.21.15.2:0/3731113044 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c00e040 con 0x7fa0441055e0 2023-11-16T16:48:02.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.948+0000 7fa0337fe700 1 -- 172.21.15.2:0/3731113044 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa03c00bcb0 con 0x7fa0441055e0 2023-11-16T16:48:02.953 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.948+0000 7fa048af9700 1 -- 172.21.15.2:0/3731113044 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa03c009d00 con 0x7fa0441055e0 2023-11-16T16:48:02.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.948+0000 7fa0337fe700 1 -- 172.21.15.2:0/3731113044 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c0058a0 con 0x7fa0441055e0 2023-11-16T16:48:02.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.948+0000 7fa048af9700 1 -- 172.21.15.2:0/3731113044 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0441103b0 con 0x7fa0441055e0 2023-11-16T16:48:02.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.950+0000 7fa048af9700 1 -- 172.21.15.2:0/3731113044 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa044110220 con 0x7fa0441055e0 2023-11-16T16:48:02.954 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.951+0000 7fa0337fe700 1 -- 172.21.15.2:0/3731113044 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fa03c004a50 con 0x7fa0441055e0 2023-11-16T16:48:02.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.953+0000 7fa0337fe700 1 --2- 172.21.15.2:0/3731113044 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa02c0a1260 0x7fa02c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:02.956 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.953+0000 7fa0337fe700 1 -- 172.21.15.2:0/3731113044 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa03c0ce5b0 con 0x7fa0441055e0 2023-11-16T16:48:02.956 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.953+0000 7fa04259c700 1 --2- 172.21.15.2:0/3731113044 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa02c0a1260 0x7fa02c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:02.956 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.954+0000 7fa04259c700 1 --2- 172.21.15.2:0/3731113044 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa02c0a1260 0x7fa02c0a3710 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa034009df0 tx=0x7fa034009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:02.957 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:02.955+0000 7fa0337fe700 1 -- 172.21.15.2:0/3731113044 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa03c0979e0 con 0x7fa0441055e0 2023-11-16T16:48:03.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:02 smithi155 ceph-mon[131607]: pgmap v300: 129 pgs: 129 active+clean; 624 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2023-11-16T16:48:03.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:02 smithi002 ceph-mon[128712]: pgmap v300: 129 pgs: 129 active+clean; 624 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2023-11-16T16:48:03.223 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:03.221+0000 7fa048af9700 1 -- 172.21.15.2:0/3731113044 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fa044110540 con 0x7fa0441055e0 2023-11-16T16:48:03.228 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:03.227+0000 7fa0337fe700 1 -- 172.21.15.2:0/3731113044 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260117 (secure 0 0 0) 0x7fa03c09b000 con 0x7fa0441055e0 2023-11-16T16:48:03.233 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:48:03.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:03.231+0000 7fa048af9700 1 -- 172.21.15.2:0/3731113044 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa02c0a1260 msgr2=0x7fa02c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:03.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:03.231+0000 7fa048af9700 1 --2- 172.21.15.2:0/3731113044 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa02c0a1260 0x7fa02c0a3710 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa034009df0 tx=0x7fa034009260 comp rx=0 tx=0).stop 2023-11-16T16:48:03.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:03.231+0000 7fa048af9700 1 -- 172.21.15.2:0/3731113044 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0441055e0 msgr2=0x7fa044198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:03.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:03.231+0000 7fa048af9700 1 --2- 172.21.15.2:0/3731113044 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0441055e0 0x7fa044198580 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa03c007820 tx=0x7fa03c007850 comp rx=0 tx=0).stop 2023-11-16T16:48:03.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:03.231+0000 7fa048af9700 1 -- 172.21.15.2:0/3731113044 shutdown_connections 2023-11-16T16:48:03.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:03.231+0000 7fa048af9700 1 --2- 172.21.15.2:0/3731113044 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa02c0a1260 0x7fa02c0a3710 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:03.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:03.231+0000 7fa048af9700 1 --2- 172.21.15.2:0/3731113044 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa044100540 0x7fa04419e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:03.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:03.231+0000 7fa048af9700 1 --2- 172.21.15.2:0/3731113044 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0441055e0 0x7fa044198580 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:03.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:03.231+0000 7fa048af9700 1 --2- 172.21.15.2:0/3731113044 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa044100ee0 0x7fa04419ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:03.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:03.231+0000 7fa048af9700 1 -- 172.21.15.2:0/3731113044 >> 172.21.15.2:0/3731113044 conn(0x7fa0440fbdc0 msgr2=0x7fa0440fc720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:03.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:03.231+0000 7fa048af9700 1 -- 172.21.15.2:0/3731113044 shutdown_connections 2023-11-16T16:48:03.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:03.231+0000 7fa048af9700 1 -- 172.21.15.2:0/3731113044 wait complete. 2023-11-16T16:48:03.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:02 smithi141 ceph-mon[131184]: pgmap v300: 129 pgs: 129 active+clean; 624 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2023-11-16T16:48:03.688 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":17,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":1793067160},{"type":"v1","addr":"172.21.15.2:6801","nonce":1793067160}]},"active_addr":"172.21.15.2:6801/1793067160","active_change":"2023-11-16T16:40:43.705239+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.2:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":3784433150}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":2834229980}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":1824419243}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":3820370584}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":1857592670}]}]} 2023-11-16T16:48:03.693 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-11-16T16:48:03.693 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-11-16T16:48:03.693 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph osd dump --format=json 2023-11-16T16:48:03.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:03 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3731113044' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-11-16T16:48:04.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:03 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3731113044' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-11-16T16:48:04.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:03 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3731113044' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-11-16T16:48:04.556 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:05.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:04 smithi155 ceph-mon[131607]: pgmap v301: 129 pgs: 129 active+clean; 624 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 5.9 KiB/s wr, 13 op/s 2023-11-16T16:48:05.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:04 smithi002 ceph-mon[128712]: pgmap v301: 129 pgs: 129 active+clean; 624 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 5.9 KiB/s wr, 13 op/s 2023-11-16T16:48:05.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:04 smithi141 ceph-mon[131184]: pgmap v301: 129 pgs: 129 active+clean; 624 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 5.9 KiB/s wr, 13 op/s 2023-11-16T16:48:06.225 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.223+0000 7f34c63f0700 1 -- 172.21.15.2:0/2382894040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34c0100ee0 msgr2=0x7f34c0104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:06.225 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.223+0000 7f34c63f0700 1 --2- 172.21.15.2:0/2382894040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34c0100ee0 0x7f34c0104eb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f34b40099c0 tx=0x7f34b4009cd0 comp rx=0 tx=0).stop 2023-11-16T16:48:06.225 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.223+0000 7f34c63f0700 1 -- 172.21.15.2:0/2382894040 shutdown_connections 2023-11-16T16:48:06.225 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.223+0000 7f34c63f0700 1 --2- 172.21.15.2:0/2382894040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34c01055e0 0x7f34c010d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:06.226 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.223+0000 7f34c63f0700 1 --2- 172.21.15.2:0/2382894040 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34c0100540 0x7f34c0100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:06.226 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.223+0000 7f34c63f0700 1 --2- 172.21.15.2:0/2382894040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34c0100ee0 0x7f34c0104eb0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:06.226 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.223+0000 7f34c63f0700 1 -- 172.21.15.2:0/2382894040 >> 172.21.15.2:0/2382894040 conn(0x7f34c00fbdc0 msgr2=0x7f34c00fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:06.226 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.223+0000 7f34c63f0700 1 -- 172.21.15.2:0/2382894040 shutdown_connections 2023-11-16T16:48:06.226 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.223+0000 7f34c63f0700 1 -- 172.21.15.2:0/2382894040 wait complete. 2023-11-16T16:48:06.228 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.224+0000 7f34c63f0700 1 Processor -- start 2023-11-16T16:48:06.228 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.224+0000 7f34c63f0700 1 -- start start 2023-11-16T16:48:06.228 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.225+0000 7f34c63f0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34c0100540 0x7f34c01963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:06.228 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.225+0000 7f34c63f0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34c0100ee0 0x7f34c0196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:06.228 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.225+0000 7f34c63f0700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34c01055e0 0x7f34c019ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:06.228 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.225+0000 7f34c63f0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c019b320 con 0x7f34c0100ee0 2023-11-16T16:48:06.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.225+0000 7f34c63f0700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c019b490 con 0x7f34c01055e0 2023-11-16T16:48:06.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.225+0000 7f34c63f0700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c019b650 con 0x7f34c0100540 2023-11-16T16:48:06.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.225+0000 7f34bf7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34c0100ee0 0x7f34c0196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:06.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.225+0000 7f34bf7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34c0100ee0 0x7f34c0196910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44840/0 (socket says 172.21.15.2:44840) 2023-11-16T16:48:06.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.225+0000 7f34bf7fe700 1 -- 172.21.15.2:0/1254148558 learned_addr learned my addr 172.21.15.2:0/1254148558 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:06.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.225+0000 7f34bffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34c0100540 0x7f34c01963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:06.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.225+0000 7f34c498d700 1 --2- 172.21.15.2:0/1254148558 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34c01055e0 0x7f34c019ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:06.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.226+0000 7f34bf7fe700 1 -- 172.21.15.2:0/1254148558 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34c0100540 msgr2=0x7f34c01963d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:06.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.226+0000 7f34bf7fe700 1 --2- 172.21.15.2:0/1254148558 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34c0100540 0x7f34c01963d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:06.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.226+0000 7f34bf7fe700 1 -- 172.21.15.2:0/1254148558 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34c01055e0 msgr2=0x7f34c019ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:06.229 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.226+0000 7f34bf7fe700 1 --2- 172.21.15.2:0/1254148558 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34c01055e0 0x7f34c019ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:06.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.226+0000 7f34bf7fe700 1 -- 172.21.15.2:0/1254148558 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b40095f0 con 0x7f34c0100ee0 2023-11-16T16:48:06.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.226+0000 7f34c498d700 1 --2- 172.21.15.2:0/1254148558 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34c01055e0 0x7f34c019ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:48:06.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.226+0000 7f34bf7fe700 1 --2- 172.21.15.2:0/1254148558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34c0100ee0 0x7f34c0196910 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f34b40098e0 tx=0x7f34b400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:06.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.226+0000 7f34bd7fa700 1 -- 172.21.15.2:0/1254148558 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b401d070 con 0x7f34c0100ee0 2023-11-16T16:48:06.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.226+0000 7f34bd7fa700 1 -- 172.21.15.2:0/1254148558 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f34b4005510 con 0x7f34c0100ee0 2023-11-16T16:48:06.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.226+0000 7f34c63f0700 1 -- 172.21.15.2:0/1254148558 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c0110360 con 0x7f34c0100ee0 2023-11-16T16:48:06.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.226+0000 7f34bd7fa700 1 -- 172.21.15.2:0/1254148558 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b4004a00 con 0x7f34c0100ee0 2023-11-16T16:48:06.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.226+0000 7f34c63f0700 1 -- 172.21.15.2:0/1254148558 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c01101d0 con 0x7f34c0100ee0 2023-11-16T16:48:06.230 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.227+0000 7f34c63f0700 1 -- 172.21.15.2:0/1254148558 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c01104f0 con 0x7f34c0100ee0 2023-11-16T16:48:06.231 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.229+0000 7f34bd7fa700 1 -- 172.21.15.2:0/1254148558 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f34b4004b80 con 0x7f34c0100ee0 2023-11-16T16:48:06.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.231+0000 7f34bd7fa700 1 --2- 172.21.15.2:0/1254148558 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f34a80a11e0 0x7f34a80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:06.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.231+0000 7f34bd7fa700 1 -- 172.21.15.2:0/1254148558 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f34b40cdf70 con 0x7f34c0100ee0 2023-11-16T16:48:06.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.231+0000 7f34bffff700 1 --2- 172.21.15.2:0/1254148558 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f34a80a11e0 0x7f34a80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:06.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.232+0000 7f34bffff700 1 --2- 172.21.15.2:0/1254148558 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f34a80a11e0 0x7f34a80a3690 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009c60 tx=0x7f34b0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:06.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.233+0000 7f34bd7fa700 1 -- 172.21.15.2:0/1254148558 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f34b4097280 con 0x7f34c0100ee0 2023-11-16T16:48:06.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.446+0000 7f34c63f0700 1 -- 172.21.15.2:0/1254148558 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34c0066ea0 con 0x7f34c0100ee0 2023-11-16T16:48:06.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.448+0000 7f34bd7fa700 1 -- 172.21.15.2:0/1254148558 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24379 (secure 0 0 0) 0x7f34b409a8a0 con 0x7f34c0100ee0 2023-11-16T16:48:06.450 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:48:06.450 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":91,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:47:44.849645+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:24.458452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:25.786005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:29.313181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:48:06.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.450+0000 7f34c63f0700 1 -- 172.21.15.2:0/1254148558 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f34a80a11e0 msgr2=0x7f34a80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:06.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.450+0000 7f34c63f0700 1 --2- 172.21.15.2:0/1254148558 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f34a80a11e0 0x7f34a80a3690 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009c60 tx=0x7f34b0009260 comp rx=0 tx=0).stop 2023-11-16T16:48:06.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.450+0000 7f34c63f0700 1 -- 172.21.15.2:0/1254148558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34c0100ee0 msgr2=0x7f34c0196910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:06.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.450+0000 7f34c63f0700 1 --2- 172.21.15.2:0/1254148558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34c0100ee0 0x7f34c0196910 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f34b40098e0 tx=0x7f34b400f690 comp rx=0 tx=0).stop 2023-11-16T16:48:06.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.450+0000 7f34c63f0700 1 -- 172.21.15.2:0/1254148558 shutdown_connections 2023-11-16T16:48:06.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.450+0000 7f34c63f0700 1 --2- 172.21.15.2:0/1254148558 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f34a80a11e0 0x7f34a80a3690 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:06.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.450+0000 7f34c63f0700 1 --2- 172.21.15.2:0/1254148558 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34c0100540 0x7f34c01963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:06.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.450+0000 7f34c63f0700 1 --2- 172.21.15.2:0/1254148558 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f34c01055e0 0x7f34c019ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:06.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.450+0000 7f34c63f0700 1 --2- 172.21.15.2:0/1254148558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34c0100ee0 0x7f34c0196910 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:06.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.450+0000 7f34c63f0700 1 -- 172.21.15.2:0/1254148558 >> 172.21.15.2:0/1254148558 conn(0x7f34c00fbdc0 msgr2=0x7f34c00fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:06.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.451+0000 7f34c63f0700 1 -- 172.21.15.2:0/1254148558 shutdown_connections 2023-11-16T16:48:06.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:06.451+0000 7f34c63f0700 1 -- 172.21.15.2:0/1254148558 wait complete. 2023-11-16T16:48:06.962 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-11-16T16:48:06.962 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph osd dump --format=json 2023-11-16T16:48:07.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:06 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:48:07.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:06 smithi155 ceph-mon[131607]: pgmap v302: 129 pgs: 129 active+clean; 652 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.9 KiB/s wr, 15 op/s 2023-11-16T16:48:07.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:06 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1254148558' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:07.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:06 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:48:07.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:06 smithi002 ceph-mon[128712]: pgmap v302: 129 pgs: 129 active+clean; 652 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.9 KiB/s wr, 15 op/s 2023-11-16T16:48:07.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:06 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1254148558' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:07.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:06 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:48:07.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:06 smithi141 ceph-mon[131184]: pgmap v302: 129 pgs: 129 active+clean; 652 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.9 KiB/s wr, 15 op/s 2023-11-16T16:48:07.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:06 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1254148558' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:07.824 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:09.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:08 smithi155 ceph-mon[131607]: pgmap v303: 129 pgs: 129 active+clean; 652 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.0 KiB/s wr, 9 op/s 2023-11-16T16:48:09.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:08 smithi002 ceph-mon[128712]: pgmap v303: 129 pgs: 129 active+clean; 652 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.0 KiB/s wr, 9 op/s 2023-11-16T16:48:09.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:08 smithi141 ceph-mon[131184]: pgmap v303: 129 pgs: 129 active+clean; 652 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.0 KiB/s wr, 9 op/s 2023-11-16T16:48:09.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.439+0000 7ff145252700 1 -- 172.21.15.2:0/916118961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1401055e0 msgr2=0x7ff14010d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:09.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.439+0000 7ff145252700 1 --2- 172.21.15.2:0/916118961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1401055e0 0x7ff14010d160 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff13800b0d0 tx=0x7ff13800b3e0 comp rx=0 tx=0).stop 2023-11-16T16:48:09.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.439+0000 7ff145252700 1 -- 172.21.15.2:0/916118961 shutdown_connections 2023-11-16T16:48:09.441 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.439+0000 7ff145252700 1 --2- 172.21.15.2:0/916118961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1401055e0 0x7ff14010d160 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:09.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.439+0000 7ff145252700 1 --2- 172.21.15.2:0/916118961 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff140100540 0x7ff140100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:09.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.439+0000 7ff145252700 1 --2- 172.21.15.2:0/916118961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff140100ee0 0x7ff140104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:09.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.439+0000 7ff145252700 1 -- 172.21.15.2:0/916118961 >> 172.21.15.2:0/916118961 conn(0x7ff1400fbdc0 msgr2=0x7ff1400fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:09.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.439+0000 7ff145252700 1 -- 172.21.15.2:0/916118961 shutdown_connections 2023-11-16T16:48:09.442 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.439+0000 7ff145252700 1 -- 172.21.15.2:0/916118961 wait complete. 2023-11-16T16:48:09.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.440+0000 7ff145252700 1 Processor -- start 2023-11-16T16:48:09.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.440+0000 7ff145252700 1 -- start start 2023-11-16T16:48:09.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.441+0000 7ff145252700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff140100540 0x7ff1401986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:09.444 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.441+0000 7ff145252700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff140100ee0 0x7ff140198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.441+0000 7ff145252700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1401055e0 0x7ff14019bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.441+0000 7ff145252700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14019c5a0 con 0x7ff1401055e0 2023-11-16T16:48:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.441+0000 7ff145252700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14019c710 con 0x7ff140100540 2023-11-16T16:48:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.441+0000 7ff145252700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14019c8d0 con 0x7ff140100ee0 2023-11-16T16:48:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.441+0000 7ff13f59e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1401055e0 0x7ff14019bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.441+0000 7ff13f59e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1401055e0 0x7ff14019bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44868/0 (socket says 172.21.15.2:44868) 2023-11-16T16:48:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.441+0000 7ff13f59e700 1 -- 172.21.15.2:0/4165643172 learned_addr learned my addr 172.21.15.2:0/4165643172 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.441+0000 7ff13ed9d700 1 --2- 172.21.15.2:0/4165643172 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff140100540 0x7ff1401986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:09.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.441+0000 7ff13e59c700 1 --2- 172.21.15.2:0/4165643172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff140100ee0 0x7ff140198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:09.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.442+0000 7ff13f59e700 1 -- 172.21.15.2:0/4165643172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff140100ee0 msgr2=0x7ff140198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:09.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.442+0000 7ff13f59e700 1 --2- 172.21.15.2:0/4165643172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff140100ee0 0x7ff140198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:09.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.442+0000 7ff13f59e700 1 -- 172.21.15.2:0/4165643172 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff140100540 msgr2=0x7ff1401986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:09.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.442+0000 7ff13f59e700 1 --2- 172.21.15.2:0/4165643172 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff140100540 0x7ff1401986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:09.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.442+0000 7ff13f59e700 1 -- 172.21.15.2:0/4165643172 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff138009d00 con 0x7ff1401055e0 2023-11-16T16:48:09.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.442+0000 7ff13f59e700 1 --2- 172.21.15.2:0/4165643172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1401055e0 0x7ff14019bfa0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff138007870 tx=0x7ff1380078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:09.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.442+0000 7ff127fff700 1 -- 172.21.15.2:0/4165643172 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13800e040 con 0x7ff1401055e0 2023-11-16T16:48:09.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.442+0000 7ff145252700 1 -- 172.21.15.2:0/4165643172 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff140110320 con 0x7ff1401055e0 2023-11-16T16:48:09.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.442+0000 7ff127fff700 1 -- 172.21.15.2:0/4165643172 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff138003e10 con 0x7ff1401055e0 2023-11-16T16:48:09.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.442+0000 7ff127fff700 1 -- 172.21.15.2:0/4165643172 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13801ba10 con 0x7ff1401055e0 2023-11-16T16:48:09.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.442+0000 7ff145252700 1 -- 172.21.15.2:0/4165643172 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff140110190 con 0x7ff1401055e0 2023-11-16T16:48:09.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.443+0000 7ff145252700 1 -- 172.21.15.2:0/4165643172 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1401104b0 con 0x7ff1401055e0 2023-11-16T16:48:09.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.445+0000 7ff127fff700 1 -- 172.21.15.2:0/4165643172 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7ff138024410 con 0x7ff1401055e0 2023-11-16T16:48:09.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.447+0000 7ff127fff700 1 --2- 172.21.15.2:0/4165643172 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff1280a1260 0x7ff1280a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:09.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.447+0000 7ff127fff700 1 -- 172.21.15.2:0/4165643172 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff1380ce3b0 con 0x7ff1401055e0 2023-11-16T16:48:09.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.447+0000 7ff13ed9d700 1 --2- 172.21.15.2:0/4165643172 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff1280a1260 0x7ff1280a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:09.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.448+0000 7ff13ed9d700 1 --2- 172.21.15.2:0/4165643172 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff1280a1260 0x7ff1280a3710 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff130005950 tx=0x7ff13000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:09.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.451+0000 7ff127fff700 1 -- 172.21.15.2:0/4165643172 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff1380976c0 con 0x7ff1401055e0 2023-11-16T16:48:09.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.661+0000 7ff145252700 1 -- 172.21.15.2:0/4165643172 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff140066ea0 con 0x7ff1401055e0 2023-11-16T16:48:09.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.663+0000 7ff127fff700 1 -- 172.21.15.2:0/4165643172 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24379 (secure 0 0 0) 0x7ff13809ace0 con 0x7ff1401055e0 2023-11-16T16:48:09.665 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:48:09.666 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":91,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:47:44.849645+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:24.458452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:25.786005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:29.313181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:48:09.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.665+0000 7ff145252700 1 -- 172.21.15.2:0/4165643172 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff1280a1260 msgr2=0x7ff1280a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:09.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.665+0000 7ff145252700 1 --2- 172.21.15.2:0/4165643172 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff1280a1260 0x7ff1280a3710 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff130005950 tx=0x7ff13000b410 comp rx=0 tx=0).stop 2023-11-16T16:48:09.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.665+0000 7ff145252700 1 -- 172.21.15.2:0/4165643172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1401055e0 msgr2=0x7ff14019bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:09.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.665+0000 7ff145252700 1 --2- 172.21.15.2:0/4165643172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1401055e0 0x7ff14019bfa0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff138007870 tx=0x7ff1380078a0 comp rx=0 tx=0).stop 2023-11-16T16:48:09.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.665+0000 7ff145252700 1 -- 172.21.15.2:0/4165643172 shutdown_connections 2023-11-16T16:48:09.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.665+0000 7ff145252700 1 --2- 172.21.15.2:0/4165643172 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff1280a1260 0x7ff1280a3710 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:09.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.665+0000 7ff145252700 1 --2- 172.21.15.2:0/4165643172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1401055e0 0x7ff14019bfa0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:09.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.665+0000 7ff145252700 1 --2- 172.21.15.2:0/4165643172 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff140100540 0x7ff1401986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:09.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.665+0000 7ff145252700 1 --2- 172.21.15.2:0/4165643172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff140100ee0 0x7ff140198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:09.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.665+0000 7ff145252700 1 -- 172.21.15.2:0/4165643172 >> 172.21.15.2:0/4165643172 conn(0x7ff1400fbdc0 msgr2=0x7ff1400fc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:09.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.666+0000 7ff145252700 1 -- 172.21.15.2:0/4165643172 shutdown_connections 2023-11-16T16:48:09.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:09.666+0000 7ff145252700 1 -- 172.21.15.2:0/4165643172 wait complete. 2023-11-16T16:48:10.231 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.0 flush_pg_stats 2023-11-16T16:48:10.231 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.1 flush_pg_stats 2023-11-16T16:48:10.232 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.2 flush_pg_stats 2023-11-16T16:48:10.232 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.3 flush_pg_stats 2023-11-16T16:48:10.233 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.4 flush_pg_stats 2023-11-16T16:48:10.233 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.5 flush_pg_stats 2023-11-16T16:48:10.234 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.6 flush_pg_stats 2023-11-16T16:48:10.234 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.7 flush_pg_stats 2023-11-16T16:48:10.235 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.8 flush_pg_stats 2023-11-16T16:48:10.235 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.9 flush_pg_stats 2023-11-16T16:48:10.236 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.10 flush_pg_stats 2023-11-16T16:48:10.236 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph tell osd.11 flush_pg_stats 2023-11-16T16:48:10.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:10 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/4165643172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:10.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:10 smithi141 ceph-mon[131184]: pgmap v304: 129 pgs: 129 active+clean; 652 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2023-11-16T16:48:10.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:10 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/4165643172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:10.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:10 smithi155 ceph-mon[131607]: pgmap v304: 129 pgs: 129 active+clean; 652 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2023-11-16T16:48:10.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:10 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/4165643172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:10.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:10 smithi002 ceph-mon[128712]: pgmap v304: 129 pgs: 129 active+clean; 652 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2023-11-16T16:48:12.243 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:12.434 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:12.515 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:12.757 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:13.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:12 smithi155 ceph-mon[131607]: pgmap v305: 129 pgs: 129 active+clean; 652 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2023-11-16T16:48:13.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:12 smithi002 ceph-mon[128712]: pgmap v305: 129 pgs: 129 active+clean; 652 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2023-11-16T16:48:13.164 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:13.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:12 smithi141 ceph-mon[131184]: pgmap v305: 129 pgs: 129 active+clean; 652 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2023-11-16T16:48:14.636 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:14.715 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:14.772 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:15.027 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:15.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:14 smithi155 ceph-mon[131607]: pgmap v306: 129 pgs: 129 active+clean; 652 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2023-11-16T16:48:15.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:14 smithi002 ceph-mon[128712]: pgmap v306: 129 pgs: 129 active+clean; 652 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2023-11-16T16:48:15.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:14 smithi141 ceph-mon[131184]: pgmap v306: 129 pgs: 129 active+clean; 652 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2023-11-16T16:48:15.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.737+0000 7fd6ebc34700 1 -- 172.21.15.2:0/3551295969 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6e4102f50 msgr2=0x7fd6e410f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:15.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.737+0000 7fd6ebc34700 1 --2- 172.21.15.2:0/3551295969 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6e4102f50 0x7fd6e410f3d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc00b0a0 tx=0x7fd6dc00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:48:15.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.738+0000 7fd6ebc34700 1 -- 172.21.15.2:0/3551295969 shutdown_connections 2023-11-16T16:48:15.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.738+0000 7fd6ebc34700 1 --2- 172.21.15.2:0/3551295969 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6e41086b0 0x7fd6e4108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:15.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.738+0000 7fd6ebc34700 1 --2- 172.21.15.2:0/3551295969 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6e4102f50 0x7fd6e410f3d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:15.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.738+0000 7fd6ebc34700 1 --2- 172.21.15.2:0/3551295969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6e41025c0 0x7fd6e4102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:15.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.738+0000 7fd6ebc34700 1 -- 172.21.15.2:0/3551295969 >> 172.21.15.2:0/3551295969 conn(0x7fd6e40fe100 msgr2=0x7fd6e4100510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:15.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.738+0000 7fd6ebc34700 1 -- 172.21.15.2:0/3551295969 shutdown_connections 2023-11-16T16:48:15.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.738+0000 7fd6ebc34700 1 -- 172.21.15.2:0/3551295969 wait complete. 2023-11-16T16:48:15.741 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:15.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.739+0000 7fd6ebc34700 1 Processor -- start 2023-11-16T16:48:15.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.739+0000 7fd6ebc34700 1 -- start start 2023-11-16T16:48:15.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.739+0000 7fd6ebc34700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6e41025c0 0x7fd6e4198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:15.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.739+0000 7fd6ebc34700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6e4102f50 0x7fd6e4198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:15.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.739+0000 7fd6ebc34700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6e41086b0 0x7fd6e419cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:15.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.739+0000 7fd6ebc34700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e419d3f0 con 0x7fd6e41025c0 2023-11-16T16:48:15.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.739+0000 7fd6ebc34700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e419d560 con 0x7fd6e41086b0 2023-11-16T16:48:15.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.739+0000 7fd6ebc34700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e419d720 con 0x7fd6e4102f50 2023-11-16T16:48:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.740+0000 7fd6e91cf700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6e4102f50 0x7fd6e4198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.740+0000 7fd6ea1d1700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6e41086b0 0x7fd6e419cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.740+0000 7fd6e91cf700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6e4102f50 0x7fd6e4198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:42974/0 (socket says 172.21.15.2:42974) 2023-11-16T16:48:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.740+0000 7fd6e91cf700 1 -- 172.21.15.2:0/1190799419 learned_addr learned my addr 172.21.15.2:0/1190799419 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.740+0000 7fd6ea1d1700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6e41086b0 0x7fd6e419cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:46114/0 (socket says 172.21.15.2:46114) 2023-11-16T16:48:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.740+0000 7fd6e99d0700 1 --2- 172.21.15.2:0/1190799419 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6e41025c0 0x7fd6e4198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.740+0000 7fd6e91cf700 1 -- 172.21.15.2:0/1190799419 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6e41086b0 msgr2=0x7fd6e419cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.740+0000 7fd6e91cf700 1 --2- 172.21.15.2:0/1190799419 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd6e41086b0 0x7fd6e419cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.740+0000 7fd6e91cf700 1 -- 172.21.15.2:0/1190799419 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6e41025c0 msgr2=0x7fd6e4198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.740+0000 7fd6e91cf700 1 --2- 172.21.15.2:0/1190799419 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6e41025c0 0x7fd6e4198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:15.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.740+0000 7fd6e91cf700 1 -- 172.21.15.2:0/1190799419 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d4009520 con 0x7fd6e4102f50 2023-11-16T16:48:15.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.741+0000 7fd6e91cf700 1 --2- 172.21.15.2:0/1190799419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6e4102f50 0x7fd6e4198970 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400eb30 tx=0x7fd6d400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:15.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.741+0000 7fd6daffd700 1 -- 172.21.15.2:0/1190799419 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4009880 con 0x7fd6e4102f50 2023-11-16T16:48:15.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.741+0000 7fd6ebc34700 1 -- 172.21.15.2:0/1190799419 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6dc009d00 con 0x7fd6e4102f50 2023-11-16T16:48:15.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.741+0000 7fd6daffd700 1 -- 172.21.15.2:0/1190799419 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6d40044e0 con 0x7fd6e4102f50 2023-11-16T16:48:15.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.741+0000 7fd6daffd700 1 -- 172.21.15.2:0/1190799419 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4003e90 con 0x7fd6e4102f50 2023-11-16T16:48:15.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.741+0000 7fd6ebc34700 1 -- 172.21.15.2:0/1190799419 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e4112560 con 0x7fd6e4102f50 2023-11-16T16:48:15.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.742+0000 7fd6ebc34700 1 -- 172.21.15.2:0/1190799419 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd6e4100130 con 0x7fd6e4102f50 2023-11-16T16:48:15.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.744+0000 7fd6daffd700 1 -- 172.21.15.2:0/1190799419 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fd6d4004000 con 0x7fd6e4102f50 2023-11-16T16:48:15.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.747+0000 7fd6daffd700 1 --2- 172.21.15.2:0/1190799419 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd6d00a11e0 0x7fd6d00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:15.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.747+0000 7fd6daffd700 1 -- 172.21.15.2:0/1190799419 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd6d4014070 con 0x7fd6e4102f50 2023-11-16T16:48:15.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.747+0000 7fd6e99d0700 1 --2- 172.21.15.2:0/1190799419 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd6d00a11e0 0x7fd6d00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:15.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.748+0000 7fd6daffd700 1 --2- 172.21.15.2:0/1190799419 >> [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294] conn(0x7fd6d00ac080 0x7fd6d00ae510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:15.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.748+0000 7fd6daffd700 1 -- 172.21.15.2:0/1190799419 --> [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd6d00aebe0 con 0x7fd6d00ac080 2023-11-16T16:48:15.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.748+0000 7fd6daffd700 1 -- 172.21.15.2:0/1190799419 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6d40cd680 con 0x7fd6e4102f50 2023-11-16T16:48:15.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.748+0000 7fd6ea1d1700 1 --2- 172.21.15.2:0/1190799419 >> [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294] conn(0x7fd6d00ac080 0x7fd6d00ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:15.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.748+0000 7fd6e99d0700 1 --2- 172.21.15.2:0/1190799419 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd6d00a11e0 0x7fd6d00a3690 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0005ea0 tx=0x7fd6e0005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:15.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.749+0000 7fd6ea1d1700 1 --2- 172.21.15.2:0/1190799419 >> [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294] conn(0x7fd6d00ac080 0x7fd6d00ae510 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:15.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.750+0000 7fd6daffd700 1 -- 172.21.15.2:0/1190799419 <== osd.0 v2:172.21.15.2:6802/3417839294 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fd6d00aebe0 con 0x7fd6d00ac080 2023-11-16T16:48:15.772 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.771+0000 7fd6ebc34700 1 -- 172.21.15.2:0/1190799419 --> [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd6e41123d0 con 0x7fd6d00ac080 2023-11-16T16:48:15.773 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.772+0000 7fd6daffd700 1 -- 172.21.15.2:0/1190799419 <== osd.0 v2:172.21.15.2:6802/3417839294 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd6e41123d0 con 0x7fd6d00ac080 2023-11-16T16:48:15.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.772+0000 7fd6ebc34700 1 -- 172.21.15.2:0/1190799419 >> [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294] conn(0x7fd6d00ac080 msgr2=0x7fd6d00ae510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:15.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.772+0000 7fd6ebc34700 1 --2- 172.21.15.2:0/1190799419 >> [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294] conn(0x7fd6d00ac080 0x7fd6d00ae510 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:15.774 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.772+0000 7fd6ebc34700 1 -- 172.21.15.2:0/1190799419 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd6d00a11e0 msgr2=0x7fd6d00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:15.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.772+0000 7fd6ebc34700 1 --2- 172.21.15.2:0/1190799419 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd6d00a11e0 0x7fd6d00a3690 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0005ea0 tx=0x7fd6e0005e30 comp rx=0 tx=0).stop 2023-11-16T16:48:15.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.772+0000 7fd6ebc34700 1 -- 172.21.15.2:0/1190799419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6e4102f50 msgr2=0x7fd6e4198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:15.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.772+0000 7fd6ebc34700 1 --2- 172.21.15.2:0/1190799419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6e4102f50 0x7fd6e4198970 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400eb30 tx=0x7fd6d400ee40 comp rx=0 tx=0).stop 2023-11-16T16:48:15.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.772+0000 7fd6ea1d1700 1 -- 172.21.15.2:0/1190799419 reap_dead start 2023-11-16T16:48:15.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.772+0000 7fd6ebc34700 1 -- 172.21.15.2:0/1190799419 shutdown_connections 2023-11-16T16:48:15.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.772+0000 7fd6ebc34700 1 -- 172.21.15.2:0/1190799419 >> 172.21.15.2:0/1190799419 conn(0x7fd6e40fe100 msgr2=0x7fd6e40ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:15.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.773+0000 7fd6ebc34700 1 -- 172.21.15.2:0/1190799419 shutdown_connections 2023-11-16T16:48:15.775 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:15.773+0000 7fd6ebc34700 1 -- 172.21.15.2:0/1190799419 wait complete. 2023-11-16T16:48:15.950 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:16.027 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:16.745 INFO:teuthology.orchestra.run.smithi002.stdout:34359738440 2023-11-16T16:48:16.746 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.0 2023-11-16T16:48:16.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.666+0000 7f6f95405700 1 -- 172.21.15.2:0/51229045 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f90105410 msgr2=0x7f6f9010cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:16.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.666+0000 7f6f95405700 1 --2- 172.21.15.2:0/51229045 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f90105410 0x7f6f9010cf90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6f8800b0d0 tx=0x7f6f8800b3e0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.666+0000 7f6f95405700 1 -- 172.21.15.2:0/51229045 shutdown_connections 2023-11-16T16:48:16.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.667+0000 7f6f95405700 1 --2- 172.21.15.2:0/51229045 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f90105410 0x7f6f9010cf90 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.667+0000 7f6f95405700 1 --2- 172.21.15.2:0/51229045 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f90100370 0x7f6f90100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.667+0000 7f6f95405700 1 --2- 172.21.15.2:0/51229045 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f90100d10 0x7f6f90104ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.667+0000 7f6f95405700 1 -- 172.21.15.2:0/51229045 >> 172.21.15.2:0/51229045 conn(0x7f6f900fbbd0 msgr2=0x7f6f900fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:16.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.667+0000 7f6f95405700 1 -- 172.21.15.2:0/51229045 shutdown_connections 2023-11-16T16:48:16.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.667+0000 7f6f95405700 1 -- 172.21.15.2:0/51229045 wait complete. 2023-11-16T16:48:16.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.668+0000 7f6f95405700 1 Processor -- start 2023-11-16T16:48:16.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.668+0000 7f6f95405700 1 -- start start 2023-11-16T16:48:16.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.668+0000 7f6f95405700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f90100370 0x7f6f90198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:16.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.668+0000 7f6f95405700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f90100d10 0x7f6f90198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:16.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.668+0000 7f6f95405700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f90105410 0x7f6f9019cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:16.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.668+0000 7f6f95405700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9019d4f0 con 0x7f6f90105410 2023-11-16T16:48:16.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.668+0000 7f6f95405700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9019d660 con 0x7f6f90100370 2023-11-16T16:48:16.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.668+0000 7f6f95405700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9019d820 con 0x7f6f90100d10 2023-11-16T16:48:16.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.669+0000 7f6f94c04700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f90105410 0x7f6f9019cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:16.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.669+0000 7f6f8ffff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f90100370 0x7f6f90198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:16.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.669+0000 7f6f94c04700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f90105410 0x7f6f9019cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40594/0 (socket says 172.21.15.2:40594) 2023-11-16T16:48:16.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.669+0000 7f6f94c04700 1 -- 172.21.15.2:0/3262751202 learned_addr learned my addr 172.21.15.2:0/3262751202 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:16.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.669+0000 7f6f8f7fe700 1 --2- 172.21.15.2:0/3262751202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f90100d10 0x7f6f90198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:16.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.669+0000 7f6f94c04700 1 -- 172.21.15.2:0/3262751202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f90100d10 msgr2=0x7f6f90198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:16.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.669+0000 7f6f94c04700 1 --2- 172.21.15.2:0/3262751202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f90100d10 0x7f6f90198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.669+0000 7f6f94c04700 1 -- 172.21.15.2:0/3262751202 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f90100370 msgr2=0x7f6f90198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:16.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.669+0000 7f6f94c04700 1 --2- 172.21.15.2:0/3262751202 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f90100370 0x7f6f90198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.669+0000 7f6f94c04700 1 -- 172.21.15.2:0/3262751202 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f88009d00 con 0x7f6f90105410 2023-11-16T16:48:16.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.669+0000 7f6f8ffff700 1 --2- 172.21.15.2:0/3262751202 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6f90100370 0x7f6f90198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:16.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.669+0000 7f6f94c04700 1 --2- 172.21.15.2:0/3262751202 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f90105410 0x7f6f9019cee0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6f88007820 tx=0x7f6f88007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:16.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.670+0000 7f6f8d7fa700 1 -- 172.21.15.2:0/3262751202 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8800e040 con 0x7f6f90105410 2023-11-16T16:48:16.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.670+0000 7f6f8d7fa700 1 -- 172.21.15.2:0/3262751202 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f8800bcb0 con 0x7f6f90105410 2023-11-16T16:48:16.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.670+0000 7f6f95405700 1 -- 172.21.15.2:0/3262751202 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f901101a0 con 0x7f6f90105410 2023-11-16T16:48:16.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.670+0000 7f6f8d7fa700 1 -- 172.21.15.2:0/3262751202 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f880058a0 con 0x7f6f90105410 2023-11-16T16:48:16.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.670+0000 7f6f95405700 1 -- 172.21.15.2:0/3262751202 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f90110010 con 0x7f6f90105410 2023-11-16T16:48:16.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.672+0000 7f6f95405700 1 -- 172.21.15.2:0/3262751202 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6f900fd0b0 con 0x7f6f90105410 2023-11-16T16:48:16.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.673+0000 7f6f8d7fa700 1 -- 172.21.15.2:0/3262751202 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f6f88004a50 con 0x7f6f90105410 2023-11-16T16:48:16.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.674+0000 7f6f8d7fa700 1 --2- 172.21.15.2:0/3262751202 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6f780a1260 0x7f6f780a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:16.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.675+0000 7f6f8d7fa700 1 -- 172.21.15.2:0/3262751202 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6f880cde40 con 0x7f6f90105410 2023-11-16T16:48:16.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.675+0000 7f6f8ffff700 1 --2- 172.21.15.2:0/3262751202 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6f780a1260 0x7f6f780a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:16.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.675+0000 7f6f8d7fa700 1 --2- 172.21.15.2:0/3262751202 >> [v2:172.21.15.141:6800/1040978026,v1:172.21.15.141:6801/1040978026] conn(0x7f6f780ac270 0x7f6f780ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:16.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.675+0000 7f6f8d7fa700 1 -- 172.21.15.2:0/3262751202 --> [v2:172.21.15.141:6800/1040978026,v1:172.21.15.141:6801/1040978026] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f780aedd0 con 0x7f6f780ac270 2023-11-16T16:48:16.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.675+0000 7f6f8d7fa700 1 -- 172.21.15.2:0/3262751202 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f880d0420 con 0x7f6f90105410 2023-11-16T16:48:16.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.675+0000 7f6f8f7fe700 1 --2- 172.21.15.2:0/3262751202 >> [v2:172.21.15.141:6800/1040978026,v1:172.21.15.141:6801/1040978026] conn(0x7f6f780ac270 0x7f6f780ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:16.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.675+0000 7f6f8ffff700 1 --2- 172.21.15.2:0/3262751202 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6f780a1260 0x7f6f780a3710 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6f80011490 tx=0x7f6f80009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:16.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.676+0000 7f6f8f7fe700 1 --2- 172.21.15.2:0/3262751202 >> [v2:172.21.15.141:6800/1040978026,v1:172.21.15.141:6801/1040978026] conn(0x7f6f780ac270 0x7f6f780ae700 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:16.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.677+0000 7f6f8d7fa700 1 -- 172.21.15.2:0/3262751202 <== osd.1 v2:172.21.15.141:6800/1040978026 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f6f780aedd0 con 0x7f6f780ac270 2023-11-16T16:48:16.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.703+0000 7f6f95405700 1 -- 172.21.15.2:0/3262751202 --> [v2:172.21.15.141:6800/1040978026,v1:172.21.15.141:6801/1040978026] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6f90110330 con 0x7f6f780ac270 2023-11-16T16:48:16.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.703+0000 7f6f8d7fa700 1 -- 172.21.15.2:0/3262751202 <== osd.1 v2:172.21.15.141:6800/1040978026 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6f90110330 con 0x7f6f780ac270 2023-11-16T16:48:16.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.704+0000 7f6f95405700 1 -- 172.21.15.2:0/3262751202 >> [v2:172.21.15.141:6800/1040978026,v1:172.21.15.141:6801/1040978026] conn(0x7f6f780ac270 msgr2=0x7f6f780ae700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:16.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.704+0000 7f6f95405700 1 --2- 172.21.15.2:0/3262751202 >> [v2:172.21.15.141:6800/1040978026,v1:172.21.15.141:6801/1040978026] conn(0x7f6f780ac270 0x7f6f780ae700 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:16.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.704+0000 7f6f95405700 1 -- 172.21.15.2:0/3262751202 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6f780a1260 msgr2=0x7f6f780a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:16.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.704+0000 7f6f95405700 1 --2- 172.21.15.2:0/3262751202 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f6f780a1260 0x7f6f780a3710 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6f80011490 tx=0x7f6f80009260 comp rx=0 tx=0).stop 2023-11-16T16:48:16.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.704+0000 7f6f95405700 1 -- 172.21.15.2:0/3262751202 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f90105410 msgr2=0x7f6f9019cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:16.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.704+0000 7f6f95405700 1 --2- 172.21.15.2:0/3262751202 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f90105410 0x7f6f9019cee0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6f88007820 tx=0x7f6f88007850 comp rx=0 tx=0).stop 2023-11-16T16:48:16.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.704+0000 7f6f94c04700 1 -- 172.21.15.2:0/3262751202 reap_dead start 2023-11-16T16:48:16.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.705+0000 7f6f95405700 1 -- 172.21.15.2:0/3262751202 shutdown_connections 2023-11-16T16:48:16.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.705+0000 7f6f95405700 1 -- 172.21.15.2:0/3262751202 >> 172.21.15.2:0/3262751202 conn(0x7f6f900fbbd0 msgr2=0x7f6f900fc660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:16.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.705+0000 7f6f95405700 1 -- 172.21.15.2:0/3262751202 shutdown_connections 2023-11-16T16:48:16.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:16.705+0000 7f6f95405700 1 -- 172.21.15.2:0/3262751202 wait complete. 2023-11-16T16:48:17.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:16 smithi155 ceph-mon[131607]: pgmap v307: 129 pgs: 129 active+clean; 654 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.4 KiB/s wr, 8 op/s 2023-11-16T16:48:17.158 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:16 smithi002 ceph-mon[128712]: pgmap v307: 129 pgs: 129 active+clean; 654 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.4 KiB/s wr, 8 op/s 2023-11-16T16:48:17.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:16 smithi141 ceph-mon[131184]: pgmap v307: 129 pgs: 129 active+clean; 654 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.4 KiB/s wr, 8 op/s 2023-11-16T16:48:18.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:18 smithi002 ceph-mon[128712]: pgmap v308: 129 pgs: 129 active+clean; 654 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 5 op/s 2023-11-16T16:48:19.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.010+0000 7ff8231a8700 1 -- 172.21.15.2:0/620428841 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff81c100d10 msgr2=0x7ff81c104ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.010+0000 7ff8231a8700 1 --2- 172.21.15.2:0/620428841 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff81c100d10 0x7ff81c104ce0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff80c009920 tx=0x7ff80c009c30 comp rx=0 tx=0).stop 2023-11-16T16:48:19.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.010+0000 7ff8231a8700 1 -- 172.21.15.2:0/620428841 shutdown_connections 2023-11-16T16:48:19.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.010+0000 7ff8231a8700 1 --2- 172.21.15.2:0/620428841 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff81c105410 0x7ff81c10cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.010+0000 7ff8231a8700 1 --2- 172.21.15.2:0/620428841 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff81c100370 0x7ff81c100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.010+0000 7ff8231a8700 1 --2- 172.21.15.2:0/620428841 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff81c100d10 0x7ff81c104ce0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.012 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.010+0000 7ff8231a8700 1 -- 172.21.15.2:0/620428841 >> 172.21.15.2:0/620428841 conn(0x7ff81c0fbbd0 msgr2=0x7ff81c0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:19.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.010+0000 7ff8231a8700 1 -- 172.21.15.2:0/620428841 shutdown_connections 2023-11-16T16:48:19.013 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.010+0000 7ff8231a8700 1 -- 172.21.15.2:0/620428841 wait complete. 2023-11-16T16:48:19.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.011+0000 7ff8231a8700 1 Processor -- start 2023-11-16T16:48:19.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.011+0000 7ff8231a8700 1 -- start start 2023-11-16T16:48:19.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.011+0000 7ff8231a8700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff81c100370 0x7ff81c1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.011+0000 7ff8231a8700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff81c100d10 0x7ff81c198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.011+0000 7ff8231a8700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff81c105410 0x7ff81c19cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.011+0000 7ff8231a8700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81c19d540 con 0x7ff81c100d10 2023-11-16T16:48:19.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.011+0000 7ff8231a8700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81c19d6b0 con 0x7ff81c100370 2023-11-16T16:48:19.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.011+0000 7ff8231a8700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81c19d870 con 0x7ff81c105410 2023-11-16T16:48:19.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.012+0000 7ff8219a5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff81c100d10 0x7ff81c198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.012+0000 7ff8221a6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff81c100370 0x7ff81c1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.012+0000 7ff8221a6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff81c100370 0x7ff81c1985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:46164/0 (socket says 172.21.15.2:46164) 2023-11-16T16:48:19.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.012+0000 7ff8219a5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff81c100d10 0x7ff81c198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40650/0 (socket says 172.21.15.2:40650) 2023-11-16T16:48:19.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.012+0000 7ff8221a6700 1 -- 172.21.15.2:0/77481371 learned_addr learned my addr 172.21.15.2:0/77481371 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:19.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.012+0000 7ff8229a7700 1 --2- 172.21.15.2:0/77481371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff81c105410 0x7ff81c19cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.012+0000 7ff8221a6700 1 -- 172.21.15.2:0/77481371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff81c105410 msgr2=0x7ff81c19cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.012+0000 7ff8221a6700 1 --2- 172.21.15.2:0/77481371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff81c105410 0x7ff81c19cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.012+0000 7ff8221a6700 1 -- 172.21.15.2:0/77481371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff81c100d10 msgr2=0x7ff81c198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.012+0000 7ff8221a6700 1 --2- 172.21.15.2:0/77481371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff81c100d10 0x7ff81c198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.012+0000 7ff8221a6700 1 -- 172.21.15.2:0/77481371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8180095f0 con 0x7ff81c100370 2023-11-16T16:48:19.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.012+0000 7ff8219a5700 1 --2- 172.21.15.2:0/77481371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff81c100d10 0x7ff81c198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:19.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.012+0000 7ff8229a7700 1 --2- 172.21.15.2:0/77481371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff81c105410 0x7ff81c19cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:48:19.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.013+0000 7ff8221a6700 1 --2- 172.21.15.2:0/77481371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff81c100370 0x7ff81c1985f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff81800ec10 tx=0x7ff81800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:19.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.013+0000 7ff8137fe700 1 -- 172.21.15.2:0/77481371 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81800cd40 con 0x7ff81c100370 2023-11-16T16:48:19.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.013+0000 7ff8231a8700 1 -- 172.21.15.2:0/77481371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80c009520 con 0x7ff81c100370 2023-11-16T16:48:19.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.013+0000 7ff8137fe700 1 -- 172.21.15.2:0/77481371 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff8180044e0 con 0x7ff81c100370 2023-11-16T16:48:19.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.013+0000 7ff8137fe700 1 -- 172.21.15.2:0/77481371 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff818010690 con 0x7ff81c100370 2023-11-16T16:48:19.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.013+0000 7ff8231a8700 1 -- 172.21.15.2:0/77481371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff81c110190 con 0x7ff81c100370 2023-11-16T16:48:19.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.014+0000 7ff8231a8700 1 -- 172.21.15.2:0/77481371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff81c073ff0 con 0x7ff81c100370 2023-11-16T16:48:19.018 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.016+0000 7ff8137fe700 1 -- 172.21.15.2:0/77481371 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7ff818003710 con 0x7ff81c100370 2023-11-16T16:48:19.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.017+0000 7ff8137fe700 1 --2- 172.21.15.2:0/77481371 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff8080a1260 0x7ff8080a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.017+0000 7ff8137fe700 1 -- 172.21.15.2:0/77481371 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff818014070 con 0x7ff81c100370 2023-11-16T16:48:19.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.018+0000 7ff8219a5700 1 --2- 172.21.15.2:0/77481371 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff8080a1260 0x7ff8080a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.018+0000 7ff8137fe700 1 --2- 172.21.15.2:0/77481371 >> [v2:172.21.15.141:6824/1084001742,v1:172.21.15.141:6825/1084001742] conn(0x7ff8080ac270 0x7ff8080ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.018+0000 7ff8137fe700 1 -- 172.21.15.2:0/77481371 --> [v2:172.21.15.141:6824/1084001742,v1:172.21.15.141:6825/1084001742] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff8080aedd0 con 0x7ff8080ac270 2023-11-16T16:48:19.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.018+0000 7ff8137fe700 1 -- 172.21.15.2:0/77481371 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8180cd780 con 0x7ff81c100370 2023-11-16T16:48:19.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.018+0000 7ff8229a7700 1 --2- 172.21.15.2:0/77481371 >> [v2:172.21.15.141:6824/1084001742,v1:172.21.15.141:6825/1084001742] conn(0x7ff8080ac270 0x7ff8080ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.018+0000 7ff8219a5700 1 --2- 172.21.15.2:0/77481371 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff8080a1260 0x7ff8080a3710 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff81c199ba0 tx=0x7ff80c019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:19.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.019+0000 7ff8229a7700 1 --2- 172.21.15.2:0/77481371 >> [v2:172.21.15.141:6824/1084001742,v1:172.21.15.141:6825/1084001742] conn(0x7ff8080ac270 0x7ff8080ae700 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:19.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.020+0000 7ff8137fe700 1 -- 172.21.15.2:0/77481371 <== osd.10 v2:172.21.15.141:6824/1084001742 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7ff8080aedd0 con 0x7ff8080ac270 2023-11-16T16:48:19.038 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.036+0000 7ff8231a8700 1 -- 172.21.15.2:0/77481371 --> [v2:172.21.15.141:6824/1084001742,v1:172.21.15.141:6825/1084001742] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff81c110000 con 0x7ff8080ac270 2023-11-16T16:48:19.039 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.037+0000 7ff8137fe700 1 -- 172.21.15.2:0/77481371 <== osd.10 v2:172.21.15.141:6824/1084001742 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff81c110000 con 0x7ff8080ac270 2023-11-16T16:48:19.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.037+0000 7ff8231a8700 1 -- 172.21.15.2:0/77481371 >> [v2:172.21.15.141:6824/1084001742,v1:172.21.15.141:6825/1084001742] conn(0x7ff8080ac270 msgr2=0x7ff8080ae700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.037+0000 7ff8231a8700 1 --2- 172.21.15.2:0/77481371 >> [v2:172.21.15.141:6824/1084001742,v1:172.21.15.141:6825/1084001742] conn(0x7ff8080ac270 0x7ff8080ae700 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.038+0000 7ff8231a8700 1 -- 172.21.15.2:0/77481371 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff8080a1260 msgr2=0x7ff8080a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.038+0000 7ff8231a8700 1 --2- 172.21.15.2:0/77481371 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff8080a1260 0x7ff8080a3710 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff81c199ba0 tx=0x7ff80c019040 comp rx=0 tx=0).stop 2023-11-16T16:48:19.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.038+0000 7ff8231a8700 1 -- 172.21.15.2:0/77481371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff81c100370 msgr2=0x7ff81c1985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.038+0000 7ff8231a8700 1 --2- 172.21.15.2:0/77481371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff81c100370 0x7ff81c1985f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff81800ec10 tx=0x7ff81800ef20 comp rx=0 tx=0).stop 2023-11-16T16:48:19.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.038+0000 7ff8229a7700 1 -- 172.21.15.2:0/77481371 reap_dead start 2023-11-16T16:48:19.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.038+0000 7ff8231a8700 1 -- 172.21.15.2:0/77481371 shutdown_connections 2023-11-16T16:48:19.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.038+0000 7ff8231a8700 1 -- 172.21.15.2:0/77481371 >> 172.21.15.2:0/77481371 conn(0x7ff81c0fbbd0 msgr2=0x7ff81c0fc930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:19.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.038+0000 7ff8231a8700 1 -- 172.21.15.2:0/77481371 shutdown_connections 2023-11-16T16:48:19.041 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:19.038+0000 7ff8231a8700 1 -- 172.21.15.2:0/77481371 wait complete. 2023-11-16T16:48:19.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:18 smithi155 ceph-mon[131607]: pgmap v308: 129 pgs: 129 active+clean; 654 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 5 op/s 2023-11-16T16:48:19.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:18 smithi141 ceph-mon[131184]: pgmap v308: 129 pgs: 129 active+clean; 654 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 5 op/s 2023-11-16T16:48:19.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.593+0000 7f8b89171700 1 -- 172.21.15.2:0/3914822656 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b841055e0 msgr2=0x7f8b8410d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.855 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.593+0000 7f8b89171700 1 --2- 172.21.15.2:0/3914822656 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b841055e0 0x7f8b8410d160 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00b0d0 tx=0x7f8b7c00b3e0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.593+0000 7f8b89171700 1 -- 172.21.15.2:0/3914822656 shutdown_connections 2023-11-16T16:48:19.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.593+0000 7f8b89171700 1 --2- 172.21.15.2:0/3914822656 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b841055e0 0x7f8b8410d160 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.593+0000 7f8b89171700 1 --2- 172.21.15.2:0/3914822656 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b84100540 0x7f8b84100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.593+0000 7f8b89171700 1 --2- 172.21.15.2:0/3914822656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b84100ee0 0x7f8b84104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.593+0000 7f8b89171700 1 -- 172.21.15.2:0/3914822656 >> 172.21.15.2:0/3914822656 conn(0x7f8b840fbdc0 msgr2=0x7f8b840fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:19.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.593+0000 7f8b89171700 1 -- 172.21.15.2:0/3914822656 shutdown_connections 2023-11-16T16:48:19.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.593+0000 7f8b89171700 1 -- 172.21.15.2:0/3914822656 wait complete. 2023-11-16T16:48:19.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.595+0000 7f8b89171700 1 Processor -- start 2023-11-16T16:48:19.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.595+0000 7f8b89171700 1 -- start start 2023-11-16T16:48:19.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.595+0000 7f8b89171700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b84100540 0x7f8b84196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.595+0000 7f8b89171700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b84100ee0 0x7f8b841969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.595+0000 7f8b89171700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b841055e0 0x7f8b8419ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.595+0000 7f8b89171700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8419b420 con 0x7f8b84100ee0 2023-11-16T16:48:19.857 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.595+0000 7f8b89171700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8419b590 con 0x7f8b84100540 2023-11-16T16:48:19.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.595+0000 7f8b89171700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8419b750 con 0x7f8b841055e0 2023-11-16T16:48:19.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.596+0000 7f8b82d9d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b84100540 0x7f8b84196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.596+0000 7f8b82d9d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b84100540 0x7f8b84196460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:46156/0 (socket says 172.21.15.2:46156) 2023-11-16T16:48:19.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.596+0000 7f8b8259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b84100ee0 0x7f8b841969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.596+0000 7f8b82d9d700 1 -- 172.21.15.2:0/867974920 learned_addr learned my addr 172.21.15.2:0/867974920 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:19.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.596+0000 7f8b8359e700 1 --2- 172.21.15.2:0/867974920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b841055e0 0x7f8b8419ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.596+0000 7f8b8259c700 1 -- 172.21.15.2:0/867974920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b841055e0 msgr2=0x7f8b8419ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.596+0000 7f8b8259c700 1 --2- 172.21.15.2:0/867974920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b841055e0 0x7f8b8419ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.596+0000 7f8b8259c700 1 -- 172.21.15.2:0/867974920 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b84100540 msgr2=0x7f8b84196460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.596+0000 7f8b8259c700 1 --2- 172.21.15.2:0/867974920 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b84100540 0x7f8b84196460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.596+0000 7f8b8259c700 1 -- 172.21.15.2:0/867974920 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b7c009d00 con 0x7f8b84100ee0 2023-11-16T16:48:19.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.596+0000 7f8b8359e700 1 --2- 172.21.15.2:0/867974920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b841055e0 0x7f8b8419ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:48:19.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.596+0000 7f8b82d9d700 1 --2- 172.21.15.2:0/867974920 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8b84100540 0x7f8b84196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:19.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.597+0000 7f8b8259c700 1 --2- 172.21.15.2:0/867974920 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b84100ee0 0x7f8b841969a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8b7800e9c0 tx=0x7f8b7800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:19.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.597+0000 7f8b6bfff700 1 -- 172.21.15.2:0/867974920 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7800cbd0 con 0x7f8b84100ee0 2023-11-16T16:48:19.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.597+0000 7f8b6bfff700 1 -- 172.21.15.2:0/867974920 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b7800eeb0 con 0x7f8b84100ee0 2023-11-16T16:48:19.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.597+0000 7f8b89171700 1 -- 172.21.15.2:0/867974920 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b84110320 con 0x7f8b84100ee0 2023-11-16T16:48:19.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.597+0000 7f8b6bfff700 1 -- 172.21.15.2:0/867974920 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b78018680 con 0x7f8b84100ee0 2023-11-16T16:48:19.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.597+0000 7f8b89171700 1 -- 172.21.15.2:0/867974920 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b84110190 con 0x7f8b84100ee0 2023-11-16T16:48:19.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.598+0000 7f8b89171700 1 -- 172.21.15.2:0/867974920 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8b840fd190 con 0x7f8b84100ee0 2023-11-16T16:48:19.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.602+0000 7f8b6bfff700 1 -- 172.21.15.2:0/867974920 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f8b78010430 con 0x7f8b84100ee0 2023-11-16T16:48:19.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.603+0000 7f8b6bfff700 1 --2- 172.21.15.2:0/867974920 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8b6c0a11e0 0x7f8b6c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.603+0000 7f8b6bfff700 1 -- 172.21.15.2:0/867974920 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8b78014070 con 0x7f8b84100ee0 2023-11-16T16:48:19.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.604+0000 7f8b82d9d700 1 --2- 172.21.15.2:0/867974920 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8b6c0a11e0 0x7f8b6c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.604+0000 7f8b6bfff700 1 --2- 172.21.15.2:0/867974920 >> [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480] conn(0x7f8b6c0ac1f0 0x7f8b6c0ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:19.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.604+0000 7f8b6bfff700 1 -- 172.21.15.2:0/867974920 --> [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b6c0aed50 con 0x7f8b6c0ac1f0 2023-11-16T16:48:19.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.604+0000 7f8b6bfff700 1 -- 172.21.15.2:0/867974920 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b780cd600 con 0x7f8b84100ee0 2023-11-16T16:48:19.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.604+0000 7f8b8359e700 1 --2- 172.21.15.2:0/867974920 >> [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480] conn(0x7f8b6c0ac1f0 0x7f8b6c0ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:19.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.604+0000 7f8b82d9d700 1 --2- 172.21.15.2:0/867974920 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8b6c0a11e0 0x7f8b6c0a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8b74011550 tx=0x7f8b74009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:19.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.605+0000 7f8b8359e700 1 --2- 172.21.15.2:0/867974920 >> [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480] conn(0x7f8b6c0ac1f0 0x7f8b6c0ae680 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:19.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.606+0000 7f8b6bfff700 1 -- 172.21.15.2:0/867974920 <== osd.3 v2:172.21.15.2:6810/2757259480 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8b6c0aed50 con 0x7f8b6c0ac1f0 2023-11-16T16:48:19.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.623+0000 7f8b89171700 1 -- 172.21.15.2:0/867974920 --> [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8b841104b0 con 0x7f8b6c0ac1f0 2023-11-16T16:48:19.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.623+0000 7f8b6bfff700 1 -- 172.21.15.2:0/867974920 <== osd.3 v2:172.21.15.2:6810/2757259480 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8b841104b0 con 0x7f8b6c0ac1f0 2023-11-16T16:48:19.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.623+0000 7f8b89171700 1 -- 172.21.15.2:0/867974920 >> [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480] conn(0x7f8b6c0ac1f0 msgr2=0x7f8b6c0ae680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.623+0000 7f8b89171700 1 --2- 172.21.15.2:0/867974920 >> [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480] conn(0x7f8b6c0ac1f0 0x7f8b6c0ae680 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.623+0000 7f8b89171700 1 -- 172.21.15.2:0/867974920 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8b6c0a11e0 msgr2=0x7f8b6c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.623+0000 7f8b89171700 1 --2- 172.21.15.2:0/867974920 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8b6c0a11e0 0x7f8b6c0a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8b74011550 tx=0x7f8b74009260 comp rx=0 tx=0).stop 2023-11-16T16:48:19.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.624+0000 7f8b89171700 1 -- 172.21.15.2:0/867974920 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b84100ee0 msgr2=0x7f8b841969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:19.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.624+0000 7f8b89171700 1 --2- 172.21.15.2:0/867974920 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b84100ee0 0x7f8b841969a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8b7800e9c0 tx=0x7f8b7800ecd0 comp rx=0 tx=0).stop 2023-11-16T16:48:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.624+0000 7f8b8359e700 1 -- 172.21.15.2:0/867974920 reap_dead start 2023-11-16T16:48:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.624+0000 7f8b89171700 1 -- 172.21.15.2:0/867974920 shutdown_connections 2023-11-16T16:48:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.624+0000 7f8b89171700 1 -- 172.21.15.2:0/867974920 >> 172.21.15.2:0/867974920 conn(0x7f8b840fbdc0 msgr2=0x7f8b840fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.624+0000 7f8b89171700 1 -- 172.21.15.2:0/867974920 shutdown_connections 2023-11-16T16:48:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:18.624+0000 7f8b89171700 1 -- 172.21.15.2:0/867974920 wait complete. 2023-11-16T16:48:20.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:20 smithi002 ceph-mon[128712]: pgmap v309: 129 pgs: 129 active+clean; 654 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 7 op/s 2023-11-16T16:48:20.764 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:20 smithi141 ceph-mon[131184]: pgmap v309: 129 pgs: 129 active+clean; 654 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 7 op/s 2023-11-16T16:48:20.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:20 smithi155 ceph-mon[131607]: pgmap v309: 129 pgs: 129 active+clean; 654 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 7 op/s 2023-11-16T16:48:20.974 INFO:teuthology.orchestra.run.smithi002.stdout:55834574918 2023-11-16T16:48:20.975 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.1 2023-11-16T16:48:21.180 INFO:teuthology.orchestra.run.smithi002.stdout:107374182460 2023-11-16T16:48:21.180 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.3 2023-11-16T16:48:21.290 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.309+0000 7ffaf4f2a700 1 -- 172.21.15.2:0/1731663379 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf00ff120 msgr2=0x7ffaf00ff4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:21.290 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.309+0000 7ffaf4f2a700 1 --2- 172.21.15.2:0/1731663379 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf00ff120 0x7ffaf00ff4f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffae0009990 tx=0x7ffae0009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:21.291 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.310+0000 7ffaf4f2a700 1 -- 172.21.15.2:0/1731663379 shutdown_connections 2023-11-16T16:48:21.291 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.310+0000 7ffaf4f2a700 1 --2- 172.21.15.2:0/1731663379 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffaf0104450 0x7ffaf01048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:21.291 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.310+0000 7ffaf4f2a700 1 --2- 172.21.15.2:0/1731663379 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf00ff120 0x7ffaf00ff4f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:21.291 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.310+0000 7ffaf4f2a700 1 --2- 172.21.15.2:0/1731663379 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffaf00ffa30 0x7ffaf0103f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:21.291 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.310+0000 7ffaf4f2a700 1 -- 172.21.15.2:0/1731663379 >> 172.21.15.2:0/1731663379 conn(0x7ffaf00761a0 msgr2=0x7ffaf00765a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:21.291 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.310+0000 7ffaf4f2a700 1 -- 172.21.15.2:0/1731663379 shutdown_connections 2023-11-16T16:48:21.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.310+0000 7ffaf4f2a700 1 -- 172.21.15.2:0/1731663379 wait complete. 2023-11-16T16:48:21.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.311+0000 7ffaf4f2a700 1 Processor -- start 2023-11-16T16:48:21.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.311+0000 7ffaf4f2a700 1 -- start start 2023-11-16T16:48:21.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.311+0000 7ffaf4f2a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffaf00ff120 0x7ffaf01a2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:21.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.311+0000 7ffaf4f2a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf00ffa30 0x7ffaf01a2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:21.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.311+0000 7ffaf4f2a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffaf0104450 0x7ffaf019c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:21.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.311+0000 7ffaf4f2a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf019cc80 con 0x7ffaf0104450 2023-11-16T16:48:21.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.311+0000 7ffaf4f2a700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf019cdf0 con 0x7ffaf00ffa30 2023-11-16T16:48:21.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.311+0000 7ffaf4f2a700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf019cfb0 con 0x7ffaf00ff120 2023-11-16T16:48:21.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.312+0000 7ffaeeffd700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf00ffa30 0x7ffaf01a2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:21.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.312+0000 7ffaeeffd700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf00ffa30 0x7ffaf01a2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:46192/0 (socket says 172.21.15.2:46192) 2023-11-16T16:48:21.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.312+0000 7ffaeffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffaf0104450 0x7ffaf019c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:21.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.312+0000 7ffaeeffd700 1 -- 172.21.15.2:0/334618693 learned_addr learned my addr 172.21.15.2:0/334618693 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:21.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.312+0000 7ffaeffff700 1 -- 172.21.15.2:0/334618693 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffaf00ff120 msgr2=0x7ffaf01a2590 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-16T16:48:21.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.312+0000 7ffaeffff700 1 --2- 172.21.15.2:0/334618693 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffaf00ff120 0x7ffaf01a2590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:21.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.312+0000 7ffaeffff700 1 -- 172.21.15.2:0/334618693 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf00ffa30 msgr2=0x7ffaf01a2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:21.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.312+0000 7ffaeffff700 1 --2- 172.21.15.2:0/334618693 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf00ffa30 0x7ffaf01a2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:21.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.312+0000 7ffaeffff700 1 -- 172.21.15.2:0/334618693 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae4009520 con 0x7ffaf0104450 2023-11-16T16:48:21.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.312+0000 7ffaeeffd700 1 --2- 172.21.15.2:0/334618693 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffaf00ffa30 0x7ffaf01a2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:21.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.312+0000 7ffaeffff700 1 --2- 172.21.15.2:0/334618693 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffaf0104450 0x7ffaf019c610 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ffae800efc0 tx=0x7ffae800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:21.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.313+0000 7ffaecff9700 1 -- 172.21.15.2:0/334618693 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae8018070 con 0x7ffaf0104450 2023-11-16T16:48:21.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.313+0000 7ffaf4f2a700 1 -- 172.21.15.2:0/334618693 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae00095f0 con 0x7ffaf0104450 2023-11-16T16:48:21.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.313+0000 7ffaecff9700 1 -- 172.21.15.2:0/334618693 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffae800f040 con 0x7ffaf0104450 2023-11-16T16:48:21.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.313+0000 7ffaecff9700 1 -- 172.21.15.2:0/334618693 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae800cd30 con 0x7ffaf0104450 2023-11-16T16:48:21.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.314+0000 7ffaf4f2a700 1 -- 172.21.15.2:0/334618693 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf0116710 con 0x7ffaf0104450 2023-11-16T16:48:21.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.315+0000 7ffaf4f2a700 1 -- 172.21.15.2:0/334618693 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffaf01039b0 con 0x7ffaf0104450 2023-11-16T16:48:21.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.316+0000 7ffaecff9700 1 -- 172.21.15.2:0/334618693 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7ffae8004000 con 0x7ffaf0104450 2023-11-16T16:48:21.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.318+0000 7ffaecff9700 1 --2- 172.21.15.2:0/334618693 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ffad80a11e0 0x7ffad80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:21.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.318+0000 7ffaecff9700 1 -- 172.21.15.2:0/334618693 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ffae80cc750 con 0x7ffaf0104450 2023-11-16T16:48:21.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.318+0000 7ffaef7fe700 1 --2- 172.21.15.2:0/334618693 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ffad80a11e0 0x7ffad80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:21.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.318+0000 7ffaecff9700 1 --2- 172.21.15.2:0/334618693 >> [v2:172.21.15.155:6824/1791652726,v1:172.21.15.155:6825/1791652726] conn(0x7ffad80ac170 0x7ffad80ae600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:21.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.318+0000 7ffaecff9700 1 -- 172.21.15.2:0/334618693 --> [v2:172.21.15.155:6824/1791652726,v1:172.21.15.155:6825/1791652726] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffad80aecd0 con 0x7ffad80ac170 2023-11-16T16:48:21.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.318+0000 7ffaecff9700 1 -- 172.21.15.2:0/334618693 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffae80ced30 con 0x7ffaf0104450 2023-11-16T16:48:21.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.319+0000 7ffaeeffd700 1 --2- 172.21.15.2:0/334618693 >> [v2:172.21.15.155:6824/1791652726,v1:172.21.15.155:6825/1791652726] conn(0x7ffad80ac170 0x7ffad80ae600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:21.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.319+0000 7ffaef7fe700 1 --2- 172.21.15.2:0/334618693 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ffad80a11e0 0x7ffad80a3690 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ffae00098e0 tx=0x7ffae0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:21.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.319+0000 7ffaeeffd700 1 --2- 172.21.15.2:0/334618693 >> [v2:172.21.15.155:6824/1791652726,v1:172.21.15.155:6825/1791652726] conn(0x7ffad80ac170 0x7ffad80ae600 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:21.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.320+0000 7ffaecff9700 1 -- 172.21.15.2:0/334618693 <== osd.11 v2:172.21.15.155:6824/1791652726 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7ffad80aecd0 con 0x7ffad80ac170 2023-11-16T16:48:21.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.337+0000 7ffaf4f2a700 1 -- 172.21.15.2:0/334618693 --> [v2:172.21.15.155:6824/1791652726,v1:172.21.15.155:6825/1791652726] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ffaf0116580 con 0x7ffad80ac170 2023-11-16T16:48:21.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.338+0000 7ffaecff9700 1 -- 172.21.15.2:0/334618693 <== osd.11 v2:172.21.15.155:6824/1791652726 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ffaf0116580 con 0x7ffad80ac170 2023-11-16T16:48:21.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.338+0000 7ffaf4f2a700 1 -- 172.21.15.2:0/334618693 >> [v2:172.21.15.155:6824/1791652726,v1:172.21.15.155:6825/1791652726] conn(0x7ffad80ac170 msgr2=0x7ffad80ae600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:21.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.338+0000 7ffaf4f2a700 1 --2- 172.21.15.2:0/334618693 >> [v2:172.21.15.155:6824/1791652726,v1:172.21.15.155:6825/1791652726] conn(0x7ffad80ac170 0x7ffad80ae600 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:21.298 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.338+0000 7ffaf4f2a700 1 -- 172.21.15.2:0/334618693 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ffad80a11e0 msgr2=0x7ffad80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:21.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.338+0000 7ffaf4f2a700 1 --2- 172.21.15.2:0/334618693 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ffad80a11e0 0x7ffad80a3690 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ffae00098e0 tx=0x7ffae0000bc0 comp rx=0 tx=0).stop 2023-11-16T16:48:21.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.338+0000 7ffaf4f2a700 1 -- 172.21.15.2:0/334618693 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffaf0104450 msgr2=0x7ffaf019c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:21.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.338+0000 7ffaf4f2a700 1 --2- 172.21.15.2:0/334618693 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffaf0104450 0x7ffaf019c610 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ffae800efc0 tx=0x7ffae800c3a0 comp rx=0 tx=0).stop 2023-11-16T16:48:21.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.338+0000 7ffaeffff700 1 -- 172.21.15.2:0/334618693 reap_dead start 2023-11-16T16:48:21.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.338+0000 7ffaf4f2a700 1 -- 172.21.15.2:0/334618693 shutdown_connections 2023-11-16T16:48:21.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.338+0000 7ffaf4f2a700 1 -- 172.21.15.2:0/334618693 >> 172.21.15.2:0/334618693 conn(0x7ffaf00761a0 msgr2=0x7ffaf0101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:21.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.338+0000 7ffaf4f2a700 1 -- 172.21.15.2:0/334618693 shutdown_connections 2023-11-16T16:48:21.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:20.338+0000 7ffaf4f2a700 1 -- 172.21.15.2:0/334618693 wait complete. 2023-11-16T16:48:22.278 INFO:teuthology.orchestra.run.smithi002.stdout:309237645340 2023-11-16T16:48:22.278 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.10 2023-11-16T16:48:22.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.780+0000 7f8e370e0700 1 -- 172.21.15.2:0/2375579756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e301055e0 msgr2=0x7f8e3010d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:22.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.780+0000 7f8e370e0700 1 --2- 172.21.15.2:0/2375579756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e301055e0 0x7f8e3010d160 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8e2800b0d0 tx=0x7f8e2800b3e0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.780+0000 7f8e370e0700 1 -- 172.21.15.2:0/2375579756 shutdown_connections 2023-11-16T16:48:22.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.780+0000 7f8e370e0700 1 --2- 172.21.15.2:0/2375579756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e301055e0 0x7f8e3010d160 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.780+0000 7f8e370e0700 1 --2- 172.21.15.2:0/2375579756 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e30100540 0x7f8e30100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.780+0000 7f8e370e0700 1 --2- 172.21.15.2:0/2375579756 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e30100ee0 0x7f8e30104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.780+0000 7f8e370e0700 1 -- 172.21.15.2:0/2375579756 >> 172.21.15.2:0/2375579756 conn(0x7f8e300fbdc0 msgr2=0x7f8e300fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.780+0000 7f8e370e0700 1 -- 172.21.15.2:0/2375579756 shutdown_connections 2023-11-16T16:48:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.780+0000 7f8e370e0700 1 -- 172.21.15.2:0/2375579756 wait complete. 2023-11-16T16:48:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.781+0000 7f8e370e0700 1 Processor -- start 2023-11-16T16:48:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.781+0000 7f8e370e0700 1 -- start start 2023-11-16T16:48:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.782+0000 7f8e370e0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e30100540 0x7f8e3019e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.782+0000 7f8e370e0700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e30100ee0 0x7f8e3019eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.782+0000 7f8e370e0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e301055e0 0x7f8e30198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.782+0000 7f8e370e0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e30198d10 con 0x7f8e301055e0 2023-11-16T16:48:22.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.782+0000 7f8e370e0700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e30198e80 con 0x7f8e30100ee0 2023-11-16T16:48:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.782+0000 7f8e370e0700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e30199040 con 0x7f8e30100540 2023-11-16T16:48:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.782+0000 7f8e3567d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e301055e0 0x7f8e30198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.782+0000 7f8e34e7c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e30100540 0x7f8e3019e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.782+0000 7f8e34e7c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e30100540 0x7f8e3019e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:43056/0 (socket says 172.21.15.2:43056) 2023-11-16T16:48:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.782+0000 7f8e34e7c700 1 -- 172.21.15.2:0/3051934565 learned_addr learned my addr 172.21.15.2:0/3051934565 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.782+0000 7f8e2ffff700 1 --2- 172.21.15.2:0/3051934565 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e30100ee0 0x7f8e3019eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.782+0000 7f8e3567d700 1 -- 172.21.15.2:0/3051934565 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e30100540 msgr2=0x7f8e3019e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.782+0000 7f8e3567d700 1 --2- 172.21.15.2:0/3051934565 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e30100540 0x7f8e3019e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.782+0000 7f8e3567d700 1 -- 172.21.15.2:0/3051934565 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e30100ee0 msgr2=0x7f8e3019eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.782+0000 7f8e3567d700 1 --2- 172.21.15.2:0/3051934565 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e30100ee0 0x7f8e3019eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.782+0000 7f8e3567d700 1 -- 172.21.15.2:0/3051934565 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e28009d00 con 0x7f8e301055e0 2023-11-16T16:48:22.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.782+0000 7f8e2ffff700 1 --2- 172.21.15.2:0/3051934565 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8e30100ee0 0x7f8e3019eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:48:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.783+0000 7f8e3567d700 1 --2- 172.21.15.2:0/3051934565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e301055e0 0x7f8e30198560 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8e28009fd0 tx=0x7f8e28007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.783+0000 7f8e2dffb700 1 -- 172.21.15.2:0/3051934565 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2800e040 con 0x7f8e301055e0 2023-11-16T16:48:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.783+0000 7f8e370e0700 1 -- 172.21.15.2:0/3051934565 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e30110370 con 0x7f8e301055e0 2023-11-16T16:48:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.783+0000 7f8e2dffb700 1 -- 172.21.15.2:0/3051934565 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e2800bd30 con 0x7f8e301055e0 2023-11-16T16:48:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.783+0000 7f8e2dffb700 1 -- 172.21.15.2:0/3051934565 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2801d470 con 0x7f8e301055e0 2023-11-16T16:48:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.783+0000 7f8e370e0700 1 -- 172.21.15.2:0/3051934565 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e301101e0 con 0x7f8e301055e0 2023-11-16T16:48:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.784+0000 7f8e370e0700 1 -- 172.21.15.2:0/3051934565 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8e300fd0b0 con 0x7f8e301055e0 2023-11-16T16:48:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.786+0000 7f8e2dffb700 1 -- 172.21.15.2:0/3051934565 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f8e28005000 con 0x7f8e301055e0 2023-11-16T16:48:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.788+0000 7f8e2dffb700 1 --2- 172.21.15.2:0/3051934565 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8e200a11e0 0x7f8e200a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.788+0000 7f8e2dffb700 1 -- 172.21.15.2:0/3051934565 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8e280ce7b0 con 0x7f8e301055e0 2023-11-16T16:48:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.789+0000 7f8e34e7c700 1 --2- 172.21.15.2:0/3051934565 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8e200a11e0 0x7f8e200a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:22.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.789+0000 7f8e2dffb700 1 --2- 172.21.15.2:0/3051934565 >> [v2:172.21.15.155:6800/844890823,v1:172.21.15.155:6801/844890823] conn(0x7f8e200ac1f0 0x7f8e200ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:22.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.789+0000 7f8e2dffb700 1 -- 172.21.15.2:0/3051934565 --> [v2:172.21.15.155:6800/844890823,v1:172.21.15.155:6801/844890823] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e200aed50 con 0x7f8e200ac1f0 2023-11-16T16:48:22.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.789+0000 7f8e2dffb700 1 -- 172.21.15.2:0/3051934565 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e280ceb20 con 0x7f8e301055e0 2023-11-16T16:48:22.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.789+0000 7f8e2ffff700 1 --2- 172.21.15.2:0/3051934565 >> [v2:172.21.15.155:6800/844890823,v1:172.21.15.155:6801/844890823] conn(0x7f8e200ac1f0 0x7f8e200ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:22.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.789+0000 7f8e34e7c700 1 --2- 172.21.15.2:0/3051934565 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8e200a11e0 0x7f8e200a3690 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8e24009d20 tx=0x7f8e24009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:22.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.790+0000 7f8e2ffff700 1 --2- 172.21.15.2:0/3051934565 >> [v2:172.21.15.155:6800/844890823,v1:172.21.15.155:6801/844890823] conn(0x7f8e200ac1f0 0x7f8e200ae680 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:22.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.791+0000 7f8e2dffb700 1 -- 172.21.15.2:0/3051934565 <== osd.2 v2:172.21.15.155:6800/844890823 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8e200aed50 con 0x7f8e200ac1f0 2023-11-16T16:48:22.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.808+0000 7f8e370e0700 1 -- 172.21.15.2:0/3051934565 --> [v2:172.21.15.155:6800/844890823,v1:172.21.15.155:6801/844890823] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8e30110500 con 0x7f8e200ac1f0 2023-11-16T16:48:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.809+0000 7f8e2dffb700 1 -- 172.21.15.2:0/3051934565 <== osd.2 v2:172.21.15.155:6800/844890823 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8e30110500 con 0x7f8e200ac1f0 2023-11-16T16:48:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.809+0000 7f8e370e0700 1 -- 172.21.15.2:0/3051934565 >> [v2:172.21.15.155:6800/844890823,v1:172.21.15.155:6801/844890823] conn(0x7f8e200ac1f0 msgr2=0x7f8e200ae680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.809+0000 7f8e370e0700 1 --2- 172.21.15.2:0/3051934565 >> [v2:172.21.15.155:6800/844890823,v1:172.21.15.155:6801/844890823] conn(0x7f8e200ac1f0 0x7f8e200ae680 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.809+0000 7f8e370e0700 1 -- 172.21.15.2:0/3051934565 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8e200a11e0 msgr2=0x7f8e200a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.809+0000 7f8e370e0700 1 --2- 172.21.15.2:0/3051934565 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f8e200a11e0 0x7f8e200a3690 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8e24009d20 tx=0x7f8e24009260 comp rx=0 tx=0).stop 2023-11-16T16:48:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.809+0000 7f8e370e0700 1 -- 172.21.15.2:0/3051934565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e301055e0 msgr2=0x7f8e30198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.809+0000 7f8e370e0700 1 --2- 172.21.15.2:0/3051934565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e301055e0 0x7f8e30198560 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8e28009fd0 tx=0x7f8e28007860 comp rx=0 tx=0).stop 2023-11-16T16:48:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.809+0000 7f8e3567d700 1 -- 172.21.15.2:0/3051934565 reap_dead start 2023-11-16T16:48:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.809+0000 7f8e370e0700 1 -- 172.21.15.2:0/3051934565 shutdown_connections 2023-11-16T16:48:22.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.809+0000 7f8e370e0700 1 -- 172.21.15.2:0/3051934565 >> 172.21.15.2:0/3051934565 conn(0x7f8e300fbdc0 msgr2=0x7f8e300fc720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:22.353 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.810+0000 7f8e370e0700 1 -- 172.21.15.2:0/3051934565 shutdown_connections 2023-11-16T16:48:22.353 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:21.810+0000 7f8e370e0700 1 -- 172.21.15.2:0/3051934565 wait complete. 2023-11-16T16:48:22.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.383+0000 7fecac446700 1 -- 172.21.15.2:0/738486329 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca4100ef0 msgr2=0x7feca4104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:22.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.383+0000 7fecac446700 1 --2- 172.21.15.2:0/738486329 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca4100ef0 0x7feca4104ec0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fec940099c0 tx=0x7fec94009cd0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.383+0000 7fecac446700 1 -- 172.21.15.2:0/738486329 shutdown_connections 2023-11-16T16:48:22.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.383+0000 7fecac446700 1 --2- 172.21.15.2:0/738486329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feca41055f0 0x7feca410d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.383+0000 7fecac446700 1 --2- 172.21.15.2:0/738486329 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feca4100550 0x7feca4100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.383+0000 7fecac446700 1 --2- 172.21.15.2:0/738486329 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca4100ef0 0x7feca4104ec0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.383+0000 7fecac446700 1 -- 172.21.15.2:0/738486329 >> 172.21.15.2:0/738486329 conn(0x7feca40fbdf0 msgr2=0x7feca40fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:22.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.384+0000 7fecac446700 1 -- 172.21.15.2:0/738486329 shutdown_connections 2023-11-16T16:48:22.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.384+0000 7fecac446700 1 -- 172.21.15.2:0/738486329 wait complete. 2023-11-16T16:48:22.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.385+0000 7fecac446700 1 Processor -- start 2023-11-16T16:48:22.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.385+0000 7fecac446700 1 -- start start 2023-11-16T16:48:22.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.385+0000 7fecac446700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feca4100550 0x7feca41985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:22.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.385+0000 7fecac446700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca4100ef0 0x7feca4198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:22.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.385+0000 7fecac446700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feca41055f0 0x7feca419cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:22.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.385+0000 7fecac446700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca419d590 con 0x7feca4100ef0 2023-11-16T16:48:22.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.385+0000 7fecac446700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca419d700 con 0x7feca41055f0 2023-11-16T16:48:22.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.385+0000 7fecac446700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca419d8c0 con 0x7feca4100550 2023-11-16T16:48:22.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.386+0000 7feca99e1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca4100ef0 0x7feca4198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:22.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.386+0000 7fecaa9e3700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feca41055f0 0x7feca419cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:22.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.386+0000 7fecaa9e3700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feca41055f0 0x7feca419cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:46226/0 (socket says 172.21.15.2:46226) 2023-11-16T16:48:22.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.386+0000 7fecaa9e3700 1 -- 172.21.15.2:0/4256737375 learned_addr learned my addr 172.21.15.2:0/4256737375 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:22.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.386+0000 7fecaa1e2700 1 --2- 172.21.15.2:0/4256737375 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feca4100550 0x7feca41985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:22.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.386+0000 7fecaa9e3700 1 -- 172.21.15.2:0/4256737375 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feca4100550 msgr2=0x7feca41985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:22.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.386+0000 7fecaa9e3700 1 --2- 172.21.15.2:0/4256737375 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feca4100550 0x7feca41985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.386+0000 7fecaa9e3700 1 -- 172.21.15.2:0/4256737375 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca4100ef0 msgr2=0x7feca4198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:22.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.386+0000 7fecaa9e3700 1 --2- 172.21.15.2:0/4256737375 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca4100ef0 0x7feca4198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.386+0000 7fecaa9e3700 1 -- 172.21.15.2:0/4256737375 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec940095f0 con 0x7feca41055f0 2023-11-16T16:48:22.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.386+0000 7feca99e1700 1 --2- 172.21.15.2:0/4256737375 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca4100ef0 0x7feca4198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:22.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.387+0000 7fecaa9e3700 1 --2- 172.21.15.2:0/4256737375 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feca41055f0 0x7feca419cf80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fec9c00ef30 tx=0x7fec9c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:22.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.387+0000 7fec9b7fe700 1 -- 172.21.15.2:0/4256737375 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c018070 con 0x7feca41055f0 2023-11-16T16:48:22.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.387+0000 7fec9b7fe700 1 -- 172.21.15.2:0/4256737375 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec9c00f040 con 0x7feca41055f0 2023-11-16T16:48:22.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.387+0000 7fecac446700 1 -- 172.21.15.2:0/4256737375 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca4110370 con 0x7feca41055f0 2023-11-16T16:48:22.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.387+0000 7fec9b7fe700 1 -- 172.21.15.2:0/4256737375 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c00cd00 con 0x7feca41055f0 2023-11-16T16:48:22.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.387+0000 7fecac446700 1 -- 172.21.15.2:0/4256737375 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca41101e0 con 0x7feca41055f0 2023-11-16T16:48:22.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.388+0000 7fecac446700 1 -- 172.21.15.2:0/4256737375 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feca40fd1f0 con 0x7feca41055f0 2023-11-16T16:48:22.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.392+0000 7fec9b7fe700 1 -- 172.21.15.2:0/4256737375 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fec9c003a00 con 0x7feca41055f0 2023-11-16T16:48:22.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.393+0000 7fec9b7fe700 1 --2- 172.21.15.2:0/4256737375 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fec900a11e0 0x7fec900a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:22.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.393+0000 7fec9b7fe700 1 -- 172.21.15.2:0/4256737375 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fec9c0cd200 con 0x7feca41055f0 2023-11-16T16:48:22.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.394+0000 7fecaa1e2700 1 --2- 172.21.15.2:0/4256737375 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fec900a11e0 0x7fec900a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:22.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.394+0000 7fec9b7fe700 1 --2- 172.21.15.2:0/4256737375 >> [v2:172.21.15.155:6816/3808747539,v1:172.21.15.155:6817/3808747539] conn(0x7fec900ac1f0 0x7fec900ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:22.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.394+0000 7fec9b7fe700 1 -- 172.21.15.2:0/4256737375 --> [v2:172.21.15.155:6816/3808747539,v1:172.21.15.155:6817/3808747539] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec900aed50 con 0x7fec900ac1f0 2023-11-16T16:48:22.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.394+0000 7fec9b7fe700 1 -- 172.21.15.2:0/4256737375 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec9c0cd570 con 0x7feca41055f0 2023-11-16T16:48:22.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.394+0000 7feca99e1700 1 --2- 172.21.15.2:0/4256737375 >> [v2:172.21.15.155:6816/3808747539,v1:172.21.15.155:6817/3808747539] conn(0x7fec900ac1f0 0x7fec900ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:22.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.394+0000 7fecaa1e2700 1 --2- 172.21.15.2:0/4256737375 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fec900a11e0 0x7fec900a3690 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7feca0009c60 tx=0x7feca0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:22.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.395+0000 7feca99e1700 1 --2- 172.21.15.2:0/4256737375 >> [v2:172.21.15.155:6816/3808747539,v1:172.21.15.155:6817/3808747539] conn(0x7fec900ac1f0 0x7fec900ae680 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:22.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.395+0000 7fec9b7fe700 1 -- 172.21.15.2:0/4256737375 <== osd.8 v2:172.21.15.155:6816/3808747539 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fec900aed50 con 0x7fec900ac1f0 2023-11-16T16:48:22.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.412+0000 7fecac446700 1 -- 172.21.15.2:0/4256737375 --> [v2:172.21.15.155:6816/3808747539,v1:172.21.15.155:6817/3808747539] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feca4110500 con 0x7fec900ac1f0 2023-11-16T16:48:22.414 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.413+0000 7fec9b7fe700 1 -- 172.21.15.2:0/4256737375 <== osd.8 v2:172.21.15.155:6816/3808747539 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7feca4110500 con 0x7fec900ac1f0 2023-11-16T16:48:22.415 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.413+0000 7fecac446700 1 -- 172.21.15.2:0/4256737375 >> [v2:172.21.15.155:6816/3808747539,v1:172.21.15.155:6817/3808747539] conn(0x7fec900ac1f0 msgr2=0x7fec900ae680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:22.415 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.413+0000 7fecac446700 1 --2- 172.21.15.2:0/4256737375 >> [v2:172.21.15.155:6816/3808747539,v1:172.21.15.155:6817/3808747539] conn(0x7fec900ac1f0 0x7fec900ae680 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.415 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.413+0000 7fecac446700 1 -- 172.21.15.2:0/4256737375 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fec900a11e0 msgr2=0x7fec900a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:22.415 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.413+0000 7fecac446700 1 --2- 172.21.15.2:0/4256737375 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fec900a11e0 0x7fec900a3690 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7feca0009c60 tx=0x7feca0009260 comp rx=0 tx=0).stop 2023-11-16T16:48:22.415 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.413+0000 7fecac446700 1 -- 172.21.15.2:0/4256737375 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feca41055f0 msgr2=0x7feca419cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:22.415 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.414+0000 7fecac446700 1 --2- 172.21.15.2:0/4256737375 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7feca41055f0 0x7feca419cf80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fec9c00ef30 tx=0x7fec9c00c3a0 comp rx=0 tx=0).stop 2023-11-16T16:48:22.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.414+0000 7fecaa9e3700 1 -- 172.21.15.2:0/4256737375 reap_dead start 2023-11-16T16:48:22.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.414+0000 7fecac446700 1 -- 172.21.15.2:0/4256737375 shutdown_connections 2023-11-16T16:48:22.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.414+0000 7fecac446700 1 -- 172.21.15.2:0/4256737375 >> 172.21.15.2:0/4256737375 conn(0x7feca40fbdf0 msgr2=0x7feca40fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:22.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.414+0000 7fecac446700 1 -- 172.21.15.2:0/4256737375 shutdown_connections 2023-11-16T16:48:22.416 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:22.414+0000 7fecac446700 1 -- 172.21.15.2:0/4256737375 wait complete. 2023-11-16T16:48:22.642 INFO:teuthology.orchestra.run.smithi002.stdout:343597383703 2023-11-16T16:48:22.643 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.11 2023-11-16T16:48:23.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:22 smithi155 ceph-mon[131607]: pgmap v310: 129 pgs: 129 active+clean; 654 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 5 op/s 2023-11-16T16:48:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:22 smithi002 ceph-mon[128712]: pgmap v310: 129 pgs: 129 active+clean; 654 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 5 op/s 2023-11-16T16:48:23.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:22 smithi141 ceph-mon[131184]: pgmap v310: 129 pgs: 129 active+clean; 654 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 5 op/s 2023-11-16T16:48:23.735 INFO:teuthology.orchestra.run.smithi002.stdout:253403070502 2023-11-16T16:48:23.735 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.8 2023-11-16T16:48:24.268 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.266+0000 7fc95ce6b700 1 -- 172.21.15.2:0/2195759658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc958100ef0 msgr2=0x7fc958104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:24.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.266+0000 7fc95ce6b700 1 --2- 172.21.15.2:0/2195759658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc958100ef0 0x7fc958104ec0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc94c0099c0 tx=0x7fc94c009cd0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.267+0000 7fc95ce6b700 1 -- 172.21.15.2:0/2195759658 shutdown_connections 2023-11-16T16:48:24.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.267+0000 7fc95ce6b700 1 --2- 172.21.15.2:0/2195759658 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc9581055f0 0x7fc95810d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.267+0000 7fc95ce6b700 1 --2- 172.21.15.2:0/2195759658 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc958100550 0x7fc958100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.267+0000 7fc95ce6b700 1 --2- 172.21.15.2:0/2195759658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc958100ef0 0x7fc958104ec0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.267+0000 7fc95ce6b700 1 -- 172.21.15.2:0/2195759658 >> 172.21.15.2:0/2195759658 conn(0x7fc9580fbdf0 msgr2=0x7fc9580fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:24.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.267+0000 7fc95ce6b700 1 -- 172.21.15.2:0/2195759658 shutdown_connections 2023-11-16T16:48:24.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.267+0000 7fc95ce6b700 1 -- 172.21.15.2:0/2195759658 wait complete. 2023-11-16T16:48:24.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.268+0000 7fc95ce6b700 1 Processor -- start 2023-11-16T16:48:24.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.268+0000 7fc95ce6b700 1 -- start start 2023-11-16T16:48:24.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.268+0000 7fc95ce6b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc958100550 0x7fc9581985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:24.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.268+0000 7fc95ce6b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc958100ef0 0x7fc958198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:24.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.268+0000 7fc95ce6b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9581055f0 0x7fc95819cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:24.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.268+0000 7fc95ce6b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95819d5a0 con 0x7fc958100ef0 2023-11-16T16:48:24.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.268+0000 7fc95ce6b700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95819d710 con 0x7fc9581055f0 2023-11-16T16:48:24.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.268+0000 7fc95ce6b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95819d8d0 con 0x7fc958100550 2023-11-16T16:48:24.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.269+0000 7fc955d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc958100ef0 0x7fc958198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:24.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.269+0000 7fc95659c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc958100550 0x7fc9581985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:24.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.269+0000 7fc955d9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc958100ef0 0x7fc958198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48666/0 (socket says 172.21.15.2:48666) 2023-11-16T16:48:24.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.269+0000 7fc955d9b700 1 -- 172.21.15.2:0/47762050 learned_addr learned my addr 172.21.15.2:0/47762050 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:24.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.269+0000 7fc95659c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc958100550 0x7fc9581985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:33332/0 (socket says 172.21.15.2:33332) 2023-11-16T16:48:24.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.269+0000 7fc956d9d700 1 --2- 172.21.15.2:0/47762050 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9581055f0 0x7fc95819cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:24.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.269+0000 7fc955d9b700 1 -- 172.21.15.2:0/47762050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc958100550 msgr2=0x7fc9581985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:24.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.269+0000 7fc955d9b700 1 --2- 172.21.15.2:0/47762050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc958100550 0x7fc9581985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.269+0000 7fc955d9b700 1 -- 172.21.15.2:0/47762050 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9581055f0 msgr2=0x7fc95819cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:24.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.269+0000 7fc955d9b700 1 --2- 172.21.15.2:0/47762050 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc9581055f0 0x7fc95819cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.269+0000 7fc955d9b700 1 -- 172.21.15.2:0/47762050 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc950009c40 con 0x7fc958100ef0 2023-11-16T16:48:24.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.269+0000 7fc955d9b700 1 --2- 172.21.15.2:0/47762050 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc958100ef0 0x7fc958198b20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc94c0098e0 tx=0x7fc94c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:24.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.270+0000 7fc9477fe700 1 -- 172.21.15.2:0/47762050 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c01d070 con 0x7fc958100ef0 2023-11-16T16:48:24.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.270+0000 7fc9477fe700 1 -- 172.21.15.2:0/47762050 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc94c00fb80 con 0x7fc958100ef0 2023-11-16T16:48:24.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.270+0000 7fc95ce6b700 1 -- 172.21.15.2:0/47762050 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc94c0095f0 con 0x7fc958100ef0 2023-11-16T16:48:24.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.270+0000 7fc9477fe700 1 -- 172.21.15.2:0/47762050 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c0173f0 con 0x7fc958100ef0 2023-11-16T16:48:24.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.270+0000 7fc95ce6b700 1 -- 172.21.15.2:0/47762050 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc958110320 con 0x7fc958100ef0 2023-11-16T16:48:24.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.271+0000 7fc95ce6b700 1 -- 172.21.15.2:0/47762050 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc958074070 con 0x7fc958100ef0 2023-11-16T16:48:24.274 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.273+0000 7fc9477fe700 1 -- 172.21.15.2:0/47762050 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fc94c00fcf0 con 0x7fc958100ef0 2023-11-16T16:48:24.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.274+0000 7fc9477fe700 1 --2- 172.21.15.2:0/47762050 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc9400a11e0 0x7fc9400a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:24.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.274+0000 7fc9477fe700 1 -- 172.21.15.2:0/47762050 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc94c0cdb50 con 0x7fc958100ef0 2023-11-16T16:48:24.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.274+0000 7fc95659c700 1 --2- 172.21.15.2:0/47762050 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc9400a11e0 0x7fc9400a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:24.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.274+0000 7fc9477fe700 1 --2- 172.21.15.2:0/47762050 >> [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640] conn(0x7fc9400ac1f0 0x7fc9400ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:24.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.274+0000 7fc9477fe700 1 -- 172.21.15.2:0/47762050 --> [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9400aed50 con 0x7fc9400ac1f0 2023-11-16T16:48:24.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.274+0000 7fc9477fe700 1 -- 172.21.15.2:0/47762050 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc94c0cdec0 con 0x7fc958100ef0 2023-11-16T16:48:24.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.275+0000 7fc956d9d700 1 --2- 172.21.15.2:0/47762050 >> [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640] conn(0x7fc9400ac1f0 0x7fc9400ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:24.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.275+0000 7fc95659c700 1 --2- 172.21.15.2:0/47762050 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc9400a11e0 0x7fc9400a3690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc948009c60 tx=0x7fc948009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:24.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.275+0000 7fc956d9d700 1 --2- 172.21.15.2:0/47762050 >> [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640] conn(0x7fc9400ac1f0 0x7fc9400ae680 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:24.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.276+0000 7fc9477fe700 1 -- 172.21.15.2:0/47762050 <== osd.6 v2:172.21.15.2:6818/2881363640 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc9400aed50 con 0x7fc9400ac1f0 2023-11-16T16:48:24.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.293+0000 7fc95ce6b700 1 -- 172.21.15.2:0/47762050 --> [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc958110190 con 0x7fc9400ac1f0 2023-11-16T16:48:24.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.293+0000 7fc9477fe700 1 -- 172.21.15.2:0/47762050 <== osd.6 v2:172.21.15.2:6818/2881363640 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc958110190 con 0x7fc9400ac1f0 2023-11-16T16:48:24.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.293+0000 7fc95ce6b700 1 -- 172.21.15.2:0/47762050 >> [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640] conn(0x7fc9400ac1f0 msgr2=0x7fc9400ae680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:24.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.293+0000 7fc95ce6b700 1 --2- 172.21.15.2:0/47762050 >> [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640] conn(0x7fc9400ac1f0 0x7fc9400ae680 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.294+0000 7fc95ce6b700 1 -- 172.21.15.2:0/47762050 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc9400a11e0 msgr2=0x7fc9400a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:24.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.294+0000 7fc95ce6b700 1 --2- 172.21.15.2:0/47762050 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc9400a11e0 0x7fc9400a3690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc948009c60 tx=0x7fc948009260 comp rx=0 tx=0).stop 2023-11-16T16:48:24.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.294+0000 7fc95ce6b700 1 -- 172.21.15.2:0/47762050 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc958100ef0 msgr2=0x7fc958198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:24.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.294+0000 7fc95ce6b700 1 --2- 172.21.15.2:0/47762050 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc958100ef0 0x7fc958198b20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc94c0098e0 tx=0x7fc94c00f690 comp rx=0 tx=0).stop 2023-11-16T16:48:24.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.294+0000 7fc956d9d700 1 -- 172.21.15.2:0/47762050 reap_dead start 2023-11-16T16:48:24.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.294+0000 7fc95ce6b700 1 -- 172.21.15.2:0/47762050 shutdown_connections 2023-11-16T16:48:24.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.294+0000 7fc95ce6b700 1 -- 172.21.15.2:0/47762050 >> 172.21.15.2:0/47762050 conn(0x7fc9580fbdf0 msgr2=0x7fc9580fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:24.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.294+0000 7fc95ce6b700 1 -- 172.21.15.2:0/47762050 shutdown_connections 2023-11-16T16:48:24.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.294+0000 7fc95ce6b700 1 -- 172.21.15.2:0/47762050 wait complete. 2023-11-16T16:48:24.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.990+0000 7fcf2c4e4700 1 -- 172.21.15.2:0/2119937662 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf24100ee0 msgr2=0x7fcf24104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:24.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.990+0000 7fcf2c4e4700 1 --2- 172.21.15.2:0/2119937662 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf24100ee0 0x7fcf24104eb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fcf140099c0 tx=0x7fcf14009cd0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.990+0000 7fcf2c4e4700 1 -- 172.21.15.2:0/2119937662 shutdown_connections 2023-11-16T16:48:24.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.990+0000 7fcf2c4e4700 1 --2- 172.21.15.2:0/2119937662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf24100540 0x7fcf24100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.990+0000 7fcf2c4e4700 1 --2- 172.21.15.2:0/2119937662 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf241055e0 0x7fcf2410d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.990+0000 7fcf2c4e4700 1 --2- 172.21.15.2:0/2119937662 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf24100ee0 0x7fcf24104eb0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.990+0000 7fcf2c4e4700 1 -- 172.21.15.2:0/2119937662 >> 172.21.15.2:0/2119937662 conn(0x7fcf240fbdc0 msgr2=0x7fcf240fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:24.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.990+0000 7fcf2c4e4700 1 -- 172.21.15.2:0/2119937662 shutdown_connections 2023-11-16T16:48:24.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.990+0000 7fcf2c4e4700 1 -- 172.21.15.2:0/2119937662 wait complete. 2023-11-16T16:48:24.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.991+0000 7fcf2c4e4700 1 Processor -- start 2023-11-16T16:48:24.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.991+0000 7fcf2c4e4700 1 -- start start 2023-11-16T16:48:24.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.992+0000 7fcf2c4e4700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf24100540 0x7fcf241985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:24.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.992+0000 7fcf2c4e4700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf24100ee0 0x7fcf24198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:24.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.992+0000 7fcf2c4e4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf241055e0 0x7fcf2419cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:24.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.992+0000 7fcf2c4e4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2419d580 con 0x7fcf241055e0 2023-11-16T16:48:24.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.992+0000 7fcf2c4e4700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2419d6f0 con 0x7fcf24100540 2023-11-16T16:48:24.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.992+0000 7fcf2c4e4700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2419d8b0 con 0x7fcf24100ee0 2023-11-16T16:48:24.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.992+0000 7fcf2aa81700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf241055e0 0x7fcf2419cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:24.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.992+0000 7fcf2aa81700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf241055e0 0x7fcf2419cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40732/0 (socket says 172.21.15.2:40732) 2023-11-16T16:48:24.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.992+0000 7fcf2aa81700 1 -- 172.21.15.2:0/1230169318 learned_addr learned my addr 172.21.15.2:0/1230169318 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:24.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.992+0000 7fcf2a280700 1 --2- 172.21.15.2:0/1230169318 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf24100540 0x7fcf241985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:24.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.992+0000 7fcf29a7f700 1 --2- 172.21.15.2:0/1230169318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf24100ee0 0x7fcf24198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:24.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.992+0000 7fcf2aa81700 1 -- 172.21.15.2:0/1230169318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf24100ee0 msgr2=0x7fcf24198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:24.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.992+0000 7fcf2aa81700 1 --2- 172.21.15.2:0/1230169318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf24100ee0 0x7fcf24198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.992+0000 7fcf2aa81700 1 -- 172.21.15.2:0/1230169318 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf24100540 msgr2=0x7fcf241985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:24.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.992+0000 7fcf2aa81700 1 --2- 172.21.15.2:0/1230169318 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf24100540 0x7fcf241985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.992+0000 7fcf2aa81700 1 -- 172.21.15.2:0/1230169318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf140095f0 con 0x7fcf241055e0 2023-11-16T16:48:24.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.993+0000 7fcf2aa81700 1 --2- 172.21.15.2:0/1230169318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf241055e0 0x7fcf2419cf70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c00ef30 tx=0x7fcf1c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:24.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.993+0000 7fcf1b7fe700 1 -- 172.21.15.2:0/1230169318 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c018070 con 0x7fcf241055e0 2023-11-16T16:48:24.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.993+0000 7fcf1b7fe700 1 -- 172.21.15.2:0/1230169318 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf1c00f040 con 0x7fcf241055e0 2023-11-16T16:48:24.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.993+0000 7fcf1b7fe700 1 -- 172.21.15.2:0/1230169318 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c00cd00 con 0x7fcf241055e0 2023-11-16T16:48:24.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.993+0000 7fcf2c4e4700 1 -- 172.21.15.2:0/1230169318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf24110360 con 0x7fcf241055e0 2023-11-16T16:48:24.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.993+0000 7fcf2c4e4700 1 -- 172.21.15.2:0/1230169318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf241101d0 con 0x7fcf241055e0 2023-11-16T16:48:24.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.994+0000 7fcf197fa700 1 -- 172.21.15.2:0/1230169318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcf24061270 con 0x7fcf241055e0 2023-11-16T16:48:24.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.996+0000 7fcf1b7fe700 1 -- 172.21.15.2:0/1230169318 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fcf1c003a00 con 0x7fcf241055e0 2023-11-16T16:48:24.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.998+0000 7fcf1b7fe700 1 --2- 172.21.15.2:0/1230169318 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fcf100a1260 0x7fcf100a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:24.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.998+0000 7fcf1b7fe700 1 -- 172.21.15.2:0/1230169318 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcf1c0cd4f0 con 0x7fcf241055e0 2023-11-16T16:48:24.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.998+0000 7fcf2a280700 1 --2- 172.21.15.2:0/1230169318 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fcf100a1260 0x7fcf100a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:24.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.998+0000 7fcf1b7fe700 1 --2- 172.21.15.2:0/1230169318 >> [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823] conn(0x7fcf100ac270 0x7fcf100ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:24.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.998+0000 7fcf1b7fe700 1 -- 172.21.15.2:0/1230169318 --> [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf100aedd0 con 0x7fcf100ac270 2023-11-16T16:48:24.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.998+0000 7fcf1b7fe700 1 -- 172.21.15.2:0/1230169318 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf1c0cd860 con 0x7fcf241055e0 2023-11-16T16:48:24.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.998+0000 7fcf29a7f700 1 --2- 172.21.15.2:0/1230169318 >> [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823] conn(0x7fcf100ac270 0x7fcf100ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:24.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.999+0000 7fcf2a280700 1 --2- 172.21.15.2:0/1230169318 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fcf100a1260 0x7fcf100a3710 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcf20009c60 tx=0x7fcf20009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:24.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:23.999+0000 7fcf29a7f700 1 --2- 172.21.15.2:0/1230169318 >> [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823] conn(0x7fcf100ac270 0x7fcf100ae700 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:24.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.000+0000 7fcf1b7fe700 1 -- 172.21.15.2:0/1230169318 <== osd.9 v2:172.21.15.2:6826/3506759823 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fcf100aedd0 con 0x7fcf100ac270 2023-11-16T16:48:24.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.016+0000 7fcf197fa700 1 -- 172.21.15.2:0/1230169318 --> [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcf241104f0 con 0x7fcf100ac270 2023-11-16T16:48:24.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.017+0000 7fcf1b7fe700 1 -- 172.21.15.2:0/1230169318 <== osd.9 v2:172.21.15.2:6826/3506759823 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcf241104f0 con 0x7fcf100ac270 2023-11-16T16:48:24.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.017+0000 7fcf197fa700 1 -- 172.21.15.2:0/1230169318 >> [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823] conn(0x7fcf100ac270 msgr2=0x7fcf100ae700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:24.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.017+0000 7fcf197fa700 1 --2- 172.21.15.2:0/1230169318 >> [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823] conn(0x7fcf100ac270 0x7fcf100ae700 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.017+0000 7fcf197fa700 1 -- 172.21.15.2:0/1230169318 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fcf100a1260 msgr2=0x7fcf100a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:24.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.017+0000 7fcf197fa700 1 --2- 172.21.15.2:0/1230169318 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fcf100a1260 0x7fcf100a3710 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcf20009c60 tx=0x7fcf20009260 comp rx=0 tx=0).stop 2023-11-16T16:48:24.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.017+0000 7fcf197fa700 1 -- 172.21.15.2:0/1230169318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf241055e0 msgr2=0x7fcf2419cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:24.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.017+0000 7fcf197fa700 1 --2- 172.21.15.2:0/1230169318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf241055e0 0x7fcf2419cf70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c00ef30 tx=0x7fcf1c00c3a0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.017+0000 7fcf2aa81700 1 -- 172.21.15.2:0/1230169318 reap_dead start 2023-11-16T16:48:24.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.018+0000 7fcf197fa700 1 -- 172.21.15.2:0/1230169318 shutdown_connections 2023-11-16T16:48:24.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.018+0000 7fcf197fa700 1 -- 172.21.15.2:0/1230169318 >> 172.21.15.2:0/1230169318 conn(0x7fcf240fbdc0 msgr2=0x7fcf2410b760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:24.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.018+0000 7fcf197fa700 1 -- 172.21.15.2:0/1230169318 shutdown_connections 2023-11-16T16:48:24.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.018+0000 7fcf197fa700 1 -- 172.21.15.2:0/1230169318 wait complete. 2023-11-16T16:48:24.407 INFO:teuthology.orchestra.run.smithi002.stdout:77309411394 2023-11-16T16:48:24.407 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.2 2023-11-16T16:48:24.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.470+0000 7f0e6411c700 1 -- 172.21.15.2:0/2379737842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e5c100540 msgr2=0x7f0e5c100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:24.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.470+0000 7f0e6411c700 1 --2- 172.21.15.2:0/2379737842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e5c100540 0x7f0e5c100910 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0e580099c0 tx=0x7f0e58009cd0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.471+0000 7f0e6411c700 1 -- 172.21.15.2:0/2379737842 shutdown_connections 2023-11-16T16:48:24.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.471+0000 7f0e6411c700 1 --2- 172.21.15.2:0/2379737842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e5c1055e0 0x7f0e5c10d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.471+0000 7f0e6411c700 1 --2- 172.21.15.2:0/2379737842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e5c100540 0x7f0e5c100910 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.557 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.471+0000 7f0e6411c700 1 --2- 172.21.15.2:0/2379737842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e5c100ee0 0x7f0e5c104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.471+0000 7f0e6411c700 1 -- 172.21.15.2:0/2379737842 >> 172.21.15.2:0/2379737842 conn(0x7f0e5c0fbdc0 msgr2=0x7f0e5c0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:24.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.471+0000 7f0e6411c700 1 -- 172.21.15.2:0/2379737842 shutdown_connections 2023-11-16T16:48:24.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.471+0000 7f0e6411c700 1 -- 172.21.15.2:0/2379737842 wait complete. 2023-11-16T16:48:24.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.472+0000 7f0e6411c700 1 Processor -- start 2023-11-16T16:48:24.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.472+0000 7f0e6411c700 1 -- start start 2023-11-16T16:48:24.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.472+0000 7f0e6411c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e5c100540 0x7f0e5c1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:24.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.472+0000 7f0e6411c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e5c100ee0 0x7f0e5c198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:24.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.472+0000 7f0e6411c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e5c1055e0 0x7f0e5c19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:24.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.472+0000 7f0e6411c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c19d5b0 con 0x7f0e5c1055e0 2023-11-16T16:48:24.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.472+0000 7f0e6411c700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c19d720 con 0x7f0e5c100ee0 2023-11-16T16:48:24.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.472+0000 7f0e6411c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c19d8e0 con 0x7f0e5c100540 2023-11-16T16:48:24.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.473+0000 7f0e626b9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e5c1055e0 0x7f0e5c19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:24.558 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.473+0000 7f0e61eb8700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e5c100540 0x7f0e5c1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:24.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.473+0000 7f0e626b9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e5c1055e0 0x7f0e5c19cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48692/0 (socket says 172.21.15.2:48692) 2023-11-16T16:48:24.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.473+0000 7f0e626b9700 1 -- 172.21.15.2:0/570475953 learned_addr learned my addr 172.21.15.2:0/570475953 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:24.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.473+0000 7f0e616b7700 1 --2- 172.21.15.2:0/570475953 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e5c100ee0 0x7f0e5c198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:24.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.473+0000 7f0e626b9700 1 -- 172.21.15.2:0/570475953 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e5c100540 msgr2=0x7f0e5c1985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:24.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.473+0000 7f0e626b9700 1 --2- 172.21.15.2:0/570475953 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e5c100540 0x7f0e5c1985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.473+0000 7f0e626b9700 1 -- 172.21.15.2:0/570475953 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e5c100ee0 msgr2=0x7f0e5c198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:24.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.473+0000 7f0e626b9700 1 --2- 172.21.15.2:0/570475953 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e5c100ee0 0x7f0e5c198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.473+0000 7f0e626b9700 1 -- 172.21.15.2:0/570475953 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e580095f0 con 0x7f0e5c1055e0 2023-11-16T16:48:24.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.473+0000 7f0e61eb8700 1 --2- 172.21.15.2:0/570475953 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e5c100540 0x7f0e5c1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:24.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.473+0000 7f0e626b9700 1 --2- 172.21.15.2:0/570475953 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e5c1055e0 0x7f0e5c19cfa0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0e5400ef30 tx=0x7f0e5400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:24.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.474+0000 7f0e52ffd700 1 -- 172.21.15.2:0/570475953 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54018070 con 0x7f0e5c1055e0 2023-11-16T16:48:24.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.474+0000 7f0e52ffd700 1 -- 172.21.15.2:0/570475953 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e5400f040 con 0x7f0e5c1055e0 2023-11-16T16:48:24.559 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.474+0000 7f0e52ffd700 1 -- 172.21.15.2:0/570475953 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5400cd00 con 0x7f0e5c1055e0 2023-11-16T16:48:24.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.474+0000 7f0e6411c700 1 -- 172.21.15.2:0/570475953 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e5c110320 con 0x7f0e5c1055e0 2023-11-16T16:48:24.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.474+0000 7f0e6411c700 1 -- 172.21.15.2:0/570475953 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5c110190 con 0x7f0e5c1055e0 2023-11-16T16:48:24.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.475+0000 7f0e6411c700 1 -- 172.21.15.2:0/570475953 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0e5c103860 con 0x7f0e5c1055e0 2023-11-16T16:48:24.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.477+0000 7f0e52ffd700 1 -- 172.21.15.2:0/570475953 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f0e54003a00 con 0x7f0e5c1055e0 2023-11-16T16:48:24.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.479+0000 7f0e52ffd700 1 --2- 172.21.15.2:0/570475953 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0e480a1260 0x7f0e480a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:24.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.479+0000 7f0e52ffd700 1 -- 172.21.15.2:0/570475953 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0e540cc800 con 0x7f0e5c1055e0 2023-11-16T16:48:24.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.479+0000 7f0e52ffd700 1 --2- 172.21.15.2:0/570475953 >> [v2:172.21.15.155:6808/2988032938,v1:172.21.15.155:6809/2988032938] conn(0x7f0e480ac1f0 0x7f0e480ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:24.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.480+0000 7f0e52ffd700 1 -- 172.21.15.2:0/570475953 --> [v2:172.21.15.155:6808/2988032938,v1:172.21.15.155:6809/2988032938] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e480aed50 con 0x7f0e480ac1f0 2023-11-16T16:48:24.560 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.480+0000 7f0e61eb8700 1 --2- 172.21.15.2:0/570475953 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0e480a1260 0x7f0e480a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:24.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.480+0000 7f0e616b7700 1 --2- 172.21.15.2:0/570475953 >> [v2:172.21.15.155:6808/2988032938,v1:172.21.15.155:6809/2988032938] conn(0x7f0e480ac1f0 0x7f0e480ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:24.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.480+0000 7f0e52ffd700 1 -- 172.21.15.2:0/570475953 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e54095b70 con 0x7f0e5c1055e0 2023-11-16T16:48:24.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.481+0000 7f0e616b7700 1 --2- 172.21.15.2:0/570475953 >> [v2:172.21.15.155:6808/2988032938,v1:172.21.15.155:6809/2988032938] conn(0x7f0e480ac1f0 0x7f0e480ae680 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:24.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.481+0000 7f0e61eb8700 1 --2- 172.21.15.2:0/570475953 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0e480a1260 0x7f0e480a3710 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0e58009990 tx=0x7f0e5801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:24.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.481+0000 7f0e52ffd700 1 -- 172.21.15.2:0/570475953 <== osd.5 v2:172.21.15.155:6808/2988032938 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f0e480aed50 con 0x7f0e480ac1f0 2023-11-16T16:48:24.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.502+0000 7f0e6411c700 1 -- 172.21.15.2:0/570475953 --> [v2:172.21.15.155:6808/2988032938,v1:172.21.15.155:6809/2988032938] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0e5c1104b0 con 0x7f0e480ac1f0 2023-11-16T16:48:24.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.503+0000 7f0e52ffd700 1 -- 172.21.15.2:0/570475953 <== osd.5 v2:172.21.15.155:6808/2988032938 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0e5c1104b0 con 0x7f0e480ac1f0 2023-11-16T16:48:24.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.503+0000 7f0e6411c700 1 -- 172.21.15.2:0/570475953 >> [v2:172.21.15.155:6808/2988032938,v1:172.21.15.155:6809/2988032938] conn(0x7f0e480ac1f0 msgr2=0x7f0e480ae680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:24.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.503+0000 7f0e6411c700 1 --2- 172.21.15.2:0/570475953 >> [v2:172.21.15.155:6808/2988032938,v1:172.21.15.155:6809/2988032938] conn(0x7f0e480ac1f0 0x7f0e480ae680 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.504+0000 7f0e6411c700 1 -- 172.21.15.2:0/570475953 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0e480a1260 msgr2=0x7f0e480a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:24.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.504+0000 7f0e6411c700 1 --2- 172.21.15.2:0/570475953 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0e480a1260 0x7f0e480a3710 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0e58009990 tx=0x7f0e5801a040 comp rx=0 tx=0).stop 2023-11-16T16:48:24.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.504+0000 7f0e6411c700 1 -- 172.21.15.2:0/570475953 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e5c1055e0 msgr2=0x7f0e5c19cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:24.561 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.504+0000 7f0e6411c700 1 --2- 172.21.15.2:0/570475953 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e5c1055e0 0x7f0e5c19cfa0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0e5400ef30 tx=0x7f0e5400c3a0 comp rx=0 tx=0).stop 2023-11-16T16:48:24.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.504+0000 7f0e626b9700 1 -- 172.21.15.2:0/570475953 reap_dead start 2023-11-16T16:48:24.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.504+0000 7f0e6411c700 1 -- 172.21.15.2:0/570475953 shutdown_connections 2023-11-16T16:48:24.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.504+0000 7f0e6411c700 1 -- 172.21.15.2:0/570475953 >> 172.21.15.2:0/570475953 conn(0x7f0e5c0fbdc0 msgr2=0x7f0e5c101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:24.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.504+0000 7f0e6411c700 1 -- 172.21.15.2:0/570475953 shutdown_connections 2023-11-16T16:48:24.562 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:24.504+0000 7f0e6411c700 1 -- 172.21.15.2:0/570475953 wait complete. 2023-11-16T16:48:24.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:24 smithi002 ceph-mon[128712]: pgmap v311: 129 pgs: 129 active+clean; 654 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-11-16T16:48:25.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:24 smithi155 ceph-mon[131607]: pgmap v311: 129 pgs: 129 active+clean; 654 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-11-16T16:48:25.174 INFO:teuthology.orchestra.run.smithi002.stdout:197568495663 2023-11-16T16:48:25.175 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.6 2023-11-16T16:48:25.246 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.006+0000 7f1088532700 1 -- 172.21.15.2:0/1950687345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1080101540 msgr2=0x7f1080101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:25.246 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.006+0000 7f1088532700 1 --2- 172.21.15.2:0/1950687345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1080101540 0x7f1080101910 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f107c009990 tx=0x7f107c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:25.246 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.006+0000 7f1088532700 1 -- 172.21.15.2:0/1950687345 shutdown_connections 2023-11-16T16:48:25.246 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.006+0000 7f1088532700 1 --2- 172.21.15.2:0/1950687345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1080101540 0x7f1080101910 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:25.246 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.006+0000 7f1088532700 1 --2- 172.21.15.2:0/1950687345 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f108010a890 0x7f108010ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:25.246 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.006+0000 7f1088532700 1 --2- 172.21.15.2:0/1950687345 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1080101e50 0x7f108010a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:25.246 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.006+0000 7f1088532700 1 -- 172.21.15.2:0/1950687345 >> 172.21.15.2:0/1950687345 conn(0x7f10800770d0 msgr2=0x7f10800774d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:25.246 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.006+0000 7f1088532700 1 -- 172.21.15.2:0/1950687345 shutdown_connections 2023-11-16T16:48:25.246 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.006+0000 7f1088532700 1 -- 172.21.15.2:0/1950687345 wait complete. 2023-11-16T16:48:25.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.007+0000 7f1088532700 1 Processor -- start 2023-11-16T16:48:25.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.007+0000 7f1088532700 1 -- start start 2023-11-16T16:48:25.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.008+0000 7f1088532700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1080101540 0x7f1080196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:25.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.008+0000 7f1088532700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1080101e50 0x7f1080196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:25.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.008+0000 7f1088532700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f108010a890 0x7f108019adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:25.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.008+0000 7f1088532700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108019b3d0 con 0x7f1080101540 2023-11-16T16:48:25.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.008+0000 7f1088532700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108019b540 con 0x7f108010a890 2023-11-16T16:48:25.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.008+0000 7f1088532700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108019b700 con 0x7f1080101e50 2023-11-16T16:48:25.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.008+0000 7f10862ce700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1080101540 0x7f1080196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:25.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.008+0000 7f10862ce700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1080101540 0x7f1080196410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48712/0 (socket says 172.21.15.2:48712) 2023-11-16T16:48:25.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.008+0000 7f10862ce700 1 -- 172.21.15.2:0/2407669310 learned_addr learned my addr 172.21.15.2:0/2407669310 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:25.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.008+0000 7f1085acd700 1 --2- 172.21.15.2:0/2407669310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1080101e50 0x7f1080196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:25.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.008+0000 7f1086acf700 1 --2- 172.21.15.2:0/2407669310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f108010a890 0x7f108019adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:25.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.008+0000 7f10862ce700 1 -- 172.21.15.2:0/2407669310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1080101e50 msgr2=0x7f1080196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:25.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.008+0000 7f10862ce700 1 --2- 172.21.15.2:0/2407669310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1080101e50 0x7f1080196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:25.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.008+0000 7f10862ce700 1 -- 172.21.15.2:0/2407669310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f108010a890 msgr2=0x7f108019adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:25.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.008+0000 7f10862ce700 1 --2- 172.21.15.2:0/2407669310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f108010a890 0x7f108019adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:25.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.008+0000 7f10862ce700 1 -- 172.21.15.2:0/2407669310 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c0095f0 con 0x7f1080101540 2023-11-16T16:48:25.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.009+0000 7f10862ce700 1 --2- 172.21.15.2:0/2407669310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1080101540 0x7f1080196410 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f107c000c00 tx=0x7f107c004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:25.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.009+0000 7f10777fe700 1 -- 172.21.15.2:0/2407669310 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107c01d070 con 0x7f1080101540 2023-11-16T16:48:25.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.009+0000 7f10777fe700 1 -- 172.21.15.2:0/2407669310 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f107c021410 con 0x7f1080101540 2023-11-16T16:48:25.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.009+0000 7f1088532700 1 -- 172.21.15.2:0/2407669310 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1080110320 con 0x7f1080101540 2023-11-16T16:48:25.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.009+0000 7f10777fe700 1 -- 172.21.15.2:0/2407669310 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107c017650 con 0x7f1080101540 2023-11-16T16:48:25.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.009+0000 7f1088532700 1 -- 172.21.15.2:0/2407669310 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1080110190 con 0x7f1080101540 2023-11-16T16:48:25.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.010+0000 7f10757fa700 1 -- 172.21.15.2:0/2407669310 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1068000fc0 con 0x7f1080101540 2023-11-16T16:48:25.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.012+0000 7f10777fe700 1 -- 172.21.15.2:0/2407669310 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f107c021580 con 0x7f1080101540 2023-11-16T16:48:25.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.013+0000 7f10777fe700 1 --2- 172.21.15.2:0/2407669310 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f106c0a11e0 0x7f106c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:25.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.013+0000 7f10777fe700 1 -- 172.21.15.2:0/2407669310 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f107c0ce730 con 0x7f1080101540 2023-11-16T16:48:25.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.014+0000 7f1085acd700 1 --2- 172.21.15.2:0/2407669310 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f106c0a11e0 0x7f106c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:25.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.014+0000 7f10777fe700 1 --2- 172.21.15.2:0/2407669310 >> [v2:172.21.15.141:6816/372138580,v1:172.21.15.141:6817/372138580] conn(0x7f106c0ac1f0 0x7f106c0ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:25.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.014+0000 7f10777fe700 1 -- 172.21.15.2:0/2407669310 --> [v2:172.21.15.141:6816/372138580,v1:172.21.15.141:6817/372138580] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f106c0aed50 con 0x7f106c0ac1f0 2023-11-16T16:48:25.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.014+0000 7f10777fe700 1 -- 172.21.15.2:0/2407669310 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f107c0ceaa0 con 0x7f1080101540 2023-11-16T16:48:25.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.014+0000 7f1085acd700 1 --2- 172.21.15.2:0/2407669310 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f106c0a11e0 0x7f106c0a3690 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1080197a30 tx=0x7f1070009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:25.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.014+0000 7f1086acf700 1 --2- 172.21.15.2:0/2407669310 >> [v2:172.21.15.141:6816/372138580,v1:172.21.15.141:6817/372138580] conn(0x7f106c0ac1f0 0x7f106c0ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:25.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.015+0000 7f1086acf700 1 --2- 172.21.15.2:0/2407669310 >> [v2:172.21.15.141:6816/372138580,v1:172.21.15.141:6817/372138580] conn(0x7f106c0ac1f0 0x7f106c0ae680 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:25.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.016+0000 7f10777fe700 1 -- 172.21.15.2:0/2407669310 <== osd.7 v2:172.21.15.141:6816/372138580 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f106c0aed50 con 0x7f106c0ac1f0 2023-11-16T16:48:25.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.033+0000 7f10757fa700 1 -- 172.21.15.2:0/2407669310 --> [v2:172.21.15.141:6816/372138580,v1:172.21.15.141:6817/372138580] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1068002d40 con 0x7f106c0ac1f0 2023-11-16T16:48:25.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.033+0000 7f10777fe700 1 -- 172.21.15.2:0/2407669310 <== osd.7 v2:172.21.15.141:6816/372138580 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1068002d40 con 0x7f106c0ac1f0 2023-11-16T16:48:25.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.034+0000 7f10757fa700 1 -- 172.21.15.2:0/2407669310 >> [v2:172.21.15.141:6816/372138580,v1:172.21.15.141:6817/372138580] conn(0x7f106c0ac1f0 msgr2=0x7f106c0ae680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:25.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.034+0000 7f10757fa700 1 --2- 172.21.15.2:0/2407669310 >> [v2:172.21.15.141:6816/372138580,v1:172.21.15.141:6817/372138580] conn(0x7f106c0ac1f0 0x7f106c0ae680 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:25.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.034+0000 7f10757fa700 1 -- 172.21.15.2:0/2407669310 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f106c0a11e0 msgr2=0x7f106c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:25.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.034+0000 7f10757fa700 1 --2- 172.21.15.2:0/2407669310 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f106c0a11e0 0x7f106c0a3690 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1080197a30 tx=0x7f1070009260 comp rx=0 tx=0).stop 2023-11-16T16:48:25.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.034+0000 7f10757fa700 1 -- 172.21.15.2:0/2407669310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1080101540 msgr2=0x7f1080196410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:25.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.034+0000 7f10757fa700 1 --2- 172.21.15.2:0/2407669310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1080101540 0x7f1080196410 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f107c000c00 tx=0x7f107c004790 comp rx=0 tx=0).stop 2023-11-16T16:48:25.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.034+0000 7f1086acf700 1 -- 172.21.15.2:0/2407669310 reap_dead start 2023-11-16T16:48:25.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.034+0000 7f10757fa700 1 -- 172.21.15.2:0/2407669310 shutdown_connections 2023-11-16T16:48:25.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.034+0000 7f10757fa700 1 -- 172.21.15.2:0/2407669310 >> 172.21.15.2:0/2407669310 conn(0x7f10800770d0 msgr2=0x7f10800fce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:25.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.034+0000 7f10757fa700 1 -- 172.21.15.2:0/2407669310 shutdown_connections 2023-11-16T16:48:25.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:25.034+0000 7f10757fa700 1 -- 172.21.15.2:0/2407669310 wait complete. 2023-11-16T16:48:25.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:24 smithi141 ceph-mon[131184]: pgmap v311: 129 pgs: 129 active+clean; 654 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-11-16T16:48:25.815 INFO:teuthology.orchestra.run.smithi002.stdout:163208757301 2023-11-16T16:48:25.815 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.5 2023-11-16T16:48:25.875 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:25.941 INFO:teuthology.orchestra.run.smithi002.stdout:287762808866 2023-11-16T16:48:25.942 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.9 2023-11-16T16:48:26.444 INFO:teuthology.orchestra.run.smithi002.stdout:219043332138 2023-11-16T16:48:26.444 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.7 2023-11-16T16:48:27.020 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:27.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:26 smithi155 ceph-mon[131607]: pgmap v312: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2023-11-16T16:48:27.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.139+0000 7fb9f57f5700 1 -- 172.21.15.2:0/1683864979 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb9f0100540 msgr2=0x7fb9f0100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:27.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.139+0000 7fb9f57f5700 1 --2- 172.21.15.2:0/1683864979 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb9f0100540 0x7fb9f0100910 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb9e00099c0 tx=0x7fb9e0009cd0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.139+0000 7fb9f57f5700 1 -- 172.21.15.2:0/1683864979 shutdown_connections 2023-11-16T16:48:27.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.139+0000 7fb9f57f5700 1 --2- 172.21.15.2:0/1683864979 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9f01055e0 0x7fb9f010d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.139+0000 7fb9f57f5700 1 --2- 172.21.15.2:0/1683864979 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb9f0100540 0x7fb9f0100910 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.139+0000 7fb9f57f5700 1 --2- 172.21.15.2:0/1683864979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9f0100ee0 0x7fb9f0104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.139+0000 7fb9f57f5700 1 -- 172.21.15.2:0/1683864979 >> 172.21.15.2:0/1683864979 conn(0x7fb9f00fbdc0 msgr2=0x7fb9f00fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:27.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.140+0000 7fb9f57f5700 1 -- 172.21.15.2:0/1683864979 shutdown_connections 2023-11-16T16:48:27.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.140+0000 7fb9f57f5700 1 -- 172.21.15.2:0/1683864979 wait complete. 2023-11-16T16:48:27.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.141+0000 7fb9f57f5700 1 Processor -- start 2023-11-16T16:48:27.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.141+0000 7fb9f57f5700 1 -- start start 2023-11-16T16:48:27.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.141+0000 7fb9f57f5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb9f0100540 0x7fb9f0198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:27.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.141+0000 7fb9f57f5700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9f0100ee0 0x7fb9f0198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:27.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.141+0000 7fb9f57f5700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9f01055e0 0x7fb9f019cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:27.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.141+0000 7fb9f57f5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f019d520 con 0x7fb9f0100540 2023-11-16T16:48:27.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.141+0000 7fb9f57f5700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f019d690 con 0x7fb9f01055e0 2023-11-16T16:48:27.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.141+0000 7fb9f57f5700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f019d850 con 0x7fb9f0100ee0 2023-11-16T16:48:27.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.142+0000 7fb9eeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb9f0100540 0x7fb9f0198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:27.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.142+0000 7fb9eeffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb9f0100540 0x7fb9f0198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48744/0 (socket says 172.21.15.2:48744) 2023-11-16T16:48:27.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.142+0000 7fb9eeffd700 1 -- 172.21.15.2:0/2555033012 learned_addr learned my addr 172.21.15.2:0/2555033012 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:27.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.142+0000 7fb9ee7fc700 1 --2- 172.21.15.2:0/2555033012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9f0100ee0 0x7fb9f0198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:27.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.142+0000 7fb9ef7fe700 1 --2- 172.21.15.2:0/2555033012 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9f01055e0 0x7fb9f019cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:27.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.142+0000 7fb9eeffd700 1 -- 172.21.15.2:0/2555033012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9f0100ee0 msgr2=0x7fb9f0198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:27.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.142+0000 7fb9eeffd700 1 --2- 172.21.15.2:0/2555033012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9f0100ee0 0x7fb9f0198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.142+0000 7fb9eeffd700 1 -- 172.21.15.2:0/2555033012 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9f01055e0 msgr2=0x7fb9f019cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:27.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.142+0000 7fb9eeffd700 1 --2- 172.21.15.2:0/2555033012 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9f01055e0 0x7fb9f019cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.142+0000 7fb9eeffd700 1 -- 172.21.15.2:0/2555033012 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e00095f0 con 0x7fb9f0100540 2023-11-16T16:48:27.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.142+0000 7fb9ef7fe700 1 --2- 172.21.15.2:0/2555033012 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9f01055e0 0x7fb9f019cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:48:27.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.143+0000 7fb9eeffd700 1 --2- 172.21.15.2:0/2555033012 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb9f0100540 0x7fb9f0198560 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb9e0000c00 tx=0x7fb9e000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:27.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.143+0000 7fb9d7fff700 1 -- 172.21.15.2:0/2555033012 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e001d070 con 0x7fb9f0100540 2023-11-16T16:48:27.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.143+0000 7fb9d7fff700 1 -- 172.21.15.2:0/2555033012 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb9e0005050 con 0x7fb9f0100540 2023-11-16T16:48:27.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.143+0000 7fb9d7fff700 1 -- 172.21.15.2:0/2555033012 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e0004b70 con 0x7fb9f0100540 2023-11-16T16:48:27.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.143+0000 7fb9f57f5700 1 -- 172.21.15.2:0/2555033012 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f0110320 con 0x7fb9f0100540 2023-11-16T16:48:27.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.143+0000 7fb9f57f5700 1 -- 172.21.15.2:0/2555033012 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f0110190 con 0x7fb9f0100540 2023-11-16T16:48:27.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.144+0000 7fb9f57f5700 1 -- 172.21.15.2:0/2555033012 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb9f00fd380 con 0x7fb9f0100540 2023-11-16T16:48:27.148 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.146+0000 7fb9d7fff700 1 -- 172.21.15.2:0/2555033012 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fb9e00051c0 con 0x7fb9f0100540 2023-11-16T16:48:27.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.147+0000 7fb9d7fff700 1 --2- 172.21.15.2:0/2555033012 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb9d80a1260 0x7fb9d80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:27.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.147+0000 7fb9d7fff700 1 -- 172.21.15.2:0/2555033012 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb9e00cdb30 con 0x7fb9f0100540 2023-11-16T16:48:27.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.148+0000 7fb9ee7fc700 1 --2- 172.21.15.2:0/2555033012 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb9d80a1260 0x7fb9d80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:27.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.148+0000 7fb9d7fff700 1 --2- 172.21.15.2:0/2555033012 >> [v2:172.21.15.141:6808/1025476846,v1:172.21.15.141:6809/1025476846] conn(0x7fb9d80ac270 0x7fb9d80ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:27.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.148+0000 7fb9d7fff700 1 -- 172.21.15.2:0/2555033012 --> [v2:172.21.15.141:6808/1025476846,v1:172.21.15.141:6809/1025476846] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb9d80aedd0 con 0x7fb9d80ac270 2023-11-16T16:48:27.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.148+0000 7fb9d7fff700 1 -- 172.21.15.2:0/2555033012 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb9e00d22c0 con 0x7fb9f0100540 2023-11-16T16:48:27.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.148+0000 7fb9ef7fe700 1 --2- 172.21.15.2:0/2555033012 >> [v2:172.21.15.141:6808/1025476846,v1:172.21.15.141:6809/1025476846] conn(0x7fb9d80ac270 0x7fb9d80ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:27.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.148+0000 7fb9ee7fc700 1 --2- 172.21.15.2:0/2555033012 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb9d80a1260 0x7fb9d80a3710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0199b80 tx=0x7fb9e4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:27.151 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.149+0000 7fb9ef7fe700 1 --2- 172.21.15.2:0/2555033012 >> [v2:172.21.15.141:6808/1025476846,v1:172.21.15.141:6809/1025476846] conn(0x7fb9d80ac270 0x7fb9d80ae700 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:27.151 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.150+0000 7fb9d7fff700 1 -- 172.21.15.2:0/2555033012 <== osd.4 v2:172.21.15.141:6808/1025476846 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb9d80aedd0 con 0x7fb9d80ac270 2023-11-16T16:48:27.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:26 smithi002 ceph-mon[128712]: pgmap v312: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2023-11-16T16:48:27.168 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.167+0000 7fb9f57f5700 1 -- 172.21.15.2:0/2555033012 --> [v2:172.21.15.141:6808/1025476846,v1:172.21.15.141:6809/1025476846] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb9f01104b0 con 0x7fb9d80ac270 2023-11-16T16:48:27.169 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.168+0000 7fb9d7fff700 1 -- 172.21.15.2:0/2555033012 <== osd.4 v2:172.21.15.141:6808/1025476846 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb9f01104b0 con 0x7fb9d80ac270 2023-11-16T16:48:27.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.168+0000 7fb9f57f5700 1 -- 172.21.15.2:0/2555033012 >> [v2:172.21.15.141:6808/1025476846,v1:172.21.15.141:6809/1025476846] conn(0x7fb9d80ac270 msgr2=0x7fb9d80ae700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:27.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.168+0000 7fb9f57f5700 1 --2- 172.21.15.2:0/2555033012 >> [v2:172.21.15.141:6808/1025476846,v1:172.21.15.141:6809/1025476846] conn(0x7fb9d80ac270 0x7fb9d80ae700 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.168+0000 7fb9f57f5700 1 -- 172.21.15.2:0/2555033012 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb9d80a1260 msgr2=0x7fb9d80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:27.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.168+0000 7fb9f57f5700 1 --2- 172.21.15.2:0/2555033012 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb9d80a1260 0x7fb9d80a3710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0199b80 tx=0x7fb9e4009260 comp rx=0 tx=0).stop 2023-11-16T16:48:27.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.168+0000 7fb9f57f5700 1 -- 172.21.15.2:0/2555033012 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb9f0100540 msgr2=0x7fb9f0198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:27.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.168+0000 7fb9f57f5700 1 --2- 172.21.15.2:0/2555033012 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb9f0100540 0x7fb9f0198560 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb9e0000c00 tx=0x7fb9e000bfd0 comp rx=0 tx=0).stop 2023-11-16T16:48:27.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.168+0000 7fb9ef7fe700 1 -- 172.21.15.2:0/2555033012 reap_dead start 2023-11-16T16:48:27.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.168+0000 7fb9f57f5700 1 -- 172.21.15.2:0/2555033012 shutdown_connections 2023-11-16T16:48:27.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.168+0000 7fb9f57f5700 1 -- 172.21.15.2:0/2555033012 >> 172.21.15.2:0/2555033012 conn(0x7fb9f00fbdc0 msgr2=0x7fb9f00fc930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:27.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.169+0000 7fb9f57f5700 1 -- 172.21.15.2:0/2555033012 shutdown_connections 2023-11-16T16:48:27.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:27.169+0000 7fb9f57f5700 1 -- 172.21.15.2:0/2555033012 wait complete. 2023-11-16T16:48:27.250 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:27.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:26 smithi141 ceph-mon[131184]: pgmap v312: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2023-11-16T16:48:27.282 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:27.913 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:28.477 INFO:teuthology.orchestra.run.smithi002.stdout:141733920825 2023-11-16T16:48:28.477 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph osd last-stat-seq osd.4 2023-11-16T16:48:28.528 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:29.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:28 smithi155 ceph-mon[131607]: pgmap v313: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2023-11-16T16:48:29.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:28 smithi002 ceph-mon[128712]: pgmap v313: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2023-11-16T16:48:29.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:28 smithi141 ceph-mon[131184]: pgmap v313: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2023-11-16T16:48:30.152 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:29 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:48:30.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.230+0000 7fcf0a2e9700 1 -- 172.21.15.2:0/2787042836 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf04101540 msgr2=0x7fcf04101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:30.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.230+0000 7fcf0a2e9700 1 --2- 172.21.15.2:0/2787042836 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf04101540 0x7fcf04101910 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fceec009990 tx=0x7fceec009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:30.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.231+0000 7fcf0a2e9700 1 -- 172.21.15.2:0/2787042836 shutdown_connections 2023-11-16T16:48:30.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.231+0000 7fcf0a2e9700 1 --2- 172.21.15.2:0/2787042836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf0410a890 0x7fcf0410ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:30.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.231+0000 7fcf0a2e9700 1 --2- 172.21.15.2:0/2787042836 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf04101540 0x7fcf04101910 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:30.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.231+0000 7fcf0a2e9700 1 --2- 172.21.15.2:0/2787042836 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf04101e50 0x7fcf0410a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:30.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.231+0000 7fcf0a2e9700 1 -- 172.21.15.2:0/2787042836 >> 172.21.15.2:0/2787042836 conn(0x7fcf040770a0 msgr2=0x7fcf040774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:30.232 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.231+0000 7fcf0a2e9700 1 -- 172.21.15.2:0/2787042836 shutdown_connections 2023-11-16T16:48:30.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.231+0000 7fcf0a2e9700 1 -- 172.21.15.2:0/2787042836 wait complete. 2023-11-16T16:48:30.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.233+0000 7fcf0a2e9700 1 Processor -- start 2023-11-16T16:48:30.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.233+0000 7fcf0a2e9700 1 -- start start 2023-11-16T16:48:30.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.234+0000 7fcf0a2e9700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf04101540 0x7fcf041a2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:30.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.234+0000 7fcf0a2e9700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf04101e50 0x7fcf041a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:30.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.234+0000 7fcf0a2e9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf0410a890 0x7fcf0419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:30.236 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.234+0000 7fcf0a2e9700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0419cef0 con 0x7fcf0410a890 2023-11-16T16:48:30.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.234+0000 7fcf0a2e9700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0419d060 con 0x7fcf04101e50 2023-11-16T16:48:30.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.234+0000 7fcf0a2e9700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0419d220 con 0x7fcf04101540 2023-11-16T16:48:30.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.235+0000 7fcefb7fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf04101e50 0x7fcf041a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:30.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.235+0000 7fcf08886700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf0410a890 0x7fcf0419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:30.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.235+0000 7fcefb7fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf04101e50 0x7fcf041a2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:36244/0 (socket says 172.21.15.2:36244) 2023-11-16T16:48:30.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.235+0000 7fcefb7fe700 1 -- 172.21.15.2:0/1049399325 learned_addr learned my addr 172.21.15.2:0/1049399325 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:30.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.235+0000 7fcefb7fe700 1 -- 172.21.15.2:0/1049399325 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf04101540 msgr2=0x7fcf041a2800 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-16T16:48:30.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.235+0000 7fcefb7fe700 1 --2- 172.21.15.2:0/1049399325 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf04101540 0x7fcf041a2800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:30.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.235+0000 7fcefb7fe700 1 -- 172.21.15.2:0/1049399325 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf0410a890 msgr2=0x7fcf0419c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:30.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.235+0000 7fcefb7fe700 1 --2- 172.21.15.2:0/1049399325 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf0410a890 0x7fcf0419c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:30.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.235+0000 7fcefb7fe700 1 -- 172.21.15.2:0/1049399325 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceec0095f0 con 0x7fcf04101e50 2023-11-16T16:48:30.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.235+0000 7fcf08886700 1 --2- 172.21.15.2:0/1049399325 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf0410a890 0x7fcf0419c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:30.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.236+0000 7fcefb7fe700 1 --2- 172.21.15.2:0/1049399325 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf04101e50 0x7fcf041a2d40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcef400e9a0 tx=0x7fcef400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:30.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.236+0000 7fcf01ffb700 1 -- 172.21.15.2:0/1049399325 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef4009790 con 0x7fcf04101e50 2023-11-16T16:48:30.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.236+0000 7fcf01ffb700 1 -- 172.21.15.2:0/1049399325 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcef400cbd0 con 0x7fcf04101e50 2023-11-16T16:48:30.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.236+0000 7fcf01ffb700 1 -- 172.21.15.2:0/1049399325 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef4018680 con 0x7fcf04101e50 2023-11-16T16:48:30.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.237+0000 7fcf0a2e9700 1 -- 172.21.15.2:0/1049399325 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf0410e210 con 0x7fcf04101e50 2023-11-16T16:48:30.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.238+0000 7fcf0a2e9700 1 -- 172.21.15.2:0/1049399325 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf0410e080 con 0x7fcf04101e50 2023-11-16T16:48:30.240 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.239+0000 7fcf0a2e9700 1 -- 172.21.15.2:0/1049399325 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf0410e3a0 con 0x7fcf04101e50 2023-11-16T16:48:30.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.242+0000 7fcf01ffb700 1 -- 172.21.15.2:0/1049399325 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fcef400cd40 con 0x7fcf04101e50 2023-11-16T16:48:30.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.245+0000 7fcf01ffb700 1 --2- 172.21.15.2:0/1049399325 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fcef00a12e0 0x7fcef00a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:30.246 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.245+0000 7fcf01ffb700 1 -- 172.21.15.2:0/1049399325 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcef4014070 con 0x7fcf04101e50 2023-11-16T16:48:30.246 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.245+0000 7fcf03fff700 1 --2- 172.21.15.2:0/1049399325 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fcef00a12e0 0x7fcef00a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:30.246 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.246+0000 7fcf03fff700 1 --2- 172.21.15.2:0/1049399325 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fcef00a12e0 0x7fcef00a3790 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fceec0098e0 tx=0x7fceec000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:30.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.248+0000 7fcf01ffb700 1 -- 172.21.15.2:0/1049399325 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcef40964b0 con 0x7fcf04101e50 2023-11-16T16:48:30.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:29 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:48:30.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:29 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:48:30.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.478+0000 7fcf0a2e9700 1 -- 172.21.15.2:0/1049399325 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fcf04066ea0 con 0x7fcf04101e50 2023-11-16T16:48:30.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.478+0000 7fcf01ffb700 1 -- 172.21.15.2:0/1049399325 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcef4099ad0 con 0x7fcf04101e50 2023-11-16T16:48:30.480 INFO:teuthology.orchestra.run.smithi002.stdout:34359738443 2023-11-16T16:48:30.482 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.481+0000 7fcf0a2e9700 1 -- 172.21.15.2:0/1049399325 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fcef00a12e0 msgr2=0x7fcef00a3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:30.482 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.481+0000 7fcf0a2e9700 1 --2- 172.21.15.2:0/1049399325 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fcef00a12e0 0x7fcef00a3790 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fceec0098e0 tx=0x7fceec000bc0 comp rx=0 tx=0).stop 2023-11-16T16:48:30.482 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.481+0000 7fcf0a2e9700 1 -- 172.21.15.2:0/1049399325 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf04101e50 msgr2=0x7fcf041a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:30.482 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.481+0000 7fcf0a2e9700 1 --2- 172.21.15.2:0/1049399325 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf04101e50 0x7fcf041a2d40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcef400e9a0 tx=0x7fcef400ecb0 comp rx=0 tx=0).stop 2023-11-16T16:48:30.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.481+0000 7fcf0a2e9700 1 -- 172.21.15.2:0/1049399325 shutdown_connections 2023-11-16T16:48:30.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.481+0000 7fcf0a2e9700 1 --2- 172.21.15.2:0/1049399325 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fcef00a12e0 0x7fcef00a3790 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:30.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.481+0000 7fcf0a2e9700 1 --2- 172.21.15.2:0/1049399325 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf04101540 0x7fcf041a2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:30.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.481+0000 7fcf0a2e9700 1 --2- 172.21.15.2:0/1049399325 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf0410a890 0x7fcf0419c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:30.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.481+0000 7fcf0a2e9700 1 --2- 172.21.15.2:0/1049399325 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcf04101e50 0x7fcf041a2d40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:30.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.481+0000 7fcf0a2e9700 1 -- 172.21.15.2:0/1049399325 >> 172.21.15.2:0/1049399325 conn(0x7fcf040770a0 msgr2=0x7fcf040fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:30.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.481+0000 7fcf0a2e9700 1 -- 172.21.15.2:0/1049399325 shutdown_connections 2023-11-16T16:48:30.483 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:30.481+0000 7fcf0a2e9700 1 -- 172.21.15.2:0/1049399325 wait complete. 2023-11-16T16:48:30.500 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:30.706 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:31.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:30 smithi002 ceph-mon[128712]: pgmap v314: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2023-11-16T16:48:31.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:30 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1049399325' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-11-16T16:48:31.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:30 smithi141 ceph-mon[131184]: pgmap v314: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2023-11-16T16:48:31.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:30 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1049399325' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-11-16T16:48:31.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:30 smithi155 ceph-mon[131607]: pgmap v314: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2023-11-16T16:48:31.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:30 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1049399325' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-11-16T16:48:31.449 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:31.629 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738440 got 34359738443 for osd.0 2023-11-16T16:48:31.630 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:31.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.771+0000 7feea0abe700 1 -- 172.21.15.2:0/3690070137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee9c104500 msgr2=0x7fee9c10c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:31.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.771+0000 7feea0abe700 1 --2- 172.21.15.2:0/3690070137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee9c104500 0x7fee9c10c080 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fee9400b0d0 tx=0x7fee9400b3e0 comp rx=0 tx=0).stop 2023-11-16T16:48:31.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.771+0000 7feea0abe700 1 -- 172.21.15.2:0/3690070137 shutdown_connections 2023-11-16T16:48:31.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.771+0000 7feea0abe700 1 --2- 172.21.15.2:0/3690070137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee9c104500 0x7fee9c10c080 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:31.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.771+0000 7feea0abe700 1 --2- 172.21.15.2:0/3690070137 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee9c0ff460 0x7fee9c0ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:31.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.771+0000 7feea0abe700 1 --2- 172.21.15.2:0/3690070137 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee9c0ffe00 0x7fee9c103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:31.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.771+0000 7feea0abe700 1 -- 172.21.15.2:0/3690070137 >> 172.21.15.2:0/3690070137 conn(0x7fee9c0770a0 msgr2=0x7fee9c0774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:31.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.772+0000 7feea0abe700 1 -- 172.21.15.2:0/3690070137 shutdown_connections 2023-11-16T16:48:31.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.773+0000 7feea0abe700 1 -- 172.21.15.2:0/3690070137 wait complete. 2023-11-16T16:48:31.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.774+0000 7feea0abe700 1 Processor -- start 2023-11-16T16:48:31.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.774+0000 7feea0abe700 1 -- start start 2023-11-16T16:48:31.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.775+0000 7feea0abe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee9c0ff460 0x7fee9c106100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:31.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.775+0000 7feea0abe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee9c0ffe00 0x7fee9c106640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:31.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.775+0000 7feea0abe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee9c104500 0x7fee9c106b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:31.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.775+0000 7feea0abe700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c1a8df0 con 0x7fee9c104500 2023-11-16T16:48:31.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.775+0000 7feea0abe700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c1a8f60 con 0x7fee9c0ffe00 2023-11-16T16:48:31.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.775+0000 7feea0abe700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c1a9120 con 0x7fee9c0ff460 2023-11-16T16:48:31.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.775+0000 7fee9a59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee9c0ff460 0x7fee9c106100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:31.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.775+0000 7fee9a59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee9c0ff460 0x7fee9c106100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:33456/0 (socket says 172.21.15.2:33456) 2023-11-16T16:48:31.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.775+0000 7fee99d9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee9c0ffe00 0x7fee9c106640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:31.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.775+0000 7fee9a59c700 1 -- 172.21.15.2:0/2602064912 learned_addr learned my addr 172.21.15.2:0/2602064912 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:31.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.775+0000 7fee9ad9d700 1 --2- 172.21.15.2:0/2602064912 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee9c104500 0x7fee9c106b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:31.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.776+0000 7fee9a59c700 1 -- 172.21.15.2:0/2602064912 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee9c0ffe00 msgr2=0x7fee9c106640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:31.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.776+0000 7fee9a59c700 1 --2- 172.21.15.2:0/2602064912 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee9c0ffe00 0x7fee9c106640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:31.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.776+0000 7fee9a59c700 1 -- 172.21.15.2:0/2602064912 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee9c104500 msgr2=0x7fee9c106b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:31.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.776+0000 7fee9a59c700 1 --2- 172.21.15.2:0/2602064912 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee9c104500 0x7fee9c106b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:31.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.776+0000 7fee9a59c700 1 -- 172.21.15.2:0/2602064912 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8c009520 con 0x7fee9c0ff460 2023-11-16T16:48:31.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.776+0000 7fee99d9b700 1 --2- 172.21.15.2:0/2602064912 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee9c0ffe00 0x7fee9c106640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:31.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.776+0000 7fee9a59c700 1 --2- 172.21.15.2:0/2602064912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee9c0ff460 0x7fee9c106100 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fee8c00eb30 tx=0x7fee8c00ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:31.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.776+0000 7fee8b7fe700 1 -- 172.21.15.2:0/2602064912 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8c009880 con 0x7fee9c0ff460 2023-11-16T16:48:31.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.776+0000 7fee8b7fe700 1 -- 172.21.15.2:0/2602064912 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee8c0044e0 con 0x7fee9c0ff460 2023-11-16T16:48:31.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.776+0000 7feea0abe700 1 -- 172.21.15.2:0/2602064912 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee94009d00 con 0x7fee9c0ff460 2023-11-16T16:48:31.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.777+0000 7fee8b7fe700 1 -- 172.21.15.2:0/2602064912 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8c003d80 con 0x7fee9c0ff460 2023-11-16T16:48:31.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.777+0000 7feea0abe700 1 -- 172.21.15.2:0/2602064912 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee9c10f290 con 0x7fee9c0ff460 2023-11-16T16:48:31.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.779+0000 7feea0abe700 1 -- 172.21.15.2:0/2602064912 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee7c005320 con 0x7fee9c0ff460 2023-11-16T16:48:31.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.781+0000 7fee8b7fe700 1 -- 172.21.15.2:0/2602064912 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fee8c004020 con 0x7fee9c0ff460 2023-11-16T16:48:31.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.783+0000 7fee8b7fe700 1 --2- 172.21.15.2:0/2602064912 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fee840a11e0 0x7fee840a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:31.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.784+0000 7fee8b7fe700 1 -- 172.21.15.2:0/2602064912 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fee8c014070 con 0x7fee9c0ff460 2023-11-16T16:48:31.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.785+0000 7fee99d9b700 1 --2- 172.21.15.2:0/2602064912 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fee840a11e0 0x7fee840a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:31.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.787+0000 7fee8b7fe700 1 -- 172.21.15.2:0/2602064912 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fee8c096540 con 0x7fee9c0ff460 2023-11-16T16:48:31.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:31.787+0000 7fee99d9b700 1 --2- 172.21.15.2:0/2602064912 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fee840a11e0 0x7fee840a3690 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fee9c1078f0 tx=0x7fee90009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:32.050 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.050+0000 7feea0abe700 1 -- 172.21.15.2:0/2602064912 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fee7c005190 con 0x7fee9c0ff460 2023-11-16T16:48:32.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.050+0000 7fee8b7fe700 1 -- 172.21.15.2:0/2602064912 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fee8c099b60 con 0x7fee9c0ff460 2023-11-16T16:48:32.051 INFO:teuthology.orchestra.run.smithi002.stdout:77309411396 2023-11-16T16:48:32.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.052+0000 7feea0abe700 1 -- 172.21.15.2:0/2602064912 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fee840a11e0 msgr2=0x7fee840a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:32.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.052+0000 7feea0abe700 1 --2- 172.21.15.2:0/2602064912 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fee840a11e0 0x7fee840a3690 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fee9c1078f0 tx=0x7fee90009260 comp rx=0 tx=0).stop 2023-11-16T16:48:32.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.052+0000 7feea0abe700 1 -- 172.21.15.2:0/2602064912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee9c0ff460 msgr2=0x7fee9c106100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:32.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.052+0000 7feea0abe700 1 --2- 172.21.15.2:0/2602064912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee9c0ff460 0x7fee9c106100 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fee8c00eb30 tx=0x7fee8c00ee40 comp rx=0 tx=0).stop 2023-11-16T16:48:32.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.053+0000 7feea0abe700 1 -- 172.21.15.2:0/2602064912 shutdown_connections 2023-11-16T16:48:32.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.053+0000 7feea0abe700 1 --2- 172.21.15.2:0/2602064912 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fee840a11e0 0x7fee840a3690 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.053+0000 7feea0abe700 1 --2- 172.21.15.2:0/2602064912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee9c0ff460 0x7fee9c106100 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.053+0000 7feea0abe700 1 --2- 172.21.15.2:0/2602064912 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fee9c104500 0x7fee9c106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.053+0000 7feea0abe700 1 --2- 172.21.15.2:0/2602064912 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fee9c0ffe00 0x7fee9c106640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.053+0000 7feea0abe700 1 -- 172.21.15.2:0/2602064912 >> 172.21.15.2:0/2602064912 conn(0x7fee9c0770a0 msgr2=0x7fee9c0fcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:32.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.053+0000 7feea0abe700 1 -- 172.21.15.2:0/2602064912 shutdown_connections 2023-11-16T16:48:32.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.053+0000 7feea0abe700 1 -- 172.21.15.2:0/2602064912 wait complete. 2023-11-16T16:48:32.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.372+0000 7fbb7ffff700 1 -- 172.21.15.2:0/2457254018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb800737a0 msgr2=0x7fbb8010c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:32.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.372+0000 7fbb7ffff700 1 --2- 172.21.15.2:0/2457254018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb800737a0 0x7fbb8010c410 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbb74009990 tx=0x7fbb74009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.373+0000 7fbb7ffff700 1 -- 172.21.15.2:0/2457254018 shutdown_connections 2023-11-16T16:48:32.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.373+0000 7fbb7ffff700 1 --2- 172.21.15.2:0/2457254018 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb8010c950 0x7fbb8010ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.373+0000 7fbb7ffff700 1 --2- 172.21.15.2:0/2457254018 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb80072e90 0x7fbb80073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.373+0000 7fbb7ffff700 1 --2- 172.21.15.2:0/2457254018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb800737a0 0x7fbb8010c410 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.373+0000 7fbb7ffff700 1 -- 172.21.15.2:0/2457254018 >> 172.21.15.2:0/2457254018 conn(0x7fbb800fbbd0 msgr2=0x7fbb800fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:32.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.373+0000 7fbb7ffff700 1 -- 172.21.15.2:0/2457254018 shutdown_connections 2023-11-16T16:48:32.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.373+0000 7fbb7ffff700 1 -- 172.21.15.2:0/2457254018 wait complete. 2023-11-16T16:48:32.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.374+0000 7fbb7ffff700 1 Processor -- start 2023-11-16T16:48:32.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.374+0000 7fbb7ffff700 1 -- start start 2023-11-16T16:48:32.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.375+0000 7fbb7ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb80072e90 0x7fbb801983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:32.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.375+0000 7fbb7ffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb800737a0 0x7fbb801988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:32.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.375+0000 7fbb7ffff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb8010c950 0x7fbb8019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:32.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.375+0000 7fbb7ffff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8019d400 con 0x7fbb80072e90 2023-11-16T16:48:32.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.375+0000 7fbb7ffff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8019d570 con 0x7fbb8010c950 2023-11-16T16:48:32.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.375+0000 7fbb7ffff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8019d730 con 0x7fbb800737a0 2023-11-16T16:48:32.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.375+0000 7fbb7effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb80072e90 0x7fbb801983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:32.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.375+0000 7fbb7effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb80072e90 0x7fbb801983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48804/0 (socket says 172.21.15.2:48804) 2023-11-16T16:48:32.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.375+0000 7fbb7effd700 1 -- 172.21.15.2:0/1399837528 learned_addr learned my addr 172.21.15.2:0/1399837528 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:32.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.375+0000 7fbb7e7fc700 1 --2- 172.21.15.2:0/1399837528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb800737a0 0x7fbb801988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:32.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.376+0000 7fbb7f7fe700 1 --2- 172.21.15.2:0/1399837528 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb8010c950 0x7fbb8019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:32.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.376+0000 7fbb7e7fc700 1 -- 172.21.15.2:0/1399837528 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb8010c950 msgr2=0x7fbb8019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:32.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.376+0000 7fbb7e7fc700 1 --2- 172.21.15.2:0/1399837528 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb8010c950 0x7fbb8019cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.376+0000 7fbb7e7fc700 1 -- 172.21.15.2:0/1399837528 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb80072e90 msgr2=0x7fbb801983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:32.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.376+0000 7fbb7e7fc700 1 --2- 172.21.15.2:0/1399837528 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb80072e90 0x7fbb801983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.608 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.376+0000 7fbb7e7fc700 1 -- 172.21.15.2:0/1399837528 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb740095f0 con 0x7fbb800737a0 2023-11-16T16:48:32.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.376+0000 7fbb7effd700 1 --2- 172.21.15.2:0/1399837528 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb80072e90 0x7fbb801983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:32.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.376+0000 7fbb7e7fc700 1 --2- 172.21.15.2:0/1399837528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb800737a0 0x7fbb801988f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbb74009990 tx=0x7fbb74009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:32.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.376+0000 7fbb67fff700 1 -- 172.21.15.2:0/1399837528 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7401d070 con 0x7fbb800737a0 2023-11-16T16:48:32.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.377+0000 7fbb7ffff700 1 -- 172.21.15.2:0/1399837528 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb80112390 con 0x7fbb800737a0 2023-11-16T16:48:32.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.377+0000 7fbb67fff700 1 -- 172.21.15.2:0/1399837528 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb74021410 con 0x7fbb800737a0 2023-11-16T16:48:32.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.377+0000 7fbb67fff700 1 -- 172.21.15.2:0/1399837528 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb74017650 con 0x7fbb800737a0 2023-11-16T16:48:32.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.377+0000 7fbb7ffff700 1 -- 172.21.15.2:0/1399837528 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb80112200 con 0x7fbb800737a0 2023-11-16T16:48:32.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.378+0000 7fbb7ffff700 1 -- 172.21.15.2:0/1399837528 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb80112520 con 0x7fbb800737a0 2023-11-16T16:48:32.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.380+0000 7fbb67fff700 1 -- 172.21.15.2:0/1399837528 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fbb74021810 con 0x7fbb800737a0 2023-11-16T16:48:32.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.381+0000 7fbb67fff700 1 --2- 172.21.15.2:0/1399837528 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbb680a11e0 0x7fbb680a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:32.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.381+0000 7fbb67fff700 1 -- 172.21.15.2:0/1399837528 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbb740ce0b0 con 0x7fbb800737a0 2023-11-16T16:48:32.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.381+0000 7fbb7effd700 1 --2- 172.21.15.2:0/1399837528 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbb680a11e0 0x7fbb680a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:32.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.382+0000 7fbb7effd700 1 --2- 172.21.15.2:0/1399837528 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbb680a11e0 0x7fbb680a3690 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbb70011550 tx=0x7fbb70009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:32.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.385+0000 7fbb67fff700 1 -- 172.21.15.2:0/1399837528 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbb740973c0 con 0x7fbb800737a0 2023-11-16T16:48:32.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.603+0000 7fbb7ffff700 1 -- 172.21.15.2:0/1399837528 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fbb8004e8c0 con 0x7fbb800737a0 2023-11-16T16:48:32.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.604+0000 7fbb67fff700 1 -- 172.21.15.2:0/1399837528 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fbb7409a9e0 con 0x7fbb800737a0 2023-11-16T16:48:32.611 INFO:teuthology.orchestra.run.smithi002.stdout:107374182462 2023-11-16T16:48:32.612 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.606+0000 7fbb7ffff700 1 -- 172.21.15.2:0/1399837528 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbb680a11e0 msgr2=0x7fbb680a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:32.612 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.606+0000 7fbb7ffff700 1 --2- 172.21.15.2:0/1399837528 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbb680a11e0 0x7fbb680a3690 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbb70011550 tx=0x7fbb70009260 comp rx=0 tx=0).stop 2023-11-16T16:48:32.612 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.606+0000 7fbb7ffff700 1 -- 172.21.15.2:0/1399837528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb800737a0 msgr2=0x7fbb801988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:32.612 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.606+0000 7fbb7ffff700 1 --2- 172.21.15.2:0/1399837528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb800737a0 0x7fbb801988f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbb74009990 tx=0x7fbb74009580 comp rx=0 tx=0).stop 2023-11-16T16:48:32.612 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.606+0000 7fbb7ffff700 1 -- 172.21.15.2:0/1399837528 shutdown_connections 2023-11-16T16:48:32.612 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.606+0000 7fbb7ffff700 1 --2- 172.21.15.2:0/1399837528 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fbb680a11e0 0x7fbb680a3690 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.612 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.606+0000 7fbb7ffff700 1 --2- 172.21.15.2:0/1399837528 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbb8010c950 0x7fbb8019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.612 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.606+0000 7fbb7ffff700 1 --2- 172.21.15.2:0/1399837528 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb80072e90 0x7fbb801983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.613 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.606+0000 7fbb7ffff700 1 --2- 172.21.15.2:0/1399837528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb800737a0 0x7fbb801988f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:32.613 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.606+0000 7fbb7ffff700 1 -- 172.21.15.2:0/1399837528 >> 172.21.15.2:0/1399837528 conn(0x7fbb800fbbd0 msgr2=0x7fbb8010ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:32.613 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.606+0000 7fbb7ffff700 1 -- 172.21.15.2:0/1399837528 shutdown_connections 2023-11-16T16:48:32.613 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:32.607+0000 7fbb7ffff700 1 -- 172.21.15.2:0/1399837528 wait complete. 2023-11-16T16:48:32.829 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:32 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:32.829 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:32 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:32.829 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:32 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:32.829 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:32 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:32.829 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:32 smithi002 ceph-mon[128712]: pgmap v315: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2023-11-16T16:48:32.829 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:32 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2602064912' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-11-16T16:48:32.829 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:32 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1399837528' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-11-16T16:48:33.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:32 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:33.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:32 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:33.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:32 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:33.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:32 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:33.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:32 smithi141 ceph-mon[131184]: pgmap v315: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2023-11-16T16:48:33.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:32 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2602064912' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-11-16T16:48:33.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:32 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1399837528' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-11-16T16:48:33.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:32 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:33.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:32 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:33.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:32 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:33.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:32 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:33.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:32 smithi155 ceph-mon[131607]: pgmap v315: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2023-11-16T16:48:33.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:32 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2602064912' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-11-16T16:48:33.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:32 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1399837528' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-11-16T16:48:34.936 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.696+0000 7f5f1b59e700 1 -- 172.21.15.2:0/2954427427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1c10c950 msgr2=0x7f5f1c10ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:34.936 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.696+0000 7f5f1b59e700 1 --2- 172.21.15.2:0/2954427427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1c10c950 0x7f5f1c10ed30 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5f14009f90 tx=0x7f5f1400b330 comp rx=0 tx=0).stop 2023-11-16T16:48:34.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.696+0000 7f5f1b59e700 1 -- 172.21.15.2:0/2954427427 shutdown_connections 2023-11-16T16:48:34.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.696+0000 7f5f1b59e700 1 --2- 172.21.15.2:0/2954427427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5f1c072e90 0x7f5f1c073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:34.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.696+0000 7f5f1b59e700 1 --2- 172.21.15.2:0/2954427427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1c10c950 0x7f5f1c10ed30 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:34.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.696+0000 7f5f1b59e700 1 --2- 172.21.15.2:0/2954427427 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f1c0737a0 0x7f5f1c10c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:34.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.696+0000 7f5f1b59e700 1 -- 172.21.15.2:0/2954427427 >> 172.21.15.2:0/2954427427 conn(0x7f5f1c0fbbd0 msgr2=0x7f5f1c0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:34.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.696+0000 7f5f1b59e700 1 -- 172.21.15.2:0/2954427427 shutdown_connections 2023-11-16T16:48:34.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.696+0000 7f5f1b59e700 1 -- 172.21.15.2:0/2954427427 wait complete. 2023-11-16T16:48:34.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.697+0000 7f5f1b59e700 1 Processor -- start 2023-11-16T16:48:34.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.697+0000 7f5f1b59e700 1 -- start start 2023-11-16T16:48:34.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.697+0000 7f5f1b59e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5f1c072e90 0x7f5f1c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:34.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.698+0000 7f5f1b59e700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f1c0737a0 0x7f5f1c198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:34.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.698+0000 7f5f1b59e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1c10c950 0x7f5f1c19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:34.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.698+0000 7f5f1b59e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c19d4a0 con 0x7f5f1c10c950 2023-11-16T16:48:34.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.698+0000 7f5f1b59e700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c19d610 con 0x7f5f1c0737a0 2023-11-16T16:48:34.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.698+0000 7f5f1b59e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c19d7d0 con 0x7f5f1c072e90 2023-11-16T16:48:34.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.698+0000 7f5f1ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1c10c950 0x7f5f1c19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:34.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.698+0000 7f5f19d9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f1c0737a0 0x7f5f1c198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:34.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.698+0000 7f5f19d9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f1c0737a0 0x7f5f1c198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:56876/0 (socket says 172.21.15.2:56876) 2023-11-16T16:48:34.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.698+0000 7f5f1ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1c10c950 0x7f5f1c19ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49736/0 (socket says 172.21.15.2:49736) 2023-11-16T16:48:34.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.698+0000 7f5f1a59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5f1c072e90 0x7f5f1c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:34.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.698+0000 7f5f19d9b700 1 -- 172.21.15.2:0/4020586169 learned_addr learned my addr 172.21.15.2:0/4020586169 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:34.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.698+0000 7f5f19d9b700 1 -- 172.21.15.2:0/4020586169 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5f1c072e90 msgr2=0x7f5f1c198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:34.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.698+0000 7f5f19d9b700 1 --2- 172.21.15.2:0/4020586169 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5f1c072e90 0x7f5f1c198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:34.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.698+0000 7f5f19d9b700 1 -- 172.21.15.2:0/4020586169 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1c10c950 msgr2=0x7f5f1c19ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:34.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.698+0000 7f5f19d9b700 1 --2- 172.21.15.2:0/4020586169 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1c10c950 0x7f5f1c19ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:34.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.698+0000 7f5f19d9b700 1 -- 172.21.15.2:0/4020586169 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f14009c40 con 0x7f5f1c0737a0 2023-11-16T16:48:34.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.698+0000 7f5f1a59c700 1 --2- 172.21.15.2:0/4020586169 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5f1c072e90 0x7f5f1c198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:34.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.699+0000 7f5f1ad9d700 1 --2- 172.21.15.2:0/4020586169 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1c10c950 0x7f5f1c19ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:34.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.699+0000 7f5f19d9b700 1 --2- 172.21.15.2:0/4020586169 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f1c0737a0 0x7f5f1c198990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5f1000e9c0 tx=0x7f5f1000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:34.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.699+0000 7f5f0b7fe700 1 -- 172.21.15.2:0/4020586169 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f10004a30 con 0x7f5f1c0737a0 2023-11-16T16:48:34.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.699+0000 7f5f0b7fe700 1 -- 172.21.15.2:0/4020586169 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f1000cc80 con 0x7f5f1c0737a0 2023-11-16T16:48:34.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.699+0000 7f5f1b59e700 1 -- 172.21.15.2:0/4020586169 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f1c112320 con 0x7f5f1c0737a0 2023-11-16T16:48:34.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.699+0000 7f5f0b7fe700 1 -- 172.21.15.2:0/4020586169 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f100054e0 con 0x7f5f1c0737a0 2023-11-16T16:48:34.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.699+0000 7f5f1b59e700 1 -- 172.21.15.2:0/4020586169 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f1c112190 con 0x7f5f1c0737a0 2023-11-16T16:48:34.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.701+0000 7f5f1b59e700 1 -- 172.21.15.2:0/4020586169 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f1c1124b0 con 0x7f5f1c0737a0 2023-11-16T16:48:34.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.702+0000 7f5f0b7fe700 1 -- 172.21.15.2:0/4020586169 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f5f10004b90 con 0x7f5f1c0737a0 2023-11-16T16:48:34.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.705+0000 7f5f0b7fe700 1 --2- 172.21.15.2:0/4020586169 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5f040a11e0 0x7f5f040a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:34.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.705+0000 7f5f0b7fe700 1 -- 172.21.15.2:0/4020586169 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5f10014070 con 0x7f5f1c0737a0 2023-11-16T16:48:34.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.705+0000 7f5f1a59c700 1 --2- 172.21.15.2:0/4020586169 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5f040a11e0 0x7f5f040a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:34.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.706+0000 7f5f1a59c700 1 --2- 172.21.15.2:0/4020586169 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5f040a11e0 0x7f5f040a3690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c011490 tx=0x7f5f0c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:34.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.708+0000 7f5f0b7fe700 1 -- 172.21.15.2:0/4020586169 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5f100960d0 con 0x7f5f1c0737a0 2023-11-16T16:48:34.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.945+0000 7f5f1b59e700 1 -- 172.21.15.2:0/4020586169 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f5f1c107270 con 0x7f5f1c0737a0 2023-11-16T16:48:34.947 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.946+0000 7f5f0b7fe700 1 -- 172.21.15.2:0/4020586169 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f5f100996f0 con 0x7f5f1c0737a0 2023-11-16T16:48:34.947 INFO:teuthology.orchestra.run.smithi002.stdout:253403070504 2023-11-16T16:48:34.949 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.948+0000 7f5f1b59e700 1 -- 172.21.15.2:0/4020586169 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5f040a11e0 msgr2=0x7f5f040a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:34.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.948+0000 7f5f1b59e700 1 --2- 172.21.15.2:0/4020586169 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5f040a11e0 0x7f5f040a3690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c011490 tx=0x7f5f0c009260 comp rx=0 tx=0).stop 2023-11-16T16:48:34.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.948+0000 7f5f1b59e700 1 -- 172.21.15.2:0/4020586169 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f1c0737a0 msgr2=0x7f5f1c198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:34.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.948+0000 7f5f1b59e700 1 --2- 172.21.15.2:0/4020586169 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f1c0737a0 0x7f5f1c198990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5f1000e9c0 tx=0x7f5f1000ecd0 comp rx=0 tx=0).stop 2023-11-16T16:48:34.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.949+0000 7f5f1b59e700 1 -- 172.21.15.2:0/4020586169 shutdown_connections 2023-11-16T16:48:34.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.949+0000 7f5f1b59e700 1 --2- 172.21.15.2:0/4020586169 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5f040a11e0 0x7f5f040a3690 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:34.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.949+0000 7f5f1b59e700 1 --2- 172.21.15.2:0/4020586169 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5f1c072e90 0x7f5f1c198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:34.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.949+0000 7f5f1b59e700 1 --2- 172.21.15.2:0/4020586169 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1c10c950 0x7f5f1c19ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:34.950 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.949+0000 7f5f1b59e700 1 --2- 172.21.15.2:0/4020586169 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5f1c0737a0 0x7f5f1c198990 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:34.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.949+0000 7f5f1b59e700 1 -- 172.21.15.2:0/4020586169 >> 172.21.15.2:0/4020586169 conn(0x7f5f1c0fbbd0 msgr2=0x7f5f1c1043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:34.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.950+0000 7f5f1b59e700 1 -- 172.21.15.2:0/4020586169 shutdown_connections 2023-11-16T16:48:34.951 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.950+0000 7f5f1b59e700 1 -- 172.21.15.2:0/4020586169 wait complete. 2023-11-16T16:48:34.985 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411394 got 77309411396 for osd.2 2023-11-16T16:48:34.985 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:35.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:34 smithi155 ceph-mon[131607]: pgmap v316: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-11-16T16:48:35.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:34 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/842039127' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-11-16T16:48:35.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:34 smithi002 ceph-mon[128712]: pgmap v316: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-11-16T16:48:35.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:34 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/842039127' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-11-16T16:48:35.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:34 smithi141 ceph-mon[131184]: pgmap v316: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-11-16T16:48:35.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:34 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/842039127' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-11-16T16:48:35.351 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182460 got 107374182462 for osd.3 2023-11-16T16:48:35.351 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:35.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.453+0000 7f0e0f1b4700 1 -- 172.21.15.2:0/710889720 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e081055e0 msgr2=0x7f0e0810d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.453+0000 7f0e0f1b4700 1 --2- 172.21.15.2:0/710889720 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e081055e0 0x7f0e0810d160 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000b0d0 tx=0x7f0e0000b3e0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.454+0000 7f0e0f1b4700 1 -- 172.21.15.2:0/710889720 shutdown_connections 2023-11-16T16:48:35.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.454+0000 7f0e0f1b4700 1 --2- 172.21.15.2:0/710889720 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e08100540 0x7f0e08100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.454+0000 7f0e0f1b4700 1 --2- 172.21.15.2:0/710889720 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e081055e0 0x7f0e0810d160 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.454+0000 7f0e0f1b4700 1 --2- 172.21.15.2:0/710889720 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e08100ee0 0x7f0e08104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.454+0000 7f0e0f1b4700 1 -- 172.21.15.2:0/710889720 >> 172.21.15.2:0/710889720 conn(0x7f0e080fbdc0 msgr2=0x7f0e080fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:35.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.454+0000 7f0e0f1b4700 1 -- 172.21.15.2:0/710889720 shutdown_connections 2023-11-16T16:48:35.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.454+0000 7f0e0f1b4700 1 -- 172.21.15.2:0/710889720 wait complete. 2023-11-16T16:48:35.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.455+0000 7f0e0f1b4700 1 Processor -- start 2023-11-16T16:48:35.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.455+0000 7f0e0f1b4700 1 -- start start 2023-11-16T16:48:35.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.455+0000 7f0e0f1b4700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e08100540 0x7f0e08072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.455+0000 7f0e0f1b4700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e08100ee0 0x7f0e08073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.455+0000 7f0e0f1b4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e081055e0 0x7f0e081a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.455+0000 7f0e0f1b4700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e08073c20 con 0x7f0e081055e0 2023-11-16T16:48:35.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.455+0000 7f0e0f1b4700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e081a4bd0 con 0x7f0e08100ee0 2023-11-16T16:48:35.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.455+0000 7f0e0f1b4700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e081a4d40 con 0x7f0e08100540 2023-11-16T16:48:35.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.456+0000 7f0e0d751700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e081055e0 0x7f0e081a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.456+0000 7f0e07fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e08100ee0 0x7f0e08073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.456+0000 7f0e0d751700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e081055e0 0x7f0e081a4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49714/0 (socket says 172.21.15.2:49714) 2023-11-16T16:48:35.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.456+0000 7f0e07fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e08100ee0 0x7f0e08073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:56852/0 (socket says 172.21.15.2:56852) 2023-11-16T16:48:35.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.456+0000 7f0e0d751700 1 -- 172.21.15.2:0/842039127 learned_addr learned my addr 172.21.15.2:0/842039127 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:35.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.456+0000 7f0e0cf50700 1 --2- 172.21.15.2:0/842039127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e08100540 0x7f0e08072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.456+0000 7f0e0d751700 1 -- 172.21.15.2:0/842039127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e08100540 msgr2=0x7f0e08072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.456+0000 7f0e0d751700 1 --2- 172.21.15.2:0/842039127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e08100540 0x7f0e08072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.456+0000 7f0e0d751700 1 -- 172.21.15.2:0/842039127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e08100ee0 msgr2=0x7f0e08073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.456+0000 7f0e0d751700 1 --2- 172.21.15.2:0/842039127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e08100ee0 0x7f0e08073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.456+0000 7f0e0d751700 1 -- 172.21.15.2:0/842039127 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e00009d00 con 0x7f0e081055e0 2023-11-16T16:48:35.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.456+0000 7f0e0cf50700 1 --2- 172.21.15.2:0/842039127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e08100540 0x7f0e08072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:48:35.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.456+0000 7f0e07fff700 1 --2- 172.21.15.2:0/842039127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e08100ee0 0x7f0e08073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:35.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.456+0000 7f0e0d751700 1 --2- 172.21.15.2:0/842039127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e081055e0 0x7f0e081a4690 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0e00007890 tx=0x7f0e000078c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:35.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.457+0000 7f0e05ffb700 1 -- 172.21.15.2:0/842039127 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0000e040 con 0x7f0e081055e0 2023-11-16T16:48:35.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.457+0000 7f0e05ffb700 1 -- 172.21.15.2:0/842039127 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e0000bdb0 con 0x7f0e081055e0 2023-11-16T16:48:35.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.457+0000 7f0e05ffb700 1 -- 172.21.15.2:0/842039127 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0001d470 con 0x7f0e081055e0 2023-11-16T16:48:35.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.457+0000 7f0e0f1b4700 1 -- 172.21.15.2:0/842039127 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e08110370 con 0x7f0e081055e0 2023-11-16T16:48:35.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.457+0000 7f0e0f1b4700 1 -- 172.21.15.2:0/842039127 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e081101e0 con 0x7f0e081055e0 2023-11-16T16:48:35.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.458+0000 7f0e0f1b4700 1 -- 172.21.15.2:0/842039127 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e08110500 con 0x7f0e081055e0 2023-11-16T16:48:35.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.460+0000 7f0e05ffb700 1 -- 172.21.15.2:0/842039127 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f0e000050b0 con 0x7f0e081055e0 2023-11-16T16:48:35.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.461+0000 7f0e05ffb700 1 --2- 172.21.15.2:0/842039127 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0df80a1260 0x7f0df80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.461+0000 7f0e05ffb700 1 -- 172.21.15.2:0/842039127 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0e000cdc30 con 0x7f0e081055e0 2023-11-16T16:48:35.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.461+0000 7f0e0cf50700 1 --2- 172.21.15.2:0/842039127 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0df80a1260 0x7f0df80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.462+0000 7f0e0cf50700 1 --2- 172.21.15.2:0/842039127 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0df80a1260 0x7f0df80a3710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00bc10 tx=0x7f0dfc00b5f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:35.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.465+0000 7f0e05ffb700 1 -- 172.21.15.2:0/842039127 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0e00096f40 con 0x7f0e081055e0 2023-11-16T16:48:35.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.690+0000 7f0e0f1b4700 1 -- 172.21.15.2:0/842039127 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f0e08066ea0 con 0x7f0e081055e0 2023-11-16T16:48:35.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.690+0000 7f0e05ffb700 1 -- 172.21.15.2:0/842039127 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f0e0009a560 con 0x7f0e081055e0 2023-11-16T16:48:35.367 INFO:teuthology.orchestra.run.smithi002.stdout:343597383706 2023-11-16T16:48:35.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.693+0000 7f0e0f1b4700 1 -- 172.21.15.2:0/842039127 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0df80a1260 msgr2=0x7f0df80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.693+0000 7f0e0f1b4700 1 --2- 172.21.15.2:0/842039127 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0df80a1260 0x7f0df80a3710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00bc10 tx=0x7f0dfc00b5f0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.693+0000 7f0e0f1b4700 1 -- 172.21.15.2:0/842039127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e081055e0 msgr2=0x7f0e081a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.693+0000 7f0e0f1b4700 1 --2- 172.21.15.2:0/842039127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e081055e0 0x7f0e081a4690 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0e00007890 tx=0x7f0e000078c0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.694+0000 7f0e0f1b4700 1 -- 172.21.15.2:0/842039127 shutdown_connections 2023-11-16T16:48:35.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.694+0000 7f0e0f1b4700 1 --2- 172.21.15.2:0/842039127 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f0df80a1260 0x7f0df80a3710 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.694+0000 7f0e0f1b4700 1 --2- 172.21.15.2:0/842039127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e08100540 0x7f0e08072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.694+0000 7f0e0f1b4700 1 --2- 172.21.15.2:0/842039127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e081055e0 0x7f0e081a4690 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.694+0000 7f0e0f1b4700 1 --2- 172.21.15.2:0/842039127 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0e08100ee0 0x7f0e08073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.694+0000 7f0e0f1b4700 1 -- 172.21.15.2:0/842039127 >> 172.21.15.2:0/842039127 conn(0x7f0e080fbdc0 msgr2=0x7f0e080fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:35.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.694+0000 7f0e0f1b4700 1 -- 172.21.15.2:0/842039127 shutdown_connections 2023-11-16T16:48:35.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.694+0000 7f0e0f1b4700 1 -- 172.21.15.2:0/842039127 wait complete. 2023-11-16T16:48:35.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.757+0000 7f4719bcc700 1 -- 172.21.15.2:0/2980490237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4714101330 msgr2=0x7f4714101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.757+0000 7f4719bcc700 1 --2- 172.21.15.2:0/2980490237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4714101330 0x7f4714101700 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4708009990 tx=0x7f4708009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.757+0000 7f4719bcc700 1 -- 172.21.15.2:0/2980490237 shutdown_connections 2023-11-16T16:48:35.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.757+0000 7f4719bcc700 1 --2- 172.21.15.2:0/2980490237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4714101330 0x7f4714101700 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.757+0000 7f4719bcc700 1 --2- 172.21.15.2:0/2980490237 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f471410a680 0x7f471410cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.757+0000 7f4719bcc700 1 --2- 172.21.15.2:0/2980490237 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4714101c40 0x7f471410a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.757+0000 7f4719bcc700 1 -- 172.21.15.2:0/2980490237 >> 172.21.15.2:0/2980490237 conn(0x7f47140761a0 msgr2=0x7f47140765a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:35.445 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.757+0000 7f4719bcc700 1 -- 172.21.15.2:0/2980490237 shutdown_connections 2023-11-16T16:48:35.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.757+0000 7f4719bcc700 1 -- 172.21.15.2:0/2980490237 wait complete. 2023-11-16T16:48:35.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.758+0000 7f4719bcc700 1 Processor -- start 2023-11-16T16:48:35.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.758+0000 7f4719bcc700 1 -- start start 2023-11-16T16:48:35.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.758+0000 7f4719bcc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4714101330 0x7f4714198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.758+0000 7f4719bcc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4714101c40 0x7f47141989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.758+0000 7f4719bcc700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f471410a680 0x7f471419ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.758+0000 7f4719bcc700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471419d420 con 0x7f4714101330 2023-11-16T16:48:35.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.758+0000 7f4719bcc700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471419d590 con 0x7f471410a680 2023-11-16T16:48:35.446 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.758+0000 7f4719bcc700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471419d750 con 0x7f4714101c40 2023-11-16T16:48:35.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.759+0000 7f4718bca700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4714101330 0x7f4714198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.759+0000 7f4718bca700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4714101330 0x7f4714198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49752/0 (socket says 172.21.15.2:49752) 2023-11-16T16:48:35.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.759+0000 7f4718bca700 1 -- 172.21.15.2:0/1059688938 learned_addr learned my addr 172.21.15.2:0/1059688938 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:35.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.759+0000 7f47193cb700 1 --2- 172.21.15.2:0/1059688938 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f471410a680 0x7f471419ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.759+0000 7f4713fff700 1 --2- 172.21.15.2:0/1059688938 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4714101c40 0x7f47141989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.759+0000 7f4718bca700 1 -- 172.21.15.2:0/1059688938 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4714101c40 msgr2=0x7f47141989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.759+0000 7f4718bca700 1 --2- 172.21.15.2:0/1059688938 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4714101c40 0x7f47141989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.759+0000 7f4718bca700 1 -- 172.21.15.2:0/1059688938 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f471410a680 msgr2=0x7f471419ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.759+0000 7f4718bca700 1 --2- 172.21.15.2:0/1059688938 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f471410a680 0x7f471419ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.759+0000 7f4718bca700 1 -- 172.21.15.2:0/1059688938 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47080095f0 con 0x7f4714101330 2023-11-16T16:48:35.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.759+0000 7f47193cb700 1 --2- 172.21.15.2:0/1059688938 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f471410a680 0x7f471419ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:35.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.760+0000 7f4718bca700 1 --2- 172.21.15.2:0/1059688938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4714101330 0x7f4714198460 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4708005910 tx=0x7f4708004710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:35.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.760+0000 7f4711ffb700 1 -- 172.21.15.2:0/1059688938 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470801d070 con 0x7f4714101330 2023-11-16T16:48:35.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.760+0000 7f4719bcc700 1 -- 172.21.15.2:0/1059688938 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4714110160 con 0x7f4714101330 2023-11-16T16:48:35.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.760+0000 7f4711ffb700 1 -- 172.21.15.2:0/1059688938 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4708022470 con 0x7f4714101330 2023-11-16T16:48:35.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.760+0000 7f4719bcc700 1 -- 172.21.15.2:0/1059688938 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471410ffd0 con 0x7f4714101330 2023-11-16T16:48:35.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.761+0000 7f4711ffb700 1 -- 172.21.15.2:0/1059688938 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47080225e0 con 0x7f4714101330 2023-11-16T16:48:35.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.761+0000 7f4719bcc700 1 -- 172.21.15.2:0/1059688938 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47141102f0 con 0x7f4714101330 2023-11-16T16:48:35.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.765+0000 7f4711ffb700 1 -- 172.21.15.2:0/1059688938 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f470800fb10 con 0x7f4714101330 2023-11-16T16:48:35.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.766+0000 7f4711ffb700 1 --2- 172.21.15.2:0/1059688938 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f47040a11e0 0x7f47040a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.766+0000 7f4711ffb700 1 -- 172.21.15.2:0/1059688938 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f47080ce440 con 0x7f4714101330 2023-11-16T16:48:35.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.766+0000 7f4713fff700 1 --2- 172.21.15.2:0/1059688938 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f47040a11e0 0x7f47040a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.767+0000 7f4711ffb700 1 -- 172.21.15.2:0/1059688938 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f47080977b0 con 0x7f4714101330 2023-11-16T16:48:35.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:34.767+0000 7f4713fff700 1 --2- 172.21.15.2:0/1059688938 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f47040a11e0 0x7f47040a3690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4714199a80 tx=0x7f4700009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:35.449 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.009+0000 7f4719bcc700 1 -- 172.21.15.2:0/1059688938 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f471404e8c0 con 0x7f4714101330 2023-11-16T16:48:35.449 INFO:teuthology.orchestra.run.smithi002.stdout:309237645342 2023-11-16T16:48:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.010+0000 7f4711ffb700 1 -- 172.21.15.2:0/1059688938 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f4708022740 con 0x7f4714101330 2023-11-16T16:48:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.012+0000 7f4719bcc700 1 -- 172.21.15.2:0/1059688938 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f47040a11e0 msgr2=0x7f47040a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.012+0000 7f4719bcc700 1 --2- 172.21.15.2:0/1059688938 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f47040a11e0 0x7f47040a3690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4714199a80 tx=0x7f4700009260 comp rx=0 tx=0).stop 2023-11-16T16:48:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.012+0000 7f4719bcc700 1 -- 172.21.15.2:0/1059688938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4714101330 msgr2=0x7f4714198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.012+0000 7f4719bcc700 1 --2- 172.21.15.2:0/1059688938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4714101330 0x7f4714198460 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4708005910 tx=0x7f4708004710 comp rx=0 tx=0).stop 2023-11-16T16:48:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.012+0000 7f4719bcc700 1 -- 172.21.15.2:0/1059688938 shutdown_connections 2023-11-16T16:48:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.012+0000 7f4719bcc700 1 --2- 172.21.15.2:0/1059688938 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f47040a11e0 0x7f47040a3690 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.012+0000 7f4719bcc700 1 --2- 172.21.15.2:0/1059688938 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f471410a680 0x7f471419ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.012+0000 7f4719bcc700 1 --2- 172.21.15.2:0/1059688938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4714101330 0x7f4714198460 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.012+0000 7f4719bcc700 1 --2- 172.21.15.2:0/1059688938 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4714101c40 0x7f47141989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.012+0000 7f4719bcc700 1 -- 172.21.15.2:0/1059688938 >> 172.21.15.2:0/1059688938 conn(0x7f47140761a0 msgr2=0x7f471410aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.012+0000 7f4719bcc700 1 -- 172.21.15.2:0/1059688938 shutdown_connections 2023-11-16T16:48:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.012+0000 7f4719bcc700 1 -- 172.21.15.2:0/1059688938 wait complete. 2023-11-16T16:48:35.472 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:35.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.815+0000 7f7e6e998700 1 -- 172.21.15.2:0/3279477830 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e68100ef0 msgr2=0x7f7e68104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.816 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.815+0000 7f7e6e998700 1 --2- 172.21.15.2:0/3279477830 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e68100ef0 0x7f7e68104ec0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c0099c0 tx=0x7f7e5c009cd0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.815+0000 7f7e6e998700 1 -- 172.21.15.2:0/3279477830 shutdown_connections 2023-11-16T16:48:35.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.815+0000 7f7e6e998700 1 --2- 172.21.15.2:0/3279477830 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e681055f0 0x7f7e6810d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.815+0000 7f7e6e998700 1 --2- 172.21.15.2:0/3279477830 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e68100550 0x7f7e68100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.815+0000 7f7e6e998700 1 --2- 172.21.15.2:0/3279477830 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e68100ef0 0x7f7e68104ec0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.815+0000 7f7e6e998700 1 -- 172.21.15.2:0/3279477830 >> 172.21.15.2:0/3279477830 conn(0x7f7e680fbdf0 msgr2=0x7f7e680fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:35.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.815+0000 7f7e6e998700 1 -- 172.21.15.2:0/3279477830 shutdown_connections 2023-11-16T16:48:35.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.815+0000 7f7e6e998700 1 -- 172.21.15.2:0/3279477830 wait complete. 2023-11-16T16:48:35.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.816+0000 7f7e6e998700 1 Processor -- start 2023-11-16T16:48:35.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.816+0000 7f7e6e998700 1 -- start start 2023-11-16T16:48:35.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.816+0000 7f7e6e998700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e68100550 0x7f7e68072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.817+0000 7f7e6e998700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e68100ef0 0x7f7e68073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.817+0000 7f7e6e998700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e681055f0 0x7f7e680738b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.817+0000 7f7e6e998700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e681a4820 con 0x7f7e68100550 2023-11-16T16:48:35.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.817+0000 7f7e6e998700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e681a4990 con 0x7f7e68100ef0 2023-11-16T16:48:35.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.817+0000 7f7e6e998700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e681a4b50 con 0x7f7e681055f0 2023-11-16T16:48:35.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.817+0000 7f7e67fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e68100550 0x7f7e68072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.817+0000 7f7e67fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e68100550 0x7f7e68072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49776/0 (socket says 172.21.15.2:49776) 2023-11-16T16:48:35.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.817+0000 7f7e67fff700 1 -- 172.21.15.2:0/3940603443 learned_addr learned my addr 172.21.15.2:0/3940603443 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:35.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.817+0000 7f7e677fe700 1 --2- 172.21.15.2:0/3940603443 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e68100ef0 0x7f7e68073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.817+0000 7f7e6cf35700 1 --2- 172.21.15.2:0/3940603443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e681055f0 0x7f7e680738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.817+0000 7f7e67fff700 1 -- 172.21.15.2:0/3940603443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e681055f0 msgr2=0x7f7e680738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.817+0000 7f7e67fff700 1 --2- 172.21.15.2:0/3940603443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e681055f0 0x7f7e680738b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.817+0000 7f7e67fff700 1 -- 172.21.15.2:0/3940603443 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e68100ef0 msgr2=0x7f7e68073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:35.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.817+0000 7f7e67fff700 1 --2- 172.21.15.2:0/3940603443 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e68100ef0 0x7f7e68073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:35.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.817+0000 7f7e67fff700 1 -- 172.21.15.2:0/3940603443 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5c0095f0 con 0x7f7e68100550 2023-11-16T16:48:35.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.817+0000 7f7e6cf35700 1 --2- 172.21.15.2:0/3940603443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e681055f0 0x7f7e680738b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:48:35.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.818+0000 7f7e67fff700 1 --2- 172.21.15.2:0/3940603443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e68100550 0x7f7e68072df0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7e5800d980 tx=0x7f7e5800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:35.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.818+0000 7f7e657fa700 1 -- 172.21.15.2:0/3940603443 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5800bb60 con 0x7f7e68100550 2023-11-16T16:48:35.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.818+0000 7f7e657fa700 1 -- 172.21.15.2:0/3940603443 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e58007640 con 0x7f7e68100550 2023-11-16T16:48:35.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.818+0000 7f7e6e998700 1 -- 172.21.15.2:0/3940603443 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e68110370 con 0x7f7e68100550 2023-11-16T16:48:35.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.818+0000 7f7e657fa700 1 -- 172.21.15.2:0/3940603443 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e58004e90 con 0x7f7e68100550 2023-11-16T16:48:35.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.818+0000 7f7e6e998700 1 -- 172.21.15.2:0/3940603443 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e681101e0 con 0x7f7e68100550 2023-11-16T16:48:35.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.819+0000 7f7e6e998700 1 -- 172.21.15.2:0/3940603443 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e68110500 con 0x7f7e68100550 2023-11-16T16:48:35.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.821+0000 7f7e657fa700 1 -- 172.21.15.2:0/3940603443 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f7e5800bcc0 con 0x7f7e68100550 2023-11-16T16:48:35.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.823+0000 7f7e657fa700 1 --2- 172.21.15.2:0/3940603443 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f7e500a11e0 0x7f7e500a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:35.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.823+0000 7f7e657fa700 1 -- 172.21.15.2:0/3940603443 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7e580ccc80 con 0x7f7e68100550 2023-11-16T16:48:35.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.823+0000 7f7e677fe700 1 --2- 172.21.15.2:0/3940603443 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f7e500a11e0 0x7f7e500a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:35.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.824+0000 7f7e677fe700 1 --2- 172.21.15.2:0/3940603443 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f7e500a11e0 0x7f7e500a3690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c0098e0 tx=0x7f7e5c01a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:35.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:35.827+0000 7f7e657fa700 1 -- 172.21.15.2:0/3940603443 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7e58095f00 con 0x7f7e68100550 2023-11-16T16:48:36.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:36.042+0000 7f7e6e998700 1 -- 172.21.15.2:0/3940603443 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7e68066ea0 con 0x7f7e68100550 2023-11-16T16:48:36.044 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:36.043+0000 7f7e657fa700 1 -- 172.21.15.2:0/3940603443 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7e58016020 con 0x7f7e68100550 2023-11-16T16:48:36.044 INFO:teuthology.orchestra.run.smithi002.stdout:55834574922 2023-11-16T16:48:36.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:36.045+0000 7f7e6e998700 1 -- 172.21.15.2:0/3940603443 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f7e500a11e0 msgr2=0x7f7e500a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:36.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:36.045+0000 7f7e6e998700 1 --2- 172.21.15.2:0/3940603443 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f7e500a11e0 0x7f7e500a3690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c0098e0 tx=0x7f7e5c01a040 comp rx=0 tx=0).stop 2023-11-16T16:48:36.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:36.045+0000 7f7e6e998700 1 -- 172.21.15.2:0/3940603443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e68100550 msgr2=0x7f7e68072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:36.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:36.045+0000 7f7e6e998700 1 --2- 172.21.15.2:0/3940603443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e68100550 0x7f7e68072df0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7e5800d980 tx=0x7f7e5800dd40 comp rx=0 tx=0).stop 2023-11-16T16:48:36.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:36.045+0000 7f7e6e998700 1 -- 172.21.15.2:0/3940603443 shutdown_connections 2023-11-16T16:48:36.047 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:36.045+0000 7f7e6e998700 1 --2- 172.21.15.2:0/3940603443 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f7e500a11e0 0x7f7e500a3690 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:36.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:36.045+0000 7f7e6e998700 1 --2- 172.21.15.2:0/3940603443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e681055f0 0x7f7e680738b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:36.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:36.045+0000 7f7e6e998700 1 --2- 172.21.15.2:0/3940603443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e68100550 0x7f7e68072df0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:36.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:36.045+0000 7f7e6e998700 1 --2- 172.21.15.2:0/3940603443 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e68100ef0 0x7f7e68073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:36.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:36.045+0000 7f7e6e998700 1 -- 172.21.15.2:0/3940603443 >> 172.21.15.2:0/3940603443 conn(0x7f7e680fbdf0 msgr2=0x7f7e680fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:36.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:36.046+0000 7f7e6e998700 1 -- 172.21.15.2:0/3940603443 shutdown_connections 2023-11-16T16:48:36.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:36.046+0000 7f7e6e998700 1 -- 172.21.15.2:0/3940603443 wait complete. 2023-11-16T16:48:36.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:35 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/4020586169' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-11-16T16:48:36.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:35 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1059688938' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-11-16T16:48:36.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:35 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/4020586169' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-11-16T16:48:36.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:35 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1059688938' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-11-16T16:48:36.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:35 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/4020586169' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-11-16T16:48:36.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:35 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1059688938' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-11-16T16:48:36.505 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070502 got 253403070504 for osd.8 2023-11-16T16:48:36.505 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:36.595 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383703 got 343597383706 for osd.11 2023-11-16T16:48:36.595 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:37.132 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:36 smithi002 ceph-mon[128712]: pgmap v317: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2023-11-16T16:48:37.132 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:36 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3940603443' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-11-16T16:48:37.193 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574918 got 55834574922 for osd.1 2023-11-16T16:48:37.193 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:37.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:36 smithi141 ceph-mon[131184]: pgmap v317: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2023-11-16T16:48:37.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:36 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3940603443' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-11-16T16:48:37.341 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645340 got 309237645342 for osd.10 2023-11-16T16:48:37.342 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:37.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:36 smithi155 ceph-mon[131607]: pgmap v317: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2023-11-16T16:48:37.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:36 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3940603443' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-11-16T16:48:37.418 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:37.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.490+0000 7ff108441700 1 -- 172.21.15.2:0/513824572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1000ffc40 msgr2=0x7ff100104120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:37.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.490+0000 7ff108441700 1 --2- 172.21.15.2:0/513824572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1000ffc40 0x7ff100104120 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff0f00098f0 tx=0x7ff0f0009c00 comp rx=0 tx=0).stop 2023-11-16T16:48:37.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.490+0000 7ff108441700 1 -- 172.21.15.2:0/513824572 shutdown_connections 2023-11-16T16:48:37.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.490+0000 7ff108441700 1 --2- 172.21.15.2:0/513824572 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff100104660 0x7ff100104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:37.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.490+0000 7ff108441700 1 --2- 172.21.15.2:0/513824572 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1000ff330 0x7ff1000ff700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:37.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.490+0000 7ff108441700 1 --2- 172.21.15.2:0/513824572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1000ffc40 0x7ff100104120 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:37.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.490+0000 7ff108441700 1 -- 172.21.15.2:0/513824572 >> 172.21.15.2:0/513824572 conn(0x7ff1000770a0 msgr2=0x7ff1000774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:37.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.490+0000 7ff108441700 1 -- 172.21.15.2:0/513824572 shutdown_connections 2023-11-16T16:48:37.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.490+0000 7ff108441700 1 -- 172.21.15.2:0/513824572 wait complete. 2023-11-16T16:48:37.492 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.492+0000 7ff108441700 1 Processor -- start 2023-11-16T16:48:37.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.492+0000 7ff108441700 1 -- start start 2023-11-16T16:48:37.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.492+0000 7ff108441700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1000ff330 0x7ff1001a2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:37.493 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.492+0000 7ff108441700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1000ffc40 0x7ff1001a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:37.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.492+0000 7ff108441700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff100104660 0x7ff10019c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:37.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.492+0000 7ff108441700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10019cf10 con 0x7ff100104660 2023-11-16T16:48:37.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.492+0000 7ff108441700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10019d080 con 0x7ff1000ff330 2023-11-16T16:48:37.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.492+0000 7ff108441700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10019d240 con 0x7ff1000ffc40 2023-11-16T16:48:37.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.493+0000 7ff1061dd700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1000ff330 0x7ff1001a2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:37.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.493+0000 7ff1061dd700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1000ff330 0x7ff1001a2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:56932/0 (socket says 172.21.15.2:56932) 2023-11-16T16:48:37.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.493+0000 7ff1059dc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1000ffc40 0x7ff1001a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:37.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.493+0000 7ff1061dd700 1 -- 172.21.15.2:0/976924371 learned_addr learned my addr 172.21.15.2:0/976924371 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:37.494 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.493+0000 7ff1069de700 1 --2- 172.21.15.2:0/976924371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff100104660 0x7ff10019c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:37.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.493+0000 7ff1061dd700 1 -- 172.21.15.2:0/976924371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1000ffc40 msgr2=0x7ff1001a2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:37.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.493+0000 7ff1061dd700 1 --2- 172.21.15.2:0/976924371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1000ffc40 0x7ff1001a2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:37.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.493+0000 7ff1061dd700 1 -- 172.21.15.2:0/976924371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff100104660 msgr2=0x7ff10019c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:37.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.493+0000 7ff1061dd700 1 --2- 172.21.15.2:0/976924371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff100104660 0x7ff10019c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:37.495 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.493+0000 7ff1061dd700 1 -- 172.21.15.2:0/976924371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f0009520 con 0x7ff1000ff330 2023-11-16T16:48:37.496 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.496+0000 7ff1061dd700 1 --2- 172.21.15.2:0/976924371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1000ff330 0x7ff1001a2820 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00e9a0 tx=0x7ff0fc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:37.497 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.496+0000 7ff0f77fe700 1 -- 172.21.15.2:0/976924371 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc009790 con 0x7ff1000ff330 2023-11-16T16:48:37.497 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.496+0000 7ff0f77fe700 1 -- 172.21.15.2:0/976924371 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff0fc010430 con 0x7ff1000ff330 2023-11-16T16:48:37.497 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.496+0000 7ff108441700 1 -- 172.21.15.2:0/976924371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff100116970 con 0x7ff1000ff330 2023-11-16T16:48:37.497 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.496+0000 7ff0f77fe700 1 -- 172.21.15.2:0/976924371 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc018680 con 0x7ff1000ff330 2023-11-16T16:48:37.497 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.496+0000 7ff108441700 1 -- 172.21.15.2:0/976924371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1001167e0 con 0x7ff1000ff330 2023-11-16T16:48:37.498 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.497+0000 7ff0f57fa700 1 -- 172.21.15.2:0/976924371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff100116b00 con 0x7ff1000ff330 2023-11-16T16:48:37.501 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.501+0000 7ff0f77fe700 1 -- 172.21.15.2:0/976924371 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7ff0fc0105a0 con 0x7ff1000ff330 2023-11-16T16:48:37.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.503+0000 7ff0f77fe700 1 --2- 172.21.15.2:0/976924371 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff0ec0a11e0 0x7ff0ec0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:37.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.503+0000 7ff0f77fe700 1 -- 172.21.15.2:0/976924371 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff0fc014070 con 0x7ff1000ff330 2023-11-16T16:48:37.504 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.503+0000 7ff1059dc700 1 --2- 172.21.15.2:0/976924371 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff0ec0a11e0 0x7ff0ec0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:37.505 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.504+0000 7ff1059dc700 1 --2- 172.21.15.2:0/976924371 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff0ec0a11e0 0x7ff0ec0a3690 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0005e90 tx=0x7ff0f0019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:37.507 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.506+0000 7ff0f77fe700 1 -- 172.21.15.2:0/976924371 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff0fc0960f0 con 0x7ff1000ff330 2023-11-16T16:48:37.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.735+0000 7ff0f57fa700 1 -- 172.21.15.2:0/976924371 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7ff100066ea0 con 0x7ff1000ff330 2023-11-16T16:48:37.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.736+0000 7ff0f77fe700 1 -- 172.21.15.2:0/976924371 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff0fc099710 con 0x7ff1000ff330 2023-11-16T16:48:37.738 INFO:teuthology.orchestra.run.smithi002.stdout:197568495665 2023-11-16T16:48:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.738+0000 7ff0f57fa700 1 -- 172.21.15.2:0/976924371 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff0ec0a11e0 msgr2=0x7ff0ec0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.738+0000 7ff0f57fa700 1 --2- 172.21.15.2:0/976924371 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff0ec0a11e0 0x7ff0ec0a3690 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0005e90 tx=0x7ff0f0019040 comp rx=0 tx=0).stop 2023-11-16T16:48:37.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.738+0000 7ff0f57fa700 1 -- 172.21.15.2:0/976924371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1000ff330 msgr2=0x7ff1001a2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:37.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.738+0000 7ff0f57fa700 1 --2- 172.21.15.2:0/976924371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1000ff330 0x7ff1001a2820 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00e9a0 tx=0x7ff0fc00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:48:37.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.739+0000 7ff0f57fa700 1 -- 172.21.15.2:0/976924371 shutdown_connections 2023-11-16T16:48:37.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.739+0000 7ff0f57fa700 1 --2- 172.21.15.2:0/976924371 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff0ec0a11e0 0x7ff0ec0a3690 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:37.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.739+0000 7ff0f57fa700 1 --2- 172.21.15.2:0/976924371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff100104660 0x7ff10019c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:37.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.739+0000 7ff0f57fa700 1 --2- 172.21.15.2:0/976924371 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1000ff330 0x7ff1001a2820 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:37.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.739+0000 7ff0f57fa700 1 --2- 172.21.15.2:0/976924371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1000ffc40 0x7ff1001a2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:37.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.739+0000 7ff0f57fa700 1 -- 172.21.15.2:0/976924371 >> 172.21.15.2:0/976924371 conn(0x7ff1000770a0 msgr2=0x7ff100102210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:37.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.739+0000 7ff0f57fa700 1 -- 172.21.15.2:0/976924371 shutdown_connections 2023-11-16T16:48:37.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:37.739+0000 7ff0f57fa700 1 -- 172.21.15.2:0/976924371 wait complete. 2023-11-16T16:48:38.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.062+0000 7f628d3ac700 1 -- 172.21.15.2:0/604018424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6288100ee0 msgr2=0x7f6288104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:38.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.062+0000 7f628d3ac700 1 --2- 172.21.15.2:0/604018424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6288100ee0 0x7f6288104eb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f627c0099c0 tx=0x7f627c009cd0 comp rx=0 tx=0).stop 2023-11-16T16:48:38.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.062+0000 7f628d3ac700 1 -- 172.21.15.2:0/604018424 shutdown_connections 2023-11-16T16:48:38.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.062+0000 7f628d3ac700 1 --2- 172.21.15.2:0/604018424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62881055e0 0x7f628810d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:38.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.062+0000 7f628d3ac700 1 --2- 172.21.15.2:0/604018424 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6288100540 0x7f6288100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:38.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.062+0000 7f628d3ac700 1 --2- 172.21.15.2:0/604018424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6288100ee0 0x7f6288104eb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:38.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.062+0000 7f628d3ac700 1 -- 172.21.15.2:0/604018424 >> 172.21.15.2:0/604018424 conn(0x7f62880fbdc0 msgr2=0x7f62880fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:38.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.063+0000 7f628d3ac700 1 -- 172.21.15.2:0/604018424 shutdown_connections 2023-11-16T16:48:38.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.063+0000 7f628d3ac700 1 -- 172.21.15.2:0/604018424 wait complete. 2023-11-16T16:48:38.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.064+0000 7f628d3ac700 1 Processor -- start 2023-11-16T16:48:38.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.064+0000 7f628d3ac700 1 -- start start 2023-11-16T16:48:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.065+0000 7f628d3ac700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6288100540 0x7f62881963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.065+0000 7f628d3ac700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6288100ee0 0x7f6288196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.065+0000 7f628d3ac700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62881055e0 0x7f628819ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.065+0000 7f628d3ac700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628819b3b0 con 0x7f6288100ee0 2023-11-16T16:48:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.065+0000 7f628d3ac700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628819b520 con 0x7f6288100540 2023-11-16T16:48:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.065+0000 7f628d3ac700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628819b6e0 con 0x7f62881055e0 2023-11-16T16:48:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.065+0000 7f6286ffd700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6288100540 0x7f62881963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.065+0000 7f62867fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6288100ee0 0x7f6288196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.065+0000 7f6286ffd700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6288100540 0x7f62881963f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:56946/0 (socket says 172.21.15.2:56946) 2023-11-16T16:48:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.065+0000 7f6286ffd700 1 -- 172.21.15.2:0/2791259908 learned_addr learned my addr 172.21.15.2:0/2791259908 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:38.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.065+0000 7f62877fe700 1 --2- 172.21.15.2:0/2791259908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62881055e0 0x7f628819ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:38.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.066+0000 7f6286ffd700 1 -- 172.21.15.2:0/2791259908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62881055e0 msgr2=0x7f628819ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:38.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.066+0000 7f6286ffd700 1 --2- 172.21.15.2:0/2791259908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62881055e0 0x7f628819ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:38.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.066+0000 7f6286ffd700 1 -- 172.21.15.2:0/2791259908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6288100ee0 msgr2=0x7f6288196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:38.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.066+0000 7f6286ffd700 1 --2- 172.21.15.2:0/2791259908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6288100ee0 0x7f6288196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:38.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.066+0000 7f6286ffd700 1 -- 172.21.15.2:0/2791259908 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627c0095f0 con 0x7f6288100540 2023-11-16T16:48:38.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.066+0000 7f62867fc700 1 --2- 172.21.15.2:0/2791259908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6288100ee0 0x7f6288196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:38.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.066+0000 7f6286ffd700 1 --2- 172.21.15.2:0/2791259908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6288100540 0x7f62881963f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f627800e9a0 tx=0x7f627800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:38.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.066+0000 7f626ffff700 1 -- 172.21.15.2:0/2791259908 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6278009790 con 0x7f6288100540 2023-11-16T16:48:38.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.066+0000 7f628d3ac700 1 -- 172.21.15.2:0/2791259908 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6288110360 con 0x7f6288100540 2023-11-16T16:48:38.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.066+0000 7f626ffff700 1 -- 172.21.15.2:0/2791259908 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f627800cbd0 con 0x7f6288100540 2023-11-16T16:48:38.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.066+0000 7f626ffff700 1 -- 172.21.15.2:0/2791259908 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6278018680 con 0x7f6288100540 2023-11-16T16:48:38.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.066+0000 7f628d3ac700 1 -- 172.21.15.2:0/2791259908 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62881101d0 con 0x7f6288100540 2023-11-16T16:48:38.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.068+0000 7f628d3ac700 1 -- 172.21.15.2:0/2791259908 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62881104f0 con 0x7f6288100540 2023-11-16T16:48:38.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.069+0000 7f626ffff700 1 -- 172.21.15.2:0/2791259908 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f627800cd40 con 0x7f6288100540 2023-11-16T16:48:38.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.071+0000 7f626ffff700 1 --2- 172.21.15.2:0/2791259908 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62700a11e0 0x7f62700a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:38.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.071+0000 7f626ffff700 1 -- 172.21.15.2:0/2791259908 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6278014070 con 0x7f6288100540 2023-11-16T16:48:38.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.071+0000 7f62867fc700 1 --2- 172.21.15.2:0/2791259908 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62700a11e0 0x7f62700a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:38.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.072+0000 7f62867fc700 1 --2- 172.21.15.2:0/2791259908 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62700a11e0 0x7f62700a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f627c009fd0 tx=0x7f627c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:38.077 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.075+0000 7f626ffff700 1 -- 172.21.15.2:0/2791259908 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6278051730 con 0x7f6288100540 2023-11-16T16:48:38.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:37 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/976924371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-11-16T16:48:38.225 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:38.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:37 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/976924371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-11-16T16:48:38.312 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.310+0000 7f628d3ac700 1 -- 172.21.15.2:0/2791259908 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f6288066ea0 con 0x7f6288100540 2023-11-16T16:48:38.312 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.310+0000 7f626ffff700 1 -- 172.21.15.2:0/2791259908 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f62780962e0 con 0x7f6288100540 2023-11-16T16:48:38.312 INFO:teuthology.orchestra.run.smithi002.stdout:287762808868 2023-11-16T16:48:38.315 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.313+0000 7f628d3ac700 1 -- 172.21.15.2:0/2791259908 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62700a11e0 msgr2=0x7f62700a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:38.315 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.313+0000 7f628d3ac700 1 --2- 172.21.15.2:0/2791259908 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62700a11e0 0x7f62700a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f627c009fd0 tx=0x7f627c005fb0 comp rx=0 tx=0).stop 2023-11-16T16:48:38.315 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.313+0000 7f628d3ac700 1 -- 172.21.15.2:0/2791259908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6288100540 msgr2=0x7f62881963f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:38.315 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.313+0000 7f628d3ac700 1 --2- 172.21.15.2:0/2791259908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6288100540 0x7f62881963f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f627800e9a0 tx=0x7f627800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:48:38.315 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.313+0000 7f628d3ac700 1 -- 172.21.15.2:0/2791259908 shutdown_connections 2023-11-16T16:48:38.315 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.313+0000 7f628d3ac700 1 --2- 172.21.15.2:0/2791259908 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62700a11e0 0x7f62700a3690 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:38.315 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.313+0000 7f628d3ac700 1 --2- 172.21.15.2:0/2791259908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62881055e0 0x7f628819ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:38.315 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.313+0000 7f628d3ac700 1 --2- 172.21.15.2:0/2791259908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6288100540 0x7f62881963f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:38.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.313+0000 7f628d3ac700 1 --2- 172.21.15.2:0/2791259908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6288100ee0 0x7f6288196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:38.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.313+0000 7f628d3ac700 1 -- 172.21.15.2:0/2791259908 >> 172.21.15.2:0/2791259908 conn(0x7f62880fbdc0 msgr2=0x7f62880fc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:38.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.313+0000 7f628d3ac700 1 -- 172.21.15.2:0/2791259908 shutdown_connections 2023-11-16T16:48:38.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:38.313+0000 7f628d3ac700 1 -- 172.21.15.2:0/2791259908 wait complete. 2023-11-16T16:48:38.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:37 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/976924371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-11-16T16:48:38.622 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495663 got 197568495665 for osd.6 2023-11-16T16:48:38.623 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:39.010 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808866 got 287762808868 for osd.9 2023-11-16T16:48:39.010 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:39.076 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:38 smithi002 ceph-mon[128712]: pgmap v318: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-11-16T16:48:39.076 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:38 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2791259908' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-11-16T16:48:39.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:38 smithi141 ceph-mon[131184]: pgmap v318: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-11-16T16:48:39.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:38 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2791259908' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-11-16T16:48:39.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:38 smithi155 ceph-mon[131607]: pgmap v318: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-11-16T16:48:39.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:38 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2791259908' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-11-16T16:48:40.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:40 smithi002 ceph-mon[128712]: pgmap v319: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-11-16T16:48:40.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.738+0000 7f3f66c7f700 1 -- 172.21.15.2:0/3140412514 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f60100ee0 msgr2=0x7f3f60104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:40.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.738+0000 7f3f66c7f700 1 --2- 172.21.15.2:0/3140412514 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f60100ee0 0x7f3f60104eb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c0099c0 tx=0x7f3f4c009cd0 comp rx=0 tx=0).stop 2023-11-16T16:48:40.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.739+0000 7f3f66c7f700 1 -- 172.21.15.2:0/3140412514 shutdown_connections 2023-11-16T16:48:40.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.739+0000 7f3f66c7f700 1 --2- 172.21.15.2:0/3140412514 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f601055e0 0x7f3f6010d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.739+0000 7f3f66c7f700 1 --2- 172.21.15.2:0/3140412514 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f60100540 0x7f3f60100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.739+0000 7f3f66c7f700 1 --2- 172.21.15.2:0/3140412514 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f60100ee0 0x7f3f60104eb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.739+0000 7f3f66c7f700 1 -- 172.21.15.2:0/3140412514 >> 172.21.15.2:0/3140412514 conn(0x7f3f600fbdc0 msgr2=0x7f3f600fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.739+0000 7f3f66c7f700 1 -- 172.21.15.2:0/3140412514 shutdown_connections 2023-11-16T16:48:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.739+0000 7f3f66c7f700 1 -- 172.21.15.2:0/3140412514 wait complete. 2023-11-16T16:48:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.740+0000 7f3f66c7f700 1 Processor -- start 2023-11-16T16:48:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.741+0000 7f3f66c7f700 1 -- start start 2023-11-16T16:48:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.741+0000 7f3f66c7f700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f60100540 0x7f3f60072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.741+0000 7f3f66c7f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f60100ee0 0x7f3f60073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.741+0000 7f3f66c7f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f601055e0 0x7f3f600738b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.741+0000 7f3f66c7f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f601a4860 con 0x7f3f60100ee0 2023-11-16T16:48:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.741+0000 7f3f66c7f700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f601a49d0 con 0x7f3f60100540 2023-11-16T16:48:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.741+0000 7f3f66c7f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f601a4b90 con 0x7f3f601055e0 2023-11-16T16:48:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.742+0000 7f3f5ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f60100ee0 0x7f3f60073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.742+0000 7f3f5ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f60100ee0 0x7f3f60073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49816/0 (socket says 172.21.15.2:49816) 2023-11-16T16:48:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.742+0000 7f3f5ffff700 1 -- 172.21.15.2:0/2956224144 learned_addr learned my addr 172.21.15.2:0/2956224144 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.742+0000 7f3f64a1b700 1 --2- 172.21.15.2:0/2956224144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f60100540 0x7f3f60072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.742+0000 7f3f6521c700 1 --2- 172.21.15.2:0/2956224144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f601055e0 0x7f3f600738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.742+0000 7f3f5ffff700 1 -- 172.21.15.2:0/2956224144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f601055e0 msgr2=0x7f3f600738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.742+0000 7f3f5ffff700 1 --2- 172.21.15.2:0/2956224144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f601055e0 0x7f3f600738b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.742+0000 7f3f5ffff700 1 -- 172.21.15.2:0/2956224144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f60100540 msgr2=0x7f3f60072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.742+0000 7f3f5ffff700 1 --2- 172.21.15.2:0/2956224144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f60100540 0x7f3f60072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.742+0000 7f3f5ffff700 1 -- 172.21.15.2:0/2956224144 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f58009c40 con 0x7f3f60100ee0 2023-11-16T16:48:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.742+0000 7f3f6521c700 1 --2- 172.21.15.2:0/2956224144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f601055e0 0x7f3f600738b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:48:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.742+0000 7f3f5ffff700 1 --2- 172.21.15.2:0/2956224144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f60100ee0 0x7f3f60073330 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c0098e0 tx=0x7f3f4c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.743+0000 7f3f5dffb700 1 -- 172.21.15.2:0/2956224144 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c01d070 con 0x7f3f60100ee0 2023-11-16T16:48:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.743+0000 7f3f5dffb700 1 -- 172.21.15.2:0/2956224144 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f4c005510 con 0x7f3f60100ee0 2023-11-16T16:48:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.743+0000 7f3f66c7f700 1 -- 172.21.15.2:0/2956224144 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4c0095f0 con 0x7f3f60100ee0 2023-11-16T16:48:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.743+0000 7f3f5dffb700 1 -- 172.21.15.2:0/2956224144 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c004a00 con 0x7f3f60100ee0 2023-11-16T16:48:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.743+0000 7f3f66c7f700 1 -- 172.21.15.2:0/2956224144 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f60110360 con 0x7f3f60100ee0 2023-11-16T16:48:40.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.745+0000 7f3f66c7f700 1 -- 172.21.15.2:0/2956224144 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f601101d0 con 0x7f3f60100ee0 2023-11-16T16:48:40.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.746+0000 7f3f5dffb700 1 -- 172.21.15.2:0/2956224144 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f3f4c004be0 con 0x7f3f60100ee0 2023-11-16T16:48:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.747+0000 7f3f5dffb700 1 --2- 172.21.15.2:0/2956224144 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3f500a11e0 0x7f3f500a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.748+0000 7f3f5dffb700 1 -- 172.21.15.2:0/2956224144 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3f4c0cdb10 con 0x7f3f60100ee0 2023-11-16T16:48:40.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.748+0000 7f3f64a1b700 1 --2- 172.21.15.2:0/2956224144 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3f500a11e0 0x7f3f500a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:40.750 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.748+0000 7f3f64a1b700 1 --2- 172.21.15.2:0/2956224144 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3f500a11e0 0x7f3f500a3690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009c60 tx=0x7f3f54009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:40.759 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.758+0000 7f3f5dffb700 1 -- 172.21.15.2:0/2956224144 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3f4c096e80 con 0x7f3f60100ee0 2023-11-16T16:48:40.764 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:40 smithi141 ceph-mon[131184]: pgmap v319: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-11-16T16:48:40.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.836+0000 7f9ffee10700 1 -- 172.21.15.2:0/1680797285 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ff81055e0 msgr2=0x7f9ff810d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:40.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.836+0000 7f9ffee10700 1 --2- 172.21.15.2:0/1680797285 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ff81055e0 0x7f9ff810d160 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9ff000b0d0 tx=0x7f9ff000b3e0 comp rx=0 tx=0).stop 2023-11-16T16:48:40.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.837+0000 7f9ffee10700 1 -- 172.21.15.2:0/1680797285 shutdown_connections 2023-11-16T16:48:40.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.837+0000 7f9ffee10700 1 --2- 172.21.15.2:0/1680797285 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ff81055e0 0x7f9ff810d160 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:40.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.837+0000 7f9ffee10700 1 --2- 172.21.15.2:0/1680797285 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ff8100540 0x7f9ff8100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:40.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.837+0000 7f9ffee10700 1 --2- 172.21.15.2:0/1680797285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ff8100ee0 0x7f9ff8104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:40.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.837+0000 7f9ffee10700 1 -- 172.21.15.2:0/1680797285 >> 172.21.15.2:0/1680797285 conn(0x7f9ff80fbdc0 msgr2=0x7f9ff80fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:40.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.837+0000 7f9ffee10700 1 -- 172.21.15.2:0/1680797285 shutdown_connections 2023-11-16T16:48:40.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.837+0000 7f9ffee10700 1 -- 172.21.15.2:0/1680797285 wait complete. 2023-11-16T16:48:40.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.838+0000 7f9ffee10700 1 Processor -- start 2023-11-16T16:48:40.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.838+0000 7f9ffee10700 1 -- start start 2023-11-16T16:48:40.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.838+0000 7f9ffee10700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ff8100540 0x7f9ff8072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:40.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.838+0000 7f9ffee10700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ff8100ee0 0x7f9ff8073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:40.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.838+0000 7f9ffee10700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ff81055e0 0x7f9ff81a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:40.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.838+0000 7f9ffee10700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff8073c20 con 0x7f9ff81055e0 2023-11-16T16:48:40.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.838+0000 7f9ffee10700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff81a4c60 con 0x7f9ff8100540 2023-11-16T16:48:40.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.838+0000 7f9ffee10700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff81a4dd0 con 0x7f9ff8100ee0 2023-11-16T16:48:40.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.839+0000 7f9ffd3ad700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ff81055e0 0x7f9ff81a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:40.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.839+0000 7f9ffcbac700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ff8100540 0x7f9ff8072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:40.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.839+0000 7f9ffd3ad700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ff81055e0 0x7f9ff81a4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49832/0 (socket says 172.21.15.2:49832) 2023-11-16T16:48:40.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.839+0000 7f9ffd3ad700 1 -- 172.21.15.2:0/3742811146 learned_addr learned my addr 172.21.15.2:0/3742811146 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:40.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.839+0000 7f9ff7fff700 1 --2- 172.21.15.2:0/3742811146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ff8100ee0 0x7f9ff8073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:40.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.839+0000 7f9ffcbac700 1 -- 172.21.15.2:0/3742811146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ff8100ee0 msgr2=0x7f9ff8073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:40.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.839+0000 7f9ffcbac700 1 --2- 172.21.15.2:0/3742811146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ff8100ee0 0x7f9ff8073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:40.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.839+0000 7f9ffcbac700 1 -- 172.21.15.2:0/3742811146 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ff81055e0 msgr2=0x7f9ff81a4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:40.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.839+0000 7f9ffcbac700 1 --2- 172.21.15.2:0/3742811146 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ff81055e0 0x7f9ff81a4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:40.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.839+0000 7f9ffcbac700 1 -- 172.21.15.2:0/3742811146 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec009520 con 0x7f9ff8100540 2023-11-16T16:48:40.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.840+0000 7f9ffcbac700 1 --2- 172.21.15.2:0/3742811146 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ff8100540 0x7f9ff8072df0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9fec00e9f0 tx=0x7f9fec00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:40.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.840+0000 7f9ff5ffb700 1 -- 172.21.15.2:0/3742811146 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec00cb70 con 0x7f9ff8100540 2023-11-16T16:48:40.843 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.840+0000 7f9ff5ffb700 1 -- 172.21.15.2:0/3742811146 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fec00ee90 con 0x7f9ff8100540 2023-11-16T16:48:40.843 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.840+0000 7f9ffee10700 1 -- 172.21.15.2:0/3742811146 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff0009d00 con 0x7f9ff8100540 2023-11-16T16:48:40.843 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.840+0000 7f9ff5ffb700 1 -- 172.21.15.2:0/3742811146 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec003cf0 con 0x7f9ff8100540 2023-11-16T16:48:40.843 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.840+0000 7f9ffee10700 1 -- 172.21.15.2:0/3742811146 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff8110370 con 0x7f9ff8100540 2023-11-16T16:48:40.843 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.842+0000 7f9ffee10700 1 -- 172.21.15.2:0/3742811146 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff81101e0 con 0x7f9ff8100540 2023-11-16T16:48:40.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.843+0000 7f9ff5ffb700 1 -- 172.21.15.2:0/3742811146 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f9fec00ccd0 con 0x7f9ff8100540 2023-11-16T16:48:40.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.844+0000 7f9ff5ffb700 1 --2- 172.21.15.2:0/3742811146 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9fe80a11e0 0x7f9fe80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:40.846 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.845+0000 7f9ff5ffb700 1 -- 172.21.15.2:0/3742811146 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9fec014070 con 0x7f9ff8100540 2023-11-16T16:48:40.847 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.845+0000 7f9ff7fff700 1 --2- 172.21.15.2:0/3742811146 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9fe80a11e0 0x7f9fe80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:40.847 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.845+0000 7f9ff7fff700 1 --2- 172.21.15.2:0/3742811146 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9fe80a11e0 0x7f9fe80a3690 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9ff80745e0 tx=0x7f9fe4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:40.850 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:40.848+0000 7f9ff5ffb700 1 -- 172.21.15.2:0/3742811146 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9fec096340 con 0x7f9ff8100540 2023-11-16T16:48:40.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:40 smithi155 ceph-mon[131607]: pgmap v319: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-11-16T16:48:41.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.025+0000 7f3f66c7f700 1 -- 172.21.15.2:0/2956224144 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f3f601104f0 con 0x7f3f60100ee0 2023-11-16T16:48:41.028 INFO:teuthology.orchestra.run.smithi002.stdout:163208757304 2023-11-16T16:48:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.026+0000 7f3f5dffb700 1 -- 172.21.15.2:0/2956224144 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f3f4c09a4a0 con 0x7f3f60100ee0 2023-11-16T16:48:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.029+0000 7f3f66c7f700 1 -- 172.21.15.2:0/2956224144 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3f500a11e0 msgr2=0x7f3f500a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.029+0000 7f3f66c7f700 1 --2- 172.21.15.2:0/2956224144 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3f500a11e0 0x7f3f500a3690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009c60 tx=0x7f3f54009260 comp rx=0 tx=0).stop 2023-11-16T16:48:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.029+0000 7f3f66c7f700 1 -- 172.21.15.2:0/2956224144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f60100ee0 msgr2=0x7f3f60073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.029+0000 7f3f66c7f700 1 --2- 172.21.15.2:0/2956224144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f60100ee0 0x7f3f60073330 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c0098e0 tx=0x7f3f4c00f690 comp rx=0 tx=0).stop 2023-11-16T16:48:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.029+0000 7f3f66c7f700 1 -- 172.21.15.2:0/2956224144 shutdown_connections 2023-11-16T16:48:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.029+0000 7f3f66c7f700 1 --2- 172.21.15.2:0/2956224144 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f3f500a11e0 0x7f3f500a3690 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.029+0000 7f3f66c7f700 1 --2- 172.21.15.2:0/2956224144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f601055e0 0x7f3f600738b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.029+0000 7f3f66c7f700 1 --2- 172.21.15.2:0/2956224144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f60100540 0x7f3f60072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.029+0000 7f3f66c7f700 1 --2- 172.21.15.2:0/2956224144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f60100ee0 0x7f3f60073330 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:41.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.029+0000 7f3f66c7f700 1 -- 172.21.15.2:0/2956224144 >> 172.21.15.2:0/2956224144 conn(0x7f3f600fbdc0 msgr2=0x7f3f600fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:41.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.029+0000 7f3f66c7f700 1 -- 172.21.15.2:0/2956224144 shutdown_connections 2023-11-16T16:48:41.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.029+0000 7f3f66c7f700 1 -- 172.21.15.2:0/2956224144 wait complete. 2023-11-16T16:48:41.106 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.105+0000 7f9ffee10700 1 -- 172.21.15.2:0/3742811146 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f9ff8110500 con 0x7f9ff8100540 2023-11-16T16:48:41.107 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.105+0000 7f9ff5ffb700 1 -- 172.21.15.2:0/3742811146 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9fec099960 con 0x7f9ff8100540 2023-11-16T16:48:41.107 INFO:teuthology.orchestra.run.smithi002.stdout:141733920828 2023-11-16T16:48:41.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.108+0000 7f9ffee10700 1 -- 172.21.15.2:0/3742811146 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9fe80a11e0 msgr2=0x7f9fe80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:41.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.108+0000 7f9ffee10700 1 --2- 172.21.15.2:0/3742811146 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9fe80a11e0 0x7f9fe80a3690 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9ff80745e0 tx=0x7f9fe4009260 comp rx=0 tx=0).stop 2023-11-16T16:48:41.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.108+0000 7f9ffee10700 1 -- 172.21.15.2:0/3742811146 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ff8100540 msgr2=0x7f9ff8072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:41.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.108+0000 7f9ffee10700 1 --2- 172.21.15.2:0/3742811146 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ff8100540 0x7f9ff8072df0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9fec00e9f0 tx=0x7f9fec00ed00 comp rx=0 tx=0).stop 2023-11-16T16:48:41.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.108+0000 7f9ffee10700 1 -- 172.21.15.2:0/3742811146 shutdown_connections 2023-11-16T16:48:41.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.108+0000 7f9ffee10700 1 --2- 172.21.15.2:0/3742811146 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9fe80a11e0 0x7f9fe80a3690 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:41.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.108+0000 7f9ffee10700 1 --2- 172.21.15.2:0/3742811146 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ff81055e0 0x7f9ff81a4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:41.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.108+0000 7f9ffee10700 1 --2- 172.21.15.2:0/3742811146 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9ff8100540 0x7f9ff8072df0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:41.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.108+0000 7f9ffee10700 1 --2- 172.21.15.2:0/3742811146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ff8100ee0 0x7f9ff8073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:41.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.108+0000 7f9ffee10700 1 -- 172.21.15.2:0/3742811146 >> 172.21.15.2:0/3742811146 conn(0x7f9ff80fbdc0 msgr2=0x7f9ff80fc720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:41.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.108+0000 7f9ffee10700 1 -- 172.21.15.2:0/3742811146 shutdown_connections 2023-11-16T16:48:41.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.108+0000 7f9ffee10700 1 -- 172.21.15.2:0/3742811146 wait complete. 2023-11-16T16:48:41.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:41 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2956224144' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-11-16T16:48:41.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:41 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3742811146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-11-16T16:48:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.661+0000 7f088ac69700 1 -- 172.21.15.2:0/905662511 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0884100540 msgr2=0x7f0884100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.661+0000 7f088ac69700 1 --2- 172.21.15.2:0/905662511 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0884100540 0x7f0884100910 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f08780099c0 tx=0x7f0878009cd0 comp rx=0 tx=0).stop 2023-11-16T16:48:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.661+0000 7f088ac69700 1 -- 172.21.15.2:0/905662511 shutdown_connections 2023-11-16T16:48:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.661+0000 7f088ac69700 1 --2- 172.21.15.2:0/905662511 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08841055e0 0x7f088410d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.661+0000 7f088ac69700 1 --2- 172.21.15.2:0/905662511 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0884100540 0x7f0884100910 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.661+0000 7f088ac69700 1 --2- 172.21.15.2:0/905662511 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0884100ee0 0x7f0884104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.661+0000 7f088ac69700 1 -- 172.21.15.2:0/905662511 >> 172.21.15.2:0/905662511 conn(0x7f08840fbdc0 msgr2=0x7f08840fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.661+0000 7f088ac69700 1 -- 172.21.15.2:0/905662511 shutdown_connections 2023-11-16T16:48:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.661+0000 7f088ac69700 1 -- 172.21.15.2:0/905662511 wait complete. 2023-11-16T16:48:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.662+0000 7f088ac69700 1 Processor -- start 2023-11-16T16:48:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.663+0000 7f088ac69700 1 -- start start 2023-11-16T16:48:41.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.663+0000 7f088ac69700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0884100540 0x7f08841985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:41.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.663+0000 7f088ac69700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0884100ee0 0x7f0884198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:41.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.663+0000 7f088ac69700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08841055e0 0x7f088419cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:41.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.663+0000 7f088ac69700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088419d5b0 con 0x7f0884100540 2023-11-16T16:48:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.663+0000 7f088ac69700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088419d720 con 0x7f0884100ee0 2023-11-16T16:48:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.663+0000 7f088ac69700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088419d8e0 con 0x7f08841055e0 2023-11-16T16:48:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.663+0000 7f0888a05700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0884100540 0x7f08841985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.663+0000 7f0888a05700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0884100540 0x7f08841985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49854/0 (socket says 172.21.15.2:49854) 2023-11-16T16:48:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.663+0000 7f0888a05700 1 -- 172.21.15.2:0/1912303899 learned_addr learned my addr 172.21.15.2:0/1912303899 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.663+0000 7f0889206700 1 --2- 172.21.15.2:0/1912303899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08841055e0 0x7f088419cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.663+0000 7f0883fff700 1 --2- 172.21.15.2:0/1912303899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0884100ee0 0x7f0884198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.664+0000 7f0888a05700 1 -- 172.21.15.2:0/1912303899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08841055e0 msgr2=0x7f088419cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.664+0000 7f0888a05700 1 --2- 172.21.15.2:0/1912303899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08841055e0 0x7f088419cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.664+0000 7f0888a05700 1 -- 172.21.15.2:0/1912303899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0884100ee0 msgr2=0x7f0884198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.664+0000 7f0888a05700 1 --2- 172.21.15.2:0/1912303899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0884100ee0 0x7f0884198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:41.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.664+0000 7f0888a05700 1 -- 172.21.15.2:0/1912303899 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08780095f0 con 0x7f0884100540 2023-11-16T16:48:41.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.664+0000 7f0889206700 1 --2- 172.21.15.2:0/1912303899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08841055e0 0x7f088419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:48:41.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.664+0000 7f0883fff700 1 --2- 172.21.15.2:0/1912303899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0884100ee0 0x7f0884198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:48:41.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.664+0000 7f0888a05700 1 --2- 172.21.15.2:0/1912303899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0884100540 0x7f08841985f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f087800f690 tx=0x7f087800f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:41.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.664+0000 7f0881ffb700 1 -- 172.21.15.2:0/1912303899 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087801c070 con 0x7f0884100540 2023-11-16T16:48:41.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.664+0000 7f088ac69700 1 -- 172.21.15.2:0/1912303899 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0884110340 con 0x7f0884100540 2023-11-16T16:48:41.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.664+0000 7f0881ffb700 1 -- 172.21.15.2:0/1912303899 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0878022420 con 0x7f0884100540 2023-11-16T16:48:41.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.664+0000 7f0881ffb700 1 -- 172.21.15.2:0/1912303899 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0878017650 con 0x7f0884100540 2023-11-16T16:48:41.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.664+0000 7f088ac69700 1 -- 172.21.15.2:0/1912303899 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08841101b0 con 0x7f0884100540 2023-11-16T16:48:41.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.666+0000 7f088ac69700 1 -- 172.21.15.2:0/1912303899 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08841104d0 con 0x7f0884100540 2023-11-16T16:48:41.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.667+0000 7f0881ffb700 1 -- 172.21.15.2:0/1912303899 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f0878020070 con 0x7f0884100540 2023-11-16T16:48:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.669+0000 7f0881ffb700 1 --2- 172.21.15.2:0/1912303899 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08740a11e0 0x7f08740a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.669+0000 7f0881ffb700 1 -- 172.21.15.2:0/1912303899 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f08780cdd00 con 0x7f0884100540 2023-11-16T16:48:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.669+0000 7f0883fff700 1 --2- 172.21.15.2:0/1912303899 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08740a11e0 0x7f08740a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.670+0000 7f0883fff700 1 --2- 172.21.15.2:0/1912303899 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08740a11e0 0x7f08740a3690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0884199c10 tx=0x7f0870009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:41.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.672+0000 7f0881ffb700 1 -- 172.21.15.2:0/1912303899 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f08780971c0 con 0x7f0884100540 2023-11-16T16:48:41.764 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:41 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2956224144' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-11-16T16:48:41.764 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:41 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3742811146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-11-16T16:48:41.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:41 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2956224144' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-11-16T16:48:41.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:41 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3742811146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-11-16T16:48:41.887 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.886+0000 7f088ac69700 1 -- 172.21.15.2:0/1912303899 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f0884066ea0 con 0x7f0884100540 2023-11-16T16:48:41.888 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.886+0000 7f0881ffb700 1 -- 172.21.15.2:0/1912303899 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0878027020 con 0x7f0884100540 2023-11-16T16:48:41.888 INFO:teuthology.orchestra.run.smithi002.stdout:219043332141 2023-11-16T16:48:41.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.889+0000 7f088ac69700 1 -- 172.21.15.2:0/1912303899 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08740a11e0 msgr2=0x7f08740a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:41.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.889+0000 7f088ac69700 1 --2- 172.21.15.2:0/1912303899 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08740a11e0 0x7f08740a3690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0884199c10 tx=0x7f0870009260 comp rx=0 tx=0).stop 2023-11-16T16:48:41.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.889+0000 7f088ac69700 1 -- 172.21.15.2:0/1912303899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0884100540 msgr2=0x7f08841985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:41.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.889+0000 7f088ac69700 1 --2- 172.21.15.2:0/1912303899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0884100540 0x7f08841985f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f087800f690 tx=0x7f087800f770 comp rx=0 tx=0).stop 2023-11-16T16:48:41.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.889+0000 7f088ac69700 1 -- 172.21.15.2:0/1912303899 shutdown_connections 2023-11-16T16:48:41.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.889+0000 7f088ac69700 1 --2- 172.21.15.2:0/1912303899 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08740a11e0 0x7f08740a3690 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:41.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.889+0000 7f088ac69700 1 --2- 172.21.15.2:0/1912303899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08841055e0 0x7f088419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:41.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.889+0000 7f088ac69700 1 --2- 172.21.15.2:0/1912303899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0884100540 0x7f08841985f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:41.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.889+0000 7f088ac69700 1 --2- 172.21.15.2:0/1912303899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0884100ee0 0x7f0884198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:41.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.889+0000 7f088ac69700 1 -- 172.21.15.2:0/1912303899 >> 172.21.15.2:0/1912303899 conn(0x7f08840fbdc0 msgr2=0x7f08840fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:41.891 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.889+0000 7f088ac69700 1 -- 172.21.15.2:0/1912303899 shutdown_connections 2023-11-16T16:48:41.892 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:41.889+0000 7f088ac69700 1 -- 172.21.15.2:0/1912303899 wait complete. 2023-11-16T16:48:41.893 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757301 got 163208757304 for osd.5 2023-11-16T16:48:41.893 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:41.974 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920825 got 141733920828 for osd.4 2023-11-16T16:48:41.974 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:42.398 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332138 got 219043332141 for osd.7 2023-11-16T16:48:42.398 DEBUG:teuthology.parallel:result is None 2023-11-16T16:48:42.398 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-11-16T16:48:42.399 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph pg dump --format=json 2023-11-16T16:48:42.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:42 smithi002 ceph-mon[128712]: pgmap v320: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T16:48:42.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:42 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1912303899' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-11-16T16:48:42.764 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:42 smithi141 ceph-mon[131184]: pgmap v320: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T16:48:42.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:42 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1912303899' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-11-16T16:48:42.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:42 smithi155 ceph-mon[131607]: pgmap v320: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T16:48:42.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:42 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1912303899' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-11-16T16:48:43.286 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:44.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:44.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:44.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:44 smithi141 ceph-mon[131184]: pgmap v321: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:48:44.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2023-11-16T16:48:44.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2023-11-16T16:48:44.768 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2023-11-16T16:48:44.768 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:44 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2023-11-16T16:48:44.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:44.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:44.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:44 smithi155 ceph-mon[131607]: pgmap v321: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:48:44.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2023-11-16T16:48:44.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2023-11-16T16:48:44.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2023-11-16T16:48:44.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:44 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2023-11-16T16:48:44.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.869+0000 7f2549ac6700 1 -- 172.21.15.2:0/1939226842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25441055e0 msgr2=0x7f254410d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:44.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.869+0000 7f2549ac6700 1 --2- 172.21.15.2:0/1939226842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25441055e0 0x7f254410d160 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f253c00b0d0 tx=0x7f253c00b3e0 comp rx=0 tx=0).stop 2023-11-16T16:48:44.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.869+0000 7f2549ac6700 1 -- 172.21.15.2:0/1939226842 shutdown_connections 2023-11-16T16:48:44.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.869+0000 7f2549ac6700 1 --2- 172.21.15.2:0/1939226842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25441055e0 0x7f254410d160 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:44.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.869+0000 7f2549ac6700 1 --2- 172.21.15.2:0/1939226842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2544100540 0x7f2544100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:44.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.869+0000 7f2549ac6700 1 --2- 172.21.15.2:0/1939226842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2544100ee0 0x7f2544104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:44.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.869+0000 7f2549ac6700 1 -- 172.21.15.2:0/1939226842 >> 172.21.15.2:0/1939226842 conn(0x7f25440fbdc0 msgr2=0x7f25440fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:44.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.869+0000 7f2549ac6700 1 -- 172.21.15.2:0/1939226842 shutdown_connections 2023-11-16T16:48:44.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.869+0000 7f2549ac6700 1 -- 172.21.15.2:0/1939226842 wait complete. 2023-11-16T16:48:44.872 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.870+0000 7f2549ac6700 1 Processor -- start 2023-11-16T16:48:44.872 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.871+0000 7f2549ac6700 1 -- start start 2023-11-16T16:48:44.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.871+0000 7f2549ac6700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2544100540 0x7f25441964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:44.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.871+0000 7f2549ac6700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2544100ee0 0x7f2544196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:44.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.871+0000 7f2549ac6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25441055e0 0x7f254419aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:44.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.871+0000 7f2549ac6700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254419b4b0 con 0x7f2544100540 2023-11-16T16:48:44.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.871+0000 7f2549ac6700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254419b620 con 0x7f2544100ee0 2023-11-16T16:48:44.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.871+0000 7f2549ac6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254419b7e0 con 0x7f25441055e0 2023-11-16T16:48:44.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.871+0000 7f25437fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2544100540 0x7f25441964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:44.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.871+0000 7f2542ffd700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2544100ee0 0x7f2544196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:44.875 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.871+0000 7f2543fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25441055e0 0x7f254419aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:44.875 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.871+0000 7f2542ffd700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2544100ee0 0x7f2544196a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:47464/0 (socket says 172.21.15.2:47464) 2023-11-16T16:48:44.875 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.871+0000 7f2542ffd700 1 -- 172.21.15.2:0/2582592980 learned_addr learned my addr 172.21.15.2:0/2582592980 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:44.875 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.872+0000 7f25437fe700 1 -- 172.21.15.2:0/2582592980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25441055e0 msgr2=0x7f254419aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:44.875 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.872+0000 7f25437fe700 1 --2- 172.21.15.2:0/2582592980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25441055e0 0x7f254419aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:44.875 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.872+0000 7f25437fe700 1 -- 172.21.15.2:0/2582592980 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2544100ee0 msgr2=0x7f2544196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:44.875 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.872+0000 7f25437fe700 1 --2- 172.21.15.2:0/2582592980 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2544100ee0 0x7f2544196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:44.875 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.872+0000 7f25437fe700 1 -- 172.21.15.2:0/2582592980 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2538009520 con 0x7f2544100540 2023-11-16T16:48:44.875 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.872+0000 7f25437fe700 1 --2- 172.21.15.2:0/2582592980 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2544100540 0x7f25441964f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f253400e9f0 tx=0x7f253400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:44.875 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.872+0000 7f2540ff9700 1 -- 172.21.15.2:0/2582592980 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253400c9c0 con 0x7f2544100540 2023-11-16T16:48:44.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.872+0000 7f2549ac6700 1 -- 172.21.15.2:0/2582592980 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f253c009d00 con 0x7f2544100540 2023-11-16T16:48:44.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.872+0000 7f2540ff9700 1 -- 172.21.15.2:0/2582592980 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2534010430 con 0x7f2544100540 2023-11-16T16:48:44.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.872+0000 7f2540ff9700 1 -- 172.21.15.2:0/2582592980 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2534018680 con 0x7f2544100540 2023-11-16T16:48:44.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.873+0000 7f2549ac6700 1 -- 172.21.15.2:0/2582592980 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2544110320 con 0x7f2544100540 2023-11-16T16:48:44.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.874+0000 7f2549ac6700 1 -- 172.21.15.2:0/2582592980 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2544110190 con 0x7f2544100540 2023-11-16T16:48:44.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.876+0000 7f2540ff9700 1 -- 172.21.15.2:0/2582592980 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f2534010860 con 0x7f2544100540 2023-11-16T16:48:44.879 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.877+0000 7f2540ff9700 1 --2- 172.21.15.2:0/2582592980 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f252c0a11e0 0x7f252c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:44.879 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.877+0000 7f2540ff9700 1 -- 172.21.15.2:0/2582592980 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f2534014070 con 0x7f2544100540 2023-11-16T16:48:44.879 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.878+0000 7f2542ffd700 1 --2- 172.21.15.2:0/2582592980 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f252c0a11e0 0x7f252c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:44.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.878+0000 7f2542ffd700 1 --2- 172.21.15.2:0/2582592980 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f252c0a11e0 0x7f252c0a3690 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2544197b10 tx=0x7f2538009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:44.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:44.879+0000 7f2540ff9700 1 -- 172.21.15.2:0/2582592980 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2534096370 con 0x7f2544100540 2023-11-16T16:48:44.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:44.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:44.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:44 smithi002 ceph-mon[128712]: pgmap v321: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:48:44.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2023-11-16T16:48:44.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2023-11-16T16:48:44.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2023-11-16T16:48:44.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:44 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2023-11-16T16:48:45.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:45.085+0000 7f2549ac6700 1 -- 172.21.15.2:0/2582592980 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f25441104b0 con 0x7f252c0a11e0 2023-11-16T16:48:45.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:45.099+0000 7f2540ff9700 1 -- 172.21.15.2:0/2582592980 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427319 (secure 0 0 0) 0x7f25441104b0 con 0x7f252c0a11e0 2023-11-16T16:48:45.107 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:48:45.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:45.104+0000 7f2549ac6700 1 -- 172.21.15.2:0/2582592980 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f252c0a11e0 msgr2=0x7f252c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:45.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:45.104+0000 7f2549ac6700 1 --2- 172.21.15.2:0/2582592980 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f252c0a11e0 0x7f252c0a3690 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2544197b10 tx=0x7f2538009260 comp rx=0 tx=0).stop 2023-11-16T16:48:45.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:45.104+0000 7f2549ac6700 1 -- 172.21.15.2:0/2582592980 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2544100540 msgr2=0x7f25441964f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:45.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:45.104+0000 7f2549ac6700 1 --2- 172.21.15.2:0/2582592980 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2544100540 0x7f25441964f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f253400e9f0 tx=0x7f253400ed00 comp rx=0 tx=0).stop 2023-11-16T16:48:45.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:45.104+0000 7f2549ac6700 1 -- 172.21.15.2:0/2582592980 shutdown_connections 2023-11-16T16:48:45.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:45.104+0000 7f2549ac6700 1 --2- 172.21.15.2:0/2582592980 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f252c0a11e0 0x7f252c0a3690 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:45.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:45.104+0000 7f2549ac6700 1 --2- 172.21.15.2:0/2582592980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25441055e0 0x7f254419aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:45.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:45.104+0000 7f2549ac6700 1 --2- 172.21.15.2:0/2582592980 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2544100540 0x7f25441964f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:45.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:45.104+0000 7f2549ac6700 1 --2- 172.21.15.2:0/2582592980 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2544100ee0 0x7f2544196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:45.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:45.104+0000 7f2549ac6700 1 -- 172.21.15.2:0/2582592980 >> 172.21.15.2:0/2582592980 conn(0x7f25440fbdc0 msgr2=0x7f25440fc720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:45.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:45.105+0000 7f2549ac6700 1 -- 172.21.15.2:0/2582592980 shutdown_connections 2023-11-16T16:48:45.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:45.105+0000 7f2549ac6700 1 -- 172.21.15.2:0/2582592980 wait complete. 2023-11-16T16:48:45.112 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-16T16:48:45.693 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2023-11-16T16:48:44.492205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":671591,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":474,"num_read_kb":372,"num_write":349,"num_write_kb":3074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":284,"ondisk_log_size":284,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":81588,"kb_used_data":12180,"kb_used_omap":0,"kb_used_meta":69312,"kb_avail":1124860236,"statfs":{"total":1151940427776,"available":1151856881664,"internally_reserved":0,"allocated":12472320,"data_stored":7992536,"data_compressed":320145,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":0,"internal_metadata":70975488},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":84,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.687923"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472424+0000","last_change":"2023-11-16T16:47:26.367598+0000","last_active":"2023-11-16T16:47:45.472424+0000","last_peered":"2023-11-16T16:47:45.472424+0000","last_clean":"2023-11-16T16:47:45.472424+0000","last_became_active":"2023-11-16T16:47:26.366233+0000","last_became_peered":"2023-11-16T16:47:26.366233+0000","last_unstale":"2023-11-16T16:47:45.472424+0000","last_undegraded":"2023-11-16T16:47:45.472424+0000","last_fullsized":"2023-11-16T16:47:45.472424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:27:47.751645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.861310+0000","last_change":"2023-11-16T16:47:26.371984+0000","last_active":"2023-11-16T16:47:44.861310+0000","last_peered":"2023-11-16T16:47:44.861310+0000","last_clean":"2023-11-16T16:47:44.861310+0000","last_became_active":"2023-11-16T16:47:26.371628+0000","last_became_peered":"2023-11-16T16:47:26.371628+0000","last_unstale":"2023-11-16T16:47:44.861310+0000","last_undegraded":"2023-11-16T16:47:44.861310+0000","last_fullsized":"2023-11-16T16:47:44.861310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:07:37.389666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.860205+0000","last_change":"2023-11-16T16:47:26.374109+0000","last_active":"2023-11-16T16:47:44.860205+0000","last_peered":"2023-11-16T16:47:44.860205+0000","last_clean":"2023-11-16T16:47:44.860205+0000","last_became_active":"2023-11-16T16:47:26.373759+0000","last_became_peered":"2023-11-16T16:47:26.373759+0000","last_unstale":"2023-11-16T16:47:44.860205+0000","last_undegraded":"2023-11-16T16:47:44.860205+0000","last_fullsized":"2023-11-16T16:47:44.860205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:38:53.188578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855484+0000","last_change":"2023-11-16T16:47:26.378404+0000","last_active":"2023-11-16T16:47:44.855484+0000","last_peered":"2023-11-16T16:47:44.855484+0000","last_clean":"2023-11-16T16:47:44.855484+0000","last_became_active":"2023-11-16T16:47:26.378067+0000","last_became_peered":"2023-11-16T16:47:26.378067+0000","last_unstale":"2023-11-16T16:47:44.855484+0000","last_undegraded":"2023-11-16T16:47:44.855484+0000","last_fullsized":"2023-11-16T16:47:44.855484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:58:13.078678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856409+0000","last_change":"2023-11-16T16:47:26.370066+0000","last_active":"2023-11-16T16:47:44.856409+0000","last_peered":"2023-11-16T16:47:44.856409+0000","last_clean":"2023-11-16T16:47:44.856409+0000","last_became_active":"2023-11-16T16:47:26.369789+0000","last_became_peered":"2023-11-16T16:47:26.369789+0000","last_unstale":"2023-11-16T16:47:44.856409+0000","last_undegraded":"2023-11-16T16:47:44.856409+0000","last_fullsized":"2023-11-16T16:47:44.856409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:33:38.139992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472727+0000","last_change":"2023-11-16T16:47:26.379403+0000","last_active":"2023-11-16T16:47:45.472727+0000","last_peered":"2023-11-16T16:47:45.472727+0000","last_clean":"2023-11-16T16:47:45.472727+0000","last_became_active":"2023-11-16T16:47:26.379086+0000","last_became_peered":"2023-11-16T16:47:26.379086+0000","last_unstale":"2023-11-16T16:47:45.472727+0000","last_undegraded":"2023-11-16T16:47:45.472727+0000","last_fullsized":"2023-11-16T16:47:45.472727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:20:51.896082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857929+0000","last_change":"2023-11-16T16:47:26.377555+0000","last_active":"2023-11-16T16:47:44.857929+0000","last_peered":"2023-11-16T16:47:44.857929+0000","last_clean":"2023-11-16T16:47:44.857929+0000","last_became_active":"2023-11-16T16:47:26.377274+0000","last_became_peered":"2023-11-16T16:47:26.377274+0000","last_unstale":"2023-11-16T16:47:44.857929+0000","last_undegraded":"2023-11-16T16:47:44.857929+0000","last_fullsized":"2023-11-16T16:47:44.857929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:40:11.254785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.412630+0000","last_change":"2023-11-16T16:47:26.380842+0000","last_active":"2023-11-16T16:47:45.412630+0000","last_peered":"2023-11-16T16:47:45.412630+0000","last_clean":"2023-11-16T16:47:45.412630+0000","last_became_active":"2023-11-16T16:47:26.380527+0000","last_became_peered":"2023-11-16T16:47:26.380527+0000","last_unstale":"2023-11-16T16:47:45.412630+0000","last_undegraded":"2023-11-16T16:47:45.412630+0000","last_fullsized":"2023-11-16T16:47:45.412630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:56:19.663855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858099+0000","last_change":"2023-11-16T16:47:26.371416+0000","last_active":"2023-11-16T16:47:44.858099+0000","last_peered":"2023-11-16T16:47:44.858099+0000","last_clean":"2023-11-16T16:47:44.858099+0000","last_became_active":"2023-11-16T16:47:26.371096+0000","last_became_peered":"2023-11-16T16:47:26.371096+0000","last_unstale":"2023-11-16T16:47:44.858099+0000","last_undegraded":"2023-11-16T16:47:44.858099+0000","last_fullsized":"2023-11-16T16:47:44.858099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:35:21.284705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473525+0000","last_change":"2023-11-16T16:47:26.380828+0000","last_active":"2023-11-16T16:47:45.473525+0000","last_peered":"2023-11-16T16:47:45.473525+0000","last_clean":"2023-11-16T16:47:45.473525+0000","last_became_active":"2023-11-16T16:47:26.377495+0000","last_became_peered":"2023-11-16T16:47:26.377495+0000","last_unstale":"2023-11-16T16:47:45.473525+0000","last_undegraded":"2023-11-16T16:47:45.473525+0000","last_fullsized":"2023-11-16T16:47:45.473525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:58:53.606878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.859142+0000","last_change":"2023-11-16T16:47:26.374127+0000","last_active":"2023-11-16T16:47:44.859142+0000","last_peered":"2023-11-16T16:47:44.859142+0000","last_clean":"2023-11-16T16:47:44.859142+0000","last_became_active":"2023-11-16T16:47:26.373794+0000","last_became_peered":"2023-11-16T16:47:26.373794+0000","last_unstale":"2023-11-16T16:47:44.859142+0000","last_undegraded":"2023-11-16T16:47:44.859142+0000","last_fullsized":"2023-11-16T16:47:44.859142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:39:54.776347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855500+0000","last_change":"2023-11-16T16:47:26.382530+0000","last_active":"2023-11-16T16:47:44.855500+0000","last_peered":"2023-11-16T16:47:44.855500+0000","last_clean":"2023-11-16T16:47:44.855500+0000","last_became_active":"2023-11-16T16:47:26.382229+0000","last_became_peered":"2023-11-16T16:47:26.382229+0000","last_unstale":"2023-11-16T16:47:44.855500+0000","last_undegraded":"2023-11-16T16:47:44.855500+0000","last_fullsized":"2023-11-16T16:47:44.855500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:28:54.780902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473174+0000","last_change":"2023-11-16T16:47:45.473174+0000","last_active":"2023-11-16T16:47:45.473174+0000","last_peered":"2023-11-16T16:47:45.473174+0000","last_clean":"2023-11-16T16:47:45.473174+0000","last_became_active":"2023-11-16T16:47:45.472853+0000","last_became_peered":"2023-11-16T16:47:45.472853+0000","last_unstale":"2023-11-16T16:47:45.473174+0000","last_undegraded":"2023-11-16T16:47:45.473174+0000","last_fullsized":"2023-11-16T16:47:45.473174+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:57:05.934853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.859123+0000","last_change":"2023-11-16T16:47:26.375546+0000","last_active":"2023-11-16T16:47:44.859123+0000","last_peered":"2023-11-16T16:47:44.859123+0000","last_clean":"2023-11-16T16:47:44.859123+0000","last_became_active":"2023-11-16T16:47:26.375218+0000","last_became_peered":"2023-11-16T16:47:26.375218+0000","last_unstale":"2023-11-16T16:47:44.859123+0000","last_undegraded":"2023-11-16T16:47:44.859123+0000","last_fullsized":"2023-11-16T16:47:44.859123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:27:43.795280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.860204+0000","last_change":"2023-11-16T16:47:44.860204+0000","last_active":"2023-11-16T16:47:44.860204+0000","last_peered":"2023-11-16T16:47:44.860204+0000","last_clean":"2023-11-16T16:47:44.860204+0000","last_became_active":"2023-11-16T16:47:44.859473+0000","last_became_peered":"2023-11-16T16:47:44.859473+0000","last_unstale":"2023-11-16T16:47:44.860204+0000","last_undegraded":"2023-11-16T16:47:44.860204+0000","last_fullsized":"2023-11-16T16:47:44.860204+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:57:23.491160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857085+0000","last_change":"2023-11-16T16:47:26.379604+0000","last_active":"2023-11-16T16:47:44.857085+0000","last_peered":"2023-11-16T16:47:44.857085+0000","last_clean":"2023-11-16T16:47:44.857085+0000","last_became_active":"2023-11-16T16:47:26.379225+0000","last_became_peered":"2023-11-16T16:47:26.379225+0000","last_unstale":"2023-11-16T16:47:44.857085+0000","last_undegraded":"2023-11-16T16:47:44.857085+0000","last_fullsized":"2023-11-16T16:47:44.857085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:26:54.929229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472135+0000","last_change":"2023-11-16T16:47:26.367373+0000","last_active":"2023-11-16T16:47:45.472135+0000","last_peered":"2023-11-16T16:47:45.472135+0000","last_clean":"2023-11-16T16:47:45.472135+0000","last_became_active":"2023-11-16T16:47:26.366549+0000","last_became_peered":"2023-11-16T16:47:26.366549+0000","last_unstale":"2023-11-16T16:47:45.472135+0000","last_undegraded":"2023-11-16T16:47:45.472135+0000","last_fullsized":"2023-11-16T16:47:45.472135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:54:59.079284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854006+0000","last_change":"2023-11-16T16:47:26.376291+0000","last_active":"2023-11-16T16:47:44.854006+0000","last_peered":"2023-11-16T16:47:44.854006+0000","last_clean":"2023-11-16T16:47:44.854006+0000","last_became_active":"2023-11-16T16:47:26.374608+0000","last_became_peered":"2023-11-16T16:47:26.374608+0000","last_unstale":"2023-11-16T16:47:44.854006+0000","last_undegraded":"2023-11-16T16:47:44.854006+0000","last_fullsized":"2023-11-16T16:47:44.854006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:17:25.971629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855424+0000","last_change":"2023-11-16T16:47:26.382444+0000","last_active":"2023-11-16T16:47:44.855424+0000","last_peered":"2023-11-16T16:47:44.855424+0000","last_clean":"2023-11-16T16:47:44.855424+0000","last_became_active":"2023-11-16T16:47:26.382130+0000","last_became_peered":"2023-11-16T16:47:26.382130+0000","last_unstale":"2023-11-16T16:47:44.855424+0000","last_undegraded":"2023-11-16T16:47:44.855424+0000","last_fullsized":"2023-11-16T16:47:44.855424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:34.208412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854988+0000","last_change":"2023-11-16T16:47:30.838569+0000","last_active":"2023-11-16T16:47:44.854988+0000","last_peered":"2023-11-16T16:47:44.854988+0000","last_clean":"2023-11-16T16:47:44.854988+0000","last_became_active":"2023-11-16T16:47:30.838282+0000","last_became_peered":"2023-11-16T16:47:30.838282+0000","last_unstale":"2023-11-16T16:47:44.854988+0000","last_undegraded":"2023-11-16T16:47:44.854988+0000","last_fullsized":"2023-11-16T16:47:44.854988+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:06:09.584975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.907867+0000","last_change":"2023-11-16T16:47:25.820254+0000","last_active":"2023-11-16T16:47:51.907867+0000","last_peered":"2023-11-16T16:47:51.907867+0000","last_clean":"2023-11-16T16:47:51.907867+0000","last_became_active":"2023-11-16T16:47:25.819924+0000","last_became_peered":"2023-11-16T16:47:25.819924+0000","last_unstale":"2023-11-16T16:47:51.907867+0000","last_undegraded":"2023-11-16T16:47:51.907867+0000","last_fullsized":"2023-11-16T16:47:51.907867+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:50:37.572954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.413299+0000","last_change":"2023-11-16T16:47:26.374560+0000","last_active":"2023-11-16T16:47:45.413299+0000","last_peered":"2023-11-16T16:47:45.413299+0000","last_clean":"2023-11-16T16:47:45.413299+0000","last_became_active":"2023-11-16T16:47:26.373554+0000","last_became_peered":"2023-11-16T16:47:26.373554+0000","last_unstale":"2023-11-16T16:47:45.413299+0000","last_undegraded":"2023-11-16T16:47:45.413299+0000","last_fullsized":"2023-11-16T16:47:45.413299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:33:15.911693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854083+0000","last_change":"2023-11-16T16:47:30.826047+0000","last_active":"2023-11-16T16:47:44.854083+0000","last_peered":"2023-11-16T16:47:44.854083+0000","last_clean":"2023-11-16T16:47:44.854083+0000","last_became_active":"2023-11-16T16:47:30.825696+0000","last_became_peered":"2023-11-16T16:47:30.825696+0000","last_unstale":"2023-11-16T16:47:44.854083+0000","last_undegraded":"2023-11-16T16:47:44.854083+0000","last_fullsized":"2023-11-16T16:47:44.854083+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:31:58.343449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.413568+0000","last_change":"2023-11-16T16:47:25.830763+0000","last_active":"2023-11-16T16:47:45.413568+0000","last_peered":"2023-11-16T16:47:45.413568+0000","last_clean":"2023-11-16T16:47:45.413568+0000","last_became_active":"2023-11-16T16:47:25.830405+0000","last_became_peered":"2023-11-16T16:47:25.830405+0000","last_unstale":"2023-11-16T16:47:45.413568+0000","last_undegraded":"2023-11-16T16:47:45.413568+0000","last_fullsized":"2023-11-16T16:47:45.413568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:39:02.087710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.859518+0000","last_change":"2023-11-16T16:47:26.385897+0000","last_active":"2023-11-16T16:47:44.859518+0000","last_peered":"2023-11-16T16:47:44.859518+0000","last_clean":"2023-11-16T16:47:44.859518+0000","last_became_active":"2023-11-16T16:47:26.382557+0000","last_became_peered":"2023-11-16T16:47:26.382557+0000","last_unstale":"2023-11-16T16:47:44.859518+0000","last_undegraded":"2023-11-16T16:47:44.859518+0000","last_fullsized":"2023-11-16T16:47:44.859518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:19:59.689853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854309+0000","last_change":"2023-11-16T16:47:30.828521+0000","last_active":"2023-11-16T16:47:44.854309+0000","last_peered":"2023-11-16T16:47:44.854309+0000","last_clean":"2023-11-16T16:47:44.854309+0000","last_became_active":"2023-11-16T16:47:30.828135+0000","last_became_peered":"2023-11-16T16:47:30.828135+0000","last_unstale":"2023-11-16T16:47:44.854309+0000","last_undegraded":"2023-11-16T16:47:44.854309+0000","last_fullsized":"2023-11-16T16:47:44.854309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:19:57.561985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.905399+0000","last_change":"2023-11-16T16:47:25.819576+0000","last_active":"2023-11-16T16:47:51.905399+0000","last_peered":"2023-11-16T16:47:51.905399+0000","last_clean":"2023-11-16T16:47:51.905399+0000","last_became_active":"2023-11-16T16:47:25.818354+0000","last_became_peered":"2023-11-16T16:47:25.818354+0000","last_unstale":"2023-11-16T16:47:51.905399+0000","last_undegraded":"2023-11-16T16:47:51.905399+0000","last_fullsized":"2023-11-16T16:47:51.905399+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:43:47.079077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857839+0000","last_change":"2023-11-16T16:47:26.364632+0000","last_active":"2023-11-16T16:47:44.857839+0000","last_peered":"2023-11-16T16:47:44.857839+0000","last_clean":"2023-11-16T16:47:44.857839+0000","last_became_active":"2023-11-16T16:47:26.364288+0000","last_became_peered":"2023-11-16T16:47:26.364288+0000","last_unstale":"2023-11-16T16:47:44.857839+0000","last_undegraded":"2023-11-16T16:47:44.857839+0000","last_fullsized":"2023-11-16T16:47:44.857839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:17:52.757571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854072+0000","last_change":"2023-11-16T16:47:30.828722+0000","last_active":"2023-11-16T16:47:44.854072+0000","last_peered":"2023-11-16T16:47:44.854072+0000","last_clean":"2023-11-16T16:47:44.854072+0000","last_became_active":"2023-11-16T16:47:30.828299+0000","last_became_peered":"2023-11-16T16:47:30.828299+0000","last_unstale":"2023-11-16T16:47:44.854072+0000","last_undegraded":"2023-11-16T16:47:44.854072+0000","last_fullsized":"2023-11-16T16:47:44.854072+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:33:38.194438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856600+0000","last_change":"2023-11-16T16:47:25.817030+0000","last_active":"2023-11-16T16:47:44.856600+0000","last_peered":"2023-11-16T16:47:44.856600+0000","last_clean":"2023-11-16T16:47:44.856600+0000","last_became_active":"2023-11-16T16:47:25.816710+0000","last_became_peered":"2023-11-16T16:47:25.816710+0000","last_unstale":"2023-11-16T16:47:44.856600+0000","last_undegraded":"2023-11-16T16:47:44.856600+0000","last_fullsized":"2023-11-16T16:47:44.856600+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:40:22.859236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473673+0000","last_change":"2023-11-16T16:47:26.371226+0000","last_active":"2023-11-16T16:47:45.473673+0000","last_peered":"2023-11-16T16:47:45.473673+0000","last_clean":"2023-11-16T16:47:45.473673+0000","last_became_active":"2023-11-16T16:47:26.370925+0000","last_became_peered":"2023-11-16T16:47:26.370925+0000","last_unstale":"2023-11-16T16:47:45.473673+0000","last_undegraded":"2023-11-16T16:47:45.473673+0000","last_fullsized":"2023-11-16T16:47:45.473673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:48:12.574242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855221+0000","last_change":"2023-11-16T16:47:30.821364+0000","last_active":"2023-11-16T16:47:44.855221+0000","last_peered":"2023-11-16T16:47:44.855221+0000","last_clean":"2023-11-16T16:47:44.855221+0000","last_became_active":"2023-11-16T16:47:30.820938+0000","last_became_peered":"2023-11-16T16:47:30.820938+0000","last_unstale":"2023-11-16T16:47:44.855221+0000","last_undegraded":"2023-11-16T16:47:44.855221+0000","last_fullsized":"2023-11-16T16:47:44.855221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:13:28.692462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'9","reported_seq":91,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:41.989059+0000","last_change":"2023-11-16T16:47:25.818612+0000","last_active":"2023-11-16T16:48:41.989059+0000","last_peered":"2023-11-16T16:48:41.989059+0000","last_clean":"2023-11-16T16:48:41.989059+0000","last_became_active":"2023-11-16T16:47:25.818272+0000","last_became_peered":"2023-11-16T16:47:25.818272+0000","last_unstale":"2023-11-16T16:48:41.989059+0000","last_undegraded":"2023-11-16T16:48:41.989059+0000","last_fullsized":"2023-11-16T16:48:41.989059+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:00:33.166271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10962,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":14,"num_write":6,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857267+0000","last_change":"2023-11-16T16:47:26.372245+0000","last_active":"2023-11-16T16:47:44.857267+0000","last_peered":"2023-11-16T16:47:44.857267+0000","last_clean":"2023-11-16T16:47:44.857267+0000","last_became_active":"2023-11-16T16:47:26.371942+0000","last_became_peered":"2023-11-16T16:47:26.371942+0000","last_unstale":"2023-11-16T16:47:44.857267+0000","last_undegraded":"2023-11-16T16:47:44.857267+0000","last_fullsized":"2023-11-16T16:47:44.857267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:25:33.613660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855696+0000","last_change":"2023-11-16T16:47:30.839669+0000","last_active":"2023-11-16T16:47:44.855696+0000","last_peered":"2023-11-16T16:47:44.855696+0000","last_clean":"2023-11-16T16:47:44.855696+0000","last_became_active":"2023-11-16T16:47:30.839165+0000","last_became_peered":"2023-11-16T16:47:30.839165+0000","last_unstale":"2023-11-16T16:47:44.855696+0000","last_undegraded":"2023-11-16T16:47:44.855696+0000","last_fullsized":"2023-11-16T16:47:44.855696+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:26:12.940109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.905121+0000","last_change":"2023-11-16T16:47:25.819867+0000","last_active":"2023-11-16T16:47:51.905121+0000","last_peered":"2023-11-16T16:47:51.905121+0000","last_clean":"2023-11-16T16:47:51.905121+0000","last_became_active":"2023-11-16T16:47:25.819516+0000","last_became_peered":"2023-11-16T16:47:25.819516+0000","last_unstale":"2023-11-16T16:47:51.905121+0000","last_undegraded":"2023-11-16T16:47:51.905121+0000","last_fullsized":"2023-11-16T16:47:51.905121+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:23:15.640449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473687+0000","last_change":"2023-11-16T16:47:26.383679+0000","last_active":"2023-11-16T16:47:45.473687+0000","last_peered":"2023-11-16T16:47:45.473687+0000","last_clean":"2023-11-16T16:47:45.473687+0000","last_became_active":"2023-11-16T16:47:26.383370+0000","last_became_peered":"2023-11-16T16:47:26.383370+0000","last_unstale":"2023-11-16T16:47:45.473687+0000","last_undegraded":"2023-11-16T16:47:45.473687+0000","last_fullsized":"2023-11-16T16:47:45.473687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:56:40.615856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472127+0000","last_change":"2023-11-16T16:47:30.832845+0000","last_active":"2023-11-16T16:47:45.472127+0000","last_peered":"2023-11-16T16:47:45.472127+0000","last_clean":"2023-11-16T16:47:45.472127+0000","last_became_active":"2023-11-16T16:47:30.832364+0000","last_became_peered":"2023-11-16T16:47:30.832364+0000","last_unstale":"2023-11-16T16:47:45.472127+0000","last_undegraded":"2023-11-16T16:47:45.472127+0000","last_fullsized":"2023-11-16T16:47:45.472127+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:41:14.777870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.910105+0000","last_change":"2023-11-16T16:47:25.817420+0000","last_active":"2023-11-16T16:47:51.910105+0000","last_peered":"2023-11-16T16:47:51.910105+0000","last_clean":"2023-11-16T16:47:51.910105+0000","last_became_active":"2023-11-16T16:47:25.816160+0000","last_became_peered":"2023-11-16T16:47:25.816160+0000","last_unstale":"2023-11-16T16:47:51.910105+0000","last_undegraded":"2023-11-16T16:47:51.910105+0000","last_fullsized":"2023-11-16T16:47:51.910105+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:14:09.295340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.413125+0000","last_change":"2023-11-16T16:47:26.370729+0000","last_active":"2023-11-16T16:47:45.413125+0000","last_peered":"2023-11-16T16:47:45.413125+0000","last_clean":"2023-11-16T16:47:45.413125+0000","last_became_active":"2023-11-16T16:47:26.370435+0000","last_became_peered":"2023-11-16T16:47:26.370435+0000","last_unstale":"2023-11-16T16:47:45.413125+0000","last_undegraded":"2023-11-16T16:47:45.413125+0000","last_fullsized":"2023-11-16T16:47:45.413125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:37:45.259435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.861176+0000","last_change":"2023-11-16T16:47:30.832895+0000","last_active":"2023-11-16T16:47:44.861176+0000","last_peered":"2023-11-16T16:47:44.861176+0000","last_clean":"2023-11-16T16:47:44.861176+0000","last_became_active":"2023-11-16T16:47:30.832192+0000","last_became_peered":"2023-11-16T16:47:30.832192+0000","last_unstale":"2023-11-16T16:47:44.861176+0000","last_undegraded":"2023-11-16T16:47:44.861176+0000","last_fullsized":"2023-11-16T16:47:44.861176+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:30:54.337765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473152+0000","last_change":"2023-11-16T16:47:25.815673+0000","last_active":"2023-11-16T16:47:45.473152+0000","last_peered":"2023-11-16T16:47:45.473152+0000","last_clean":"2023-11-16T16:47:45.473152+0000","last_became_active":"2023-11-16T16:47:25.815362+0000","last_became_peered":"2023-11-16T16:47:25.815362+0000","last_unstale":"2023-11-16T16:47:45.473152+0000","last_undegraded":"2023-11-16T16:47:45.473152+0000","last_fullsized":"2023-11-16T16:47:45.473152+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:50:06.110785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.853033+0000","last_change":"2023-11-16T16:47:26.383483+0000","last_active":"2023-11-16T16:47:44.853033+0000","last_peered":"2023-11-16T16:47:44.853033+0000","last_clean":"2023-11-16T16:47:44.853033+0000","last_became_active":"2023-11-16T16:47:26.383177+0000","last_became_peered":"2023-11-16T16:47:26.383177+0000","last_unstale":"2023-11-16T16:47:44.853033+0000","last_undegraded":"2023-11-16T16:47:44.853033+0000","last_fullsized":"2023-11-16T16:47:44.853033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:27:38.101289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.413695+0000","last_change":"2023-11-16T16:47:30.829044+0000","last_active":"2023-11-16T16:47:45.413695+0000","last_peered":"2023-11-16T16:47:45.413695+0000","last_clean":"2023-11-16T16:47:45.413695+0000","last_became_active":"2023-11-16T16:47:30.827398+0000","last_became_peered":"2023-11-16T16:47:30.827398+0000","last_unstale":"2023-11-16T16:47:45.413695+0000","last_undegraded":"2023-11-16T16:47:45.413695+0000","last_fullsized":"2023-11-16T16:47:45.413695+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:54:57.131841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:42.990133+0000","last_change":"2023-11-16T16:47:25.817461+0000","last_active":"2023-11-16T16:48:42.990133+0000","last_peered":"2023-11-16T16:48:42.990133+0000","last_clean":"2023-11-16T16:48:42.990133+0000","last_became_active":"2023-11-16T16:47:25.817135+0000","last_became_peered":"2023-11-16T16:47:25.817135+0000","last_unstale":"2023-11-16T16:48:42.990133+0000","last_undegraded":"2023-11-16T16:48:42.990133+0000","last_fullsized":"2023-11-16T16:48:42.990133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:00:21.140226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858098+0000","last_change":"2023-11-16T16:47:26.380218+0000","last_active":"2023-11-16T16:47:44.858098+0000","last_peered":"2023-11-16T16:47:44.858098+0000","last_clean":"2023-11-16T16:47:44.858098+0000","last_became_active":"2023-11-16T16:47:26.379849+0000","last_became_peered":"2023-11-16T16:47:26.379849+0000","last_unstale":"2023-11-16T16:47:44.858098+0000","last_undegraded":"2023-11-16T16:47:44.858098+0000","last_fullsized":"2023-11-16T16:47:44.858098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:16:44.031031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.852960+0000","last_change":"2023-11-16T16:47:30.836464+0000","last_active":"2023-11-16T16:47:44.852960+0000","last_peered":"2023-11-16T16:47:44.852960+0000","last_clean":"2023-11-16T16:47:44.852960+0000","last_became_active":"2023-11-16T16:47:30.835185+0000","last_became_peered":"2023-11-16T16:47:30.835185+0000","last_unstale":"2023-11-16T16:47:44.852960+0000","last_undegraded":"2023-11-16T16:47:44.852960+0000","last_fullsized":"2023-11-16T16:47:44.852960+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:00:34.542077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.898666+0000","last_change":"2023-11-16T16:47:25.816583+0000","last_active":"2023-11-16T16:47:50.898666+0000","last_peered":"2023-11-16T16:47:50.898666+0000","last_clean":"2023-11-16T16:47:50.898666+0000","last_became_active":"2023-11-16T16:47:25.816286+0000","last_became_peered":"2023-11-16T16:47:25.816286+0000","last_unstale":"2023-11-16T16:47:50.898666+0000","last_undegraded":"2023-11-16T16:47:50.898666+0000","last_fullsized":"2023-11-16T16:47:50.898666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:55:47.335416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.860328+0000","last_change":"2023-11-16T16:47:26.379964+0000","last_active":"2023-11-16T16:47:44.860328+0000","last_peered":"2023-11-16T16:47:44.860328+0000","last_clean":"2023-11-16T16:47:44.860328+0000","last_became_active":"2023-11-16T16:47:26.376857+0000","last_became_peered":"2023-11-16T16:47:26.376857+0000","last_unstale":"2023-11-16T16:47:44.860328+0000","last_undegraded":"2023-11-16T16:47:44.860328+0000","last_fullsized":"2023-11-16T16:47:44.860328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:44:33.471523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854625+0000","last_change":"2023-11-16T16:47:30.832572+0000","last_active":"2023-11-16T16:47:44.854625+0000","last_peered":"2023-11-16T16:47:44.854625+0000","last_clean":"2023-11-16T16:47:44.854625+0000","last_became_active":"2023-11-16T16:47:30.832138+0000","last_became_peered":"2023-11-16T16:47:30.832138+0000","last_unstale":"2023-11-16T16:47:44.854625+0000","last_undegraded":"2023-11-16T16:47:44.854625+0000","last_fullsized":"2023-11-16T16:47:44.854625+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:25:20.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.900690+0000","last_change":"2023-11-16T16:47:25.826848+0000","last_active":"2023-11-16T16:47:50.900690+0000","last_peered":"2023-11-16T16:47:50.900690+0000","last_clean":"2023-11-16T16:47:50.900690+0000","last_became_active":"2023-11-16T16:47:25.826532+0000","last_became_peered":"2023-11-16T16:47:25.826532+0000","last_unstale":"2023-11-16T16:47:50.900690+0000","last_undegraded":"2023-11-16T16:47:50.900690+0000","last_fullsized":"2023-11-16T16:47:50.900690+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:34:36.051545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.863649+0000","last_change":"2023-11-16T16:47:44.863649+0000","last_active":"2023-11-16T16:47:44.863649+0000","last_peered":"2023-11-16T16:47:44.863649+0000","last_clean":"2023-11-16T16:47:44.863649+0000","last_became_active":"2023-11-16T16:47:44.863350+0000","last_became_peered":"2023-11-16T16:47:44.863350+0000","last_unstale":"2023-11-16T16:47:44.863649+0000","last_undegraded":"2023-11-16T16:47:44.863649+0000","last_fullsized":"2023-11-16T16:47:44.863649+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:45:32.381225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.853019+0000","last_change":"2023-11-16T16:47:30.829074+0000","last_active":"2023-11-16T16:47:44.853019+0000","last_peered":"2023-11-16T16:47:44.853019+0000","last_clean":"2023-11-16T16:47:44.853019+0000","last_became_active":"2023-11-16T16:47:30.828688+0000","last_became_peered":"2023-11-16T16:47:30.828688+0000","last_unstale":"2023-11-16T16:47:44.853019+0000","last_undegraded":"2023-11-16T16:47:44.853019+0000","last_fullsized":"2023-11-16T16:47:44.853019+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:56:56.319865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'35","reported_seq":173,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:43.991507+0000","last_change":"2023-11-16T16:47:25.816469+0000","last_active":"2023-11-16T16:48:43.991507+0000","last_peered":"2023-11-16T16:48:43.991507+0000","last_clean":"2023-11-16T16:48:43.991507+0000","last_became_active":"2023-11-16T16:47:25.816115+0000","last_became_peered":"2023-11-16T16:47:25.816115+0000","last_unstale":"2023-11-16T16:48:43.991507+0000","last_undegraded":"2023-11-16T16:48:43.991507+0000","last_fullsized":"2023-11-16T16:48:43.991507+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:33:32.434023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66685,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":112,"num_read_kb":121,"num_write":36,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856961+0000","last_change":"2023-11-16T16:47:26.373382+0000","last_active":"2023-11-16T16:47:44.856961+0000","last_peered":"2023-11-16T16:47:44.856961+0000","last_clean":"2023-11-16T16:47:44.856961+0000","last_became_active":"2023-11-16T16:47:26.372951+0000","last_became_peered":"2023-11-16T16:47:26.372951+0000","last_unstale":"2023-11-16T16:47:44.856961+0000","last_undegraded":"2023-11-16T16:47:44.856961+0000","last_fullsized":"2023-11-16T16:47:44.856961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:29:19.977662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473826+0000","last_change":"2023-11-16T16:47:30.833890+0000","last_active":"2023-11-16T16:47:45.473826+0000","last_peered":"2023-11-16T16:47:45.473826+0000","last_clean":"2023-11-16T16:47:45.473826+0000","last_became_active":"2023-11-16T16:47:30.833489+0000","last_became_peered":"2023-11-16T16:47:30.833489+0000","last_unstale":"2023-11-16T16:47:45.473826+0000","last_undegraded":"2023-11-16T16:47:45.473826+0000","last_fullsized":"2023-11-16T16:47:45.473826+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:21:14.794417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.905467+0000","last_change":"2023-11-16T16:47:25.815950+0000","last_active":"2023-11-16T16:47:51.905467+0000","last_peered":"2023-11-16T16:47:51.905467+0000","last_clean":"2023-11-16T16:47:51.905467+0000","last_became_active":"2023-11-16T16:47:25.815657+0000","last_became_peered":"2023-11-16T16:47:25.815657+0000","last_unstale":"2023-11-16T16:47:51.905467+0000","last_undegraded":"2023-11-16T16:47:51.905467+0000","last_fullsized":"2023-11-16T16:47:51.905467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:31:06.425915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856308+0000","last_change":"2023-11-16T16:47:26.378190+0000","last_active":"2023-11-16T16:47:44.856308+0000","last_peered":"2023-11-16T16:47:44.856308+0000","last_clean":"2023-11-16T16:47:44.856308+0000","last_became_active":"2023-11-16T16:47:26.374581+0000","last_became_peered":"2023-11-16T16:47:26.374581+0000","last_unstale":"2023-11-16T16:47:44.856308+0000","last_undegraded":"2023-11-16T16:47:44.856308+0000","last_fullsized":"2023-11-16T16:47:44.856308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:22:26.437085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.471899+0000","last_change":"2023-11-16T16:47:30.826637+0000","last_active":"2023-11-16T16:47:45.471899+0000","last_peered":"2023-11-16T16:47:45.471899+0000","last_clean":"2023-11-16T16:47:45.471899+0000","last_became_active":"2023-11-16T16:47:30.826156+0000","last_became_peered":"2023-11-16T16:47:30.826156+0000","last_unstale":"2023-11-16T16:47:45.471899+0000","last_undegraded":"2023-11-16T16:47:45.471899+0000","last_fullsized":"2023-11-16T16:47:45.471899+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:01:29.047562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.901236+0000","last_change":"2023-11-16T16:47:25.822749+0000","last_active":"2023-11-16T16:47:50.901236+0000","last_peered":"2023-11-16T16:47:50.901236+0000","last_clean":"2023-11-16T16:47:50.901236+0000","last_became_active":"2023-11-16T16:47:25.822441+0000","last_became_peered":"2023-11-16T16:47:25.822441+0000","last_unstale":"2023-11-16T16:47:50.901236+0000","last_undegraded":"2023-11-16T16:47:50.901236+0000","last_fullsized":"2023-11-16T16:47:50.901236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:42:27.484194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858444+0000","last_change":"2023-11-16T16:47:26.375131+0000","last_active":"2023-11-16T16:47:44.858444+0000","last_peered":"2023-11-16T16:47:44.858444+0000","last_clean":"2023-11-16T16:47:44.858444+0000","last_became_active":"2023-11-16T16:47:26.374740+0000","last_became_peered":"2023-11-16T16:47:26.374740+0000","last_unstale":"2023-11-16T16:47:44.858444+0000","last_undegraded":"2023-11-16T16:47:44.858444+0000","last_fullsized":"2023-11-16T16:47:44.858444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:58:23.761900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.853864+0000","last_change":"2023-11-16T16:47:30.836806+0000","last_active":"2023-11-16T16:47:44.853864+0000","last_peered":"2023-11-16T16:47:44.853864+0000","last_clean":"2023-11-16T16:47:44.853864+0000","last_became_active":"2023-11-16T16:47:30.835299+0000","last_became_peered":"2023-11-16T16:47:30.835299+0000","last_unstale":"2023-11-16T16:47:44.853864+0000","last_undegraded":"2023-11-16T16:47:44.853864+0000","last_fullsized":"2023-11-16T16:47:44.853864+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:53:26.843235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":85,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:42.990174+0000","last_change":"2023-11-16T16:47:25.816793+0000","last_active":"2023-11-16T16:48:42.990174+0000","last_peered":"2023-11-16T16:48:42.990174+0000","last_clean":"2023-11-16T16:48:42.990174+0000","last_became_active":"2023-11-16T16:47:25.816433+0000","last_became_peered":"2023-11-16T16:47:25.816433+0000","last_unstale":"2023-11-16T16:48:42.990174+0000","last_undegraded":"2023-11-16T16:48:42.990174+0000","last_fullsized":"2023-11-16T16:48:42.990174+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:30:24.755449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858132+0000","last_change":"2023-11-16T16:47:26.375383+0000","last_active":"2023-11-16T16:47:44.858132+0000","last_peered":"2023-11-16T16:47:44.858132+0000","last_clean":"2023-11-16T16:47:44.858132+0000","last_became_active":"2023-11-16T16:47:26.375026+0000","last_became_peered":"2023-11-16T16:47:26.375026+0000","last_unstale":"2023-11-16T16:47:44.858132+0000","last_undegraded":"2023-11-16T16:47:44.858132+0000","last_fullsized":"2023-11-16T16:47:44.858132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:07:01.547781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.475115+0000","last_change":"2023-11-16T16:47:45.475115+0000","last_active":"2023-11-16T16:47:45.475115+0000","last_peered":"2023-11-16T16:47:45.475115+0000","last_clean":"2023-11-16T16:47:45.475115+0000","last_became_active":"2023-11-16T16:47:45.474475+0000","last_became_peered":"2023-11-16T16:47:45.474475+0000","last_unstale":"2023-11-16T16:47:45.475115+0000","last_undegraded":"2023-11-16T16:47:45.475115+0000","last_fullsized":"2023-11-16T16:47:45.475115+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:14:57.456154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.909026+0000","last_change":"2023-11-16T16:47:25.826911+0000","last_active":"2023-11-16T16:47:51.909026+0000","last_peered":"2023-11-16T16:47:51.909026+0000","last_clean":"2023-11-16T16:47:51.909026+0000","last_became_active":"2023-11-16T16:47:25.826605+0000","last_became_peered":"2023-11-16T16:47:25.826605+0000","last_unstale":"2023-11-16T16:47:51.909026+0000","last_undegraded":"2023-11-16T16:47:51.909026+0000","last_fullsized":"2023-11-16T16:47:51.909026+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:43:53.441571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472956+0000","last_change":"2023-11-16T16:47:26.376072+0000","last_active":"2023-11-16T16:47:45.472956+0000","last_peered":"2023-11-16T16:47:45.472956+0000","last_clean":"2023-11-16T16:47:45.472956+0000","last_became_active":"2023-11-16T16:47:26.375585+0000","last_became_peered":"2023-11-16T16:47:26.375585+0000","last_unstale":"2023-11-16T16:47:45.472956+0000","last_undegraded":"2023-11-16T16:47:45.472956+0000","last_fullsized":"2023-11-16T16:47:45.472956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:30:44.970814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472838+0000","last_change":"2023-11-16T16:47:30.834558+0000","last_active":"2023-11-16T16:47:45.472838+0000","last_peered":"2023-11-16T16:47:45.472838+0000","last_clean":"2023-11-16T16:47:45.472838+0000","last_became_active":"2023-11-16T16:47:30.833989+0000","last_became_peered":"2023-11-16T16:47:30.833989+0000","last_unstale":"2023-11-16T16:47:45.472838+0000","last_undegraded":"2023-11-16T16:47:45.472838+0000","last_fullsized":"2023-11-16T16:47:45.472838+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:54:42.983565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:52.910715+0000","last_change":"2023-11-16T16:47:25.817016+0000","last_active":"2023-11-16T16:47:52.910715+0000","last_peered":"2023-11-16T16:47:52.910715+0000","last_clean":"2023-11-16T16:47:52.910715+0000","last_became_active":"2023-11-16T16:47:25.816752+0000","last_became_peered":"2023-11-16T16:47:25.816752+0000","last_unstale":"2023-11-16T16:47:52.910715+0000","last_undegraded":"2023-11-16T16:47:52.910715+0000","last_fullsized":"2023-11-16T16:47:52.910715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:40:44.076735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472022+0000","last_change":"2023-11-16T16:47:26.374947+0000","last_active":"2023-11-16T16:47:45.472022+0000","last_peered":"2023-11-16T16:47:45.472022+0000","last_clean":"2023-11-16T16:47:45.472022+0000","last_became_active":"2023-11-16T16:47:26.374533+0000","last_became_peered":"2023-11-16T16:47:26.374533+0000","last_unstale":"2023-11-16T16:47:45.472022+0000","last_undegraded":"2023-11-16T16:47:45.472022+0000","last_fullsized":"2023-11-16T16:47:45.472022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:58:16.020607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856695+0000","last_change":"2023-11-16T16:47:30.830506+0000","last_active":"2023-11-16T16:47:44.856695+0000","last_peered":"2023-11-16T16:47:44.856695+0000","last_clean":"2023-11-16T16:47:44.856695+0000","last_became_active":"2023-11-16T16:47:30.830086+0000","last_became_peered":"2023-11-16T16:47:30.830086+0000","last_unstale":"2023-11-16T16:47:44.856695+0000","last_undegraded":"2023-11-16T16:47:44.856695+0000","last_fullsized":"2023-11-16T16:47:44.856695+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:50:36.980481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.853657+0000","last_change":"2023-11-16T16:47:25.830104+0000","last_active":"2023-11-16T16:47:44.853657+0000","last_peered":"2023-11-16T16:47:44.853657+0000","last_clean":"2023-11-16T16:47:44.853657+0000","last_became_active":"2023-11-16T16:47:25.829232+0000","last_became_peered":"2023-11-16T16:47:25.829232+0000","last_unstale":"2023-11-16T16:47:44.853657+0000","last_undegraded":"2023-11-16T16:47:44.853657+0000","last_fullsized":"2023-11-16T16:47:44.853657+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:29:15.724324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473108+0000","last_change":"2023-11-16T16:47:26.368918+0000","last_active":"2023-11-16T16:47:45.473108+0000","last_peered":"2023-11-16T16:47:45.473108+0000","last_clean":"2023-11-16T16:47:45.473108+0000","last_became_active":"2023-11-16T16:47:26.368592+0000","last_became_peered":"2023-11-16T16:47:26.368592+0000","last_unstale":"2023-11-16T16:47:45.473108+0000","last_undegraded":"2023-11-16T16:47:45.473108+0000","last_fullsized":"2023-11-16T16:47:45.473108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:09:36.266152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857755+0000","last_change":"2023-11-16T16:47:30.830558+0000","last_active":"2023-11-16T16:47:44.857755+0000","last_peered":"2023-11-16T16:47:44.857755+0000","last_clean":"2023-11-16T16:47:44.857755+0000","last_became_active":"2023-11-16T16:47:30.829777+0000","last_became_peered":"2023-11-16T16:47:30.829777+0000","last_unstale":"2023-11-16T16:47:44.857755+0000","last_undegraded":"2023-11-16T16:47:44.857755+0000","last_fullsized":"2023-11-16T16:47:44.857755+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:15:47.182273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858408+0000","last_change":"2023-11-16T16:47:25.820315+0000","last_active":"2023-11-16T16:47:44.858408+0000","last_peered":"2023-11-16T16:47:44.858408+0000","last_clean":"2023-11-16T16:47:44.858408+0000","last_became_active":"2023-11-16T16:47:25.820032+0000","last_became_peered":"2023-11-16T16:47:25.820032+0000","last_unstale":"2023-11-16T16:47:44.858408+0000","last_undegraded":"2023-11-16T16:47:44.858408+0000","last_fullsized":"2023-11-16T16:47:44.858408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:36:13.332902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472754+0000","last_change":"2023-11-16T16:47:26.379841+0000","last_active":"2023-11-16T16:47:45.472754+0000","last_peered":"2023-11-16T16:47:45.472754+0000","last_clean":"2023-11-16T16:47:45.472754+0000","last_became_active":"2023-11-16T16:47:26.377401+0000","last_became_peered":"2023-11-16T16:47:26.377401+0000","last_unstale":"2023-11-16T16:47:45.472754+0000","last_undegraded":"2023-11-16T16:47:45.472754+0000","last_fullsized":"2023-11-16T16:47:45.472754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:21:55.430094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473595+0000","last_change":"2023-11-16T16:47:45.473595+0000","last_active":"2023-11-16T16:47:45.473595+0000","last_peered":"2023-11-16T16:47:45.473595+0000","last_clean":"2023-11-16T16:47:45.473595+0000","last_became_active":"2023-11-16T16:47:45.473100+0000","last_became_peered":"2023-11-16T16:47:45.473100+0000","last_unstale":"2023-11-16T16:47:45.473595+0000","last_undegraded":"2023-11-16T16:47:45.473595+0000","last_fullsized":"2023-11-16T16:47:45.473595+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:17:45.050424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473340+0000","last_change":"2023-11-16T16:47:26.364112+0000","last_active":"2023-11-16T16:47:45.473340+0000","last_peered":"2023-11-16T16:47:45.473340+0000","last_clean":"2023-11-16T16:47:45.473340+0000","last_became_active":"2023-11-16T16:47:26.363767+0000","last_became_peered":"2023-11-16T16:47:26.363767+0000","last_unstale":"2023-11-16T16:47:45.473340+0000","last_undegraded":"2023-11-16T16:47:45.473340+0000","last_fullsized":"2023-11-16T16:47:45.473340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:01:40.809234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'4","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:39.984779+0000","last_change":"2023-11-16T16:47:25.830012+0000","last_active":"2023-11-16T16:48:39.984779+0000","last_peered":"2023-11-16T16:48:39.984779+0000","last_clean":"2023-11-16T16:48:39.984779+0000","last_became_active":"2023-11-16T16:47:25.829474+0000","last_became_peered":"2023-11-16T16:47:25.829474+0000","last_unstale":"2023-11-16T16:48:39.984779+0000","last_undegraded":"2023-11-16T16:48:39.984779+0000","last_fullsized":"2023-11-16T16:48:39.984779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:01:21.296554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":49,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858885+0000","last_change":"2023-11-16T16:47:30.832724+0000","last_active":"2023-11-16T16:47:44.858885+0000","last_peered":"2023-11-16T16:47:44.858885+0000","last_clean":"2023-11-16T16:47:44.858885+0000","last_became_active":"2023-11-16T16:47:30.832226+0000","last_became_peered":"2023-11-16T16:47:30.832226+0000","last_unstale":"2023-11-16T16:47:44.858885+0000","last_undegraded":"2023-11-16T16:47:44.858885+0000","last_fullsized":"2023-11-16T16:47:44.858885+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:11:15.234460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.476312+0000","last_change":"2023-11-16T16:47:45.474859+0000","last_active":"2023-11-16T16:47:45.476312+0000","last_peered":"2023-11-16T16:47:45.476312+0000","last_clean":"2023-11-16T16:47:45.476312+0000","last_became_active":"2023-11-16T16:47:45.474439+0000","last_became_peered":"2023-11-16T16:47:45.474439+0000","last_unstale":"2023-11-16T16:47:45.476312+0000","last_undegraded":"2023-11-16T16:47:45.476312+0000","last_fullsized":"2023-11-16T16:47:45.476312+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:05:04.346537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:52.910430+0000","last_change":"2023-11-16T16:47:25.813894+0000","last_active":"2023-11-16T16:47:52.910430+0000","last_peered":"2023-11-16T16:47:52.910430+0000","last_clean":"2023-11-16T16:47:52.910430+0000","last_became_active":"2023-11-16T16:47:25.813526+0000","last_became_peered":"2023-11-16T16:47:25.813526+0000","last_unstale":"2023-11-16T16:47:52.910430+0000","last_undegraded":"2023-11-16T16:47:52.910430+0000","last_fullsized":"2023-11-16T16:47:52.910430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:26:19.570859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":359,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857425+0000","last_change":"2023-11-16T16:46:18.016268+0000","last_active":"2023-11-16T16:47:44.857425+0000","last_peered":"2023-11-16T16:47:44.857425+0000","last_clean":"2023-11-16T16:47:44.857425+0000","last_became_active":"2023-11-16T16:46:18.015772+0000","last_became_peered":"2023-11-16T16:46:18.015772+0000","last_unstale":"2023-11-16T16:47:44.857425+0000","last_undegraded":"2023-11-16T16:47:44.857425+0000","last_fullsized":"2023-11-16T16:47:44.857425+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:43:04.489460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:43:04.489460+0000","last_clean_scrub_stamp":"2023-11-16T16:43:04.489460+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:22:57.779720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473045+0000","last_change":"2023-11-16T16:47:30.830574+0000","last_active":"2023-11-16T16:47:45.473045+0000","last_peered":"2023-11-16T16:47:45.473045+0000","last_clean":"2023-11-16T16:47:45.473045+0000","last_became_active":"2023-11-16T16:47:30.830040+0000","last_became_peered":"2023-11-16T16:47:30.830040+0000","last_unstale":"2023-11-16T16:47:45.473045+0000","last_undegraded":"2023-11-16T16:47:45.473045+0000","last_fullsized":"2023-11-16T16:47:45.473045+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:29:55.893689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856017+0000","last_change":"2023-11-16T16:47:26.378278+0000","last_active":"2023-11-16T16:47:44.856017+0000","last_peered":"2023-11-16T16:47:44.856017+0000","last_clean":"2023-11-16T16:47:44.856017+0000","last_became_active":"2023-11-16T16:47:26.377139+0000","last_became_peered":"2023-11-16T16:47:26.377139+0000","last_unstale":"2023-11-16T16:47:44.856017+0000","last_undegraded":"2023-11-16T16:47:44.856017+0000","last_fullsized":"2023-11-16T16:47:44.856017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:43:35.952129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.906018+0000","last_change":"2023-11-16T16:47:25.815714+0000","last_active":"2023-11-16T16:47:51.906018+0000","last_peered":"2023-11-16T16:47:51.906018+0000","last_clean":"2023-11-16T16:47:51.906018+0000","last_became_active":"2023-11-16T16:47:25.815452+0000","last_became_peered":"2023-11-16T16:47:25.815452+0000","last_unstale":"2023-11-16T16:47:51.906018+0000","last_undegraded":"2023-11-16T16:47:51.906018+0000","last_fullsized":"2023-11-16T16:47:51.906018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:33:18.235656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472075+0000","last_change":"2023-11-16T16:47:30.826690+0000","last_active":"2023-11-16T16:47:45.472075+0000","last_peered":"2023-11-16T16:47:45.472075+0000","last_clean":"2023-11-16T16:47:45.472075+0000","last_became_active":"2023-11-16T16:47:30.826216+0000","last_became_peered":"2023-11-16T16:47:30.826216+0000","last_unstale":"2023-11-16T16:47:45.472075+0000","last_undegraded":"2023-11-16T16:47:45.472075+0000","last_fullsized":"2023-11-16T16:47:45.472075+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:59:47.136636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.901761+0000","last_change":"2023-11-16T16:47:25.826960+0000","last_active":"2023-11-16T16:47:50.901761+0000","last_peered":"2023-11-16T16:47:50.901761+0000","last_clean":"2023-11-16T16:47:50.901761+0000","last_became_active":"2023-11-16T16:47:25.826668+0000","last_became_peered":"2023-11-16T16:47:25.826668+0000","last_unstale":"2023-11-16T16:47:50.901761+0000","last_undegraded":"2023-11-16T16:47:50.901761+0000","last_fullsized":"2023-11-16T16:47:50.901761+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:45:36.995013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856905+0000","last_change":"2023-11-16T16:47:26.370684+0000","last_active":"2023-11-16T16:47:44.856905+0000","last_peered":"2023-11-16T16:47:44.856905+0000","last_clean":"2023-11-16T16:47:44.856905+0000","last_became_active":"2023-11-16T16:47:26.370323+0000","last_became_peered":"2023-11-16T16:47:26.370323+0000","last_unstale":"2023-11-16T16:47:44.856905+0000","last_undegraded":"2023-11-16T16:47:44.856905+0000","last_fullsized":"2023-11-16T16:47:44.856905+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:33:01.058990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.412979+0000","last_change":"2023-11-16T16:47:30.837921+0000","last_active":"2023-11-16T16:47:45.412979+0000","last_peered":"2023-11-16T16:47:45.412979+0000","last_clean":"2023-11-16T16:47:45.412979+0000","last_became_active":"2023-11-16T16:47:30.831491+0000","last_became_peered":"2023-11-16T16:47:30.831491+0000","last_unstale":"2023-11-16T16:47:45.412979+0000","last_undegraded":"2023-11-16T16:47:45.412979+0000","last_fullsized":"2023-11-16T16:47:45.412979+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:02:07.321263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:00.309552+0000","last_change":"2023-11-16T16:47:25.816416+0000","last_active":"2023-11-16T16:48:00.309552+0000","last_peered":"2023-11-16T16:48:00.309552+0000","last_clean":"2023-11-16T16:48:00.309552+0000","last_became_active":"2023-11-16T16:47:25.815875+0000","last_became_peered":"2023-11-16T16:47:25.815875+0000","last_unstale":"2023-11-16T16:48:00.309552+0000","last_undegraded":"2023-11-16T16:48:00.309552+0000","last_fullsized":"2023-11-16T16:48:00.309552+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:34:15.271617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.413301+0000","last_change":"2023-11-16T16:47:26.386831+0000","last_active":"2023-11-16T16:47:45.413301+0000","last_peered":"2023-11-16T16:47:45.413301+0000","last_clean":"2023-11-16T16:47:45.413301+0000","last_became_active":"2023-11-16T16:47:26.386518+0000","last_became_peered":"2023-11-16T16:47:26.386518+0000","last_unstale":"2023-11-16T16:47:45.413301+0000","last_undegraded":"2023-11-16T16:47:45.413301+0000","last_fullsized":"2023-11-16T16:47:45.413301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:28:40.491752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854593+0000","last_change":"2023-11-16T16:47:30.823733+0000","last_active":"2023-11-16T16:47:44.854593+0000","last_peered":"2023-11-16T16:47:44.854593+0000","last_clean":"2023-11-16T16:47:44.854593+0000","last_became_active":"2023-11-16T16:47:30.823293+0000","last_became_peered":"2023-11-16T16:47:30.823293+0000","last_unstale":"2023-11-16T16:47:44.854593+0000","last_undegraded":"2023-11-16T16:47:44.854593+0000","last_fullsized":"2023-11-16T16:47:44.854593+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:30:09.995445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.899469+0000","last_change":"2023-11-16T16:47:25.818137+0000","last_active":"2023-11-16T16:47:50.899469+0000","last_peered":"2023-11-16T16:47:50.899469+0000","last_clean":"2023-11-16T16:47:50.899469+0000","last_became_active":"2023-11-16T16:47:25.817661+0000","last_became_peered":"2023-11-16T16:47:25.817661+0000","last_unstale":"2023-11-16T16:47:50.899469+0000","last_undegraded":"2023-11-16T16:47:50.899469+0000","last_fullsized":"2023-11-16T16:47:50.899469+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:17:33.467718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.415106+0000","last_change":"2023-11-16T16:47:45.415106+0000","last_active":"2023-11-16T16:47:45.415106+0000","last_peered":"2023-11-16T16:47:45.415106+0000","last_clean":"2023-11-16T16:47:45.415106+0000","last_became_active":"2023-11-16T16:47:45.414554+0000","last_became_peered":"2023-11-16T16:47:45.414554+0000","last_unstale":"2023-11-16T16:47:45.415106+0000","last_undegraded":"2023-11-16T16:47:45.415106+0000","last_fullsized":"2023-11-16T16:47:45.415106+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:20:08.005726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,8],"acting":[0,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854667+0000","last_change":"2023-11-16T16:47:30.820944+0000","last_active":"2023-11-16T16:47:44.854667+0000","last_peered":"2023-11-16T16:47:44.854667+0000","last_clean":"2023-11-16T16:47:44.854667+0000","last_became_active":"2023-11-16T16:47:30.820514+0000","last_became_peered":"2023-11-16T16:47:30.820514+0000","last_unstale":"2023-11-16T16:47:44.854667+0000","last_undegraded":"2023-11-16T16:47:44.854667+0000","last_fullsized":"2023-11-16T16:47:44.854667+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:18:55.421306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.899895+0000","last_change":"2023-11-16T16:47:25.826960+0000","last_active":"2023-11-16T16:47:50.899895+0000","last_peered":"2023-11-16T16:47:50.899895+0000","last_clean":"2023-11-16T16:47:50.899895+0000","last_became_active":"2023-11-16T16:47:25.826671+0000","last_became_peered":"2023-11-16T16:47:25.826671+0000","last_unstale":"2023-11-16T16:47:50.899895+0000","last_undegraded":"2023-11-16T16:47:50.899895+0000","last_fullsized":"2023-11-16T16:47:50.899895+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:41:34.201037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856299+0000","last_change":"2023-11-16T16:47:26.364697+0000","last_active":"2023-11-16T16:47:44.856299+0000","last_peered":"2023-11-16T16:47:44.856299+0000","last_clean":"2023-11-16T16:47:44.856299+0000","last_became_active":"2023-11-16T16:47:26.364418+0000","last_became_peered":"2023-11-16T16:47:26.364418+0000","last_unstale":"2023-11-16T16:47:44.856299+0000","last_undegraded":"2023-11-16T16:47:44.856299+0000","last_fullsized":"2023-11-16T16:47:44.856299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:31:32.598699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.475963+0000","last_change":"2023-11-16T16:47:45.474571+0000","last_active":"2023-11-16T16:47:45.475963+0000","last_peered":"2023-11-16T16:47:45.475963+0000","last_clean":"2023-11-16T16:47:45.475963+0000","last_became_active":"2023-11-16T16:47:45.474093+0000","last_became_peered":"2023-11-16T16:47:45.474093+0000","last_unstale":"2023-11-16T16:47:45.475963+0000","last_undegraded":"2023-11-16T16:47:45.475963+0000","last_fullsized":"2023-11-16T16:47:45.475963+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:24:09.557557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.899977+0000","last_change":"2023-11-16T16:47:25.826208+0000","last_active":"2023-11-16T16:47:50.899977+0000","last_peered":"2023-11-16T16:47:50.899977+0000","last_clean":"2023-11-16T16:47:50.899977+0000","last_became_active":"2023-11-16T16:47:25.825853+0000","last_became_peered":"2023-11-16T16:47:25.825853+0000","last_unstale":"2023-11-16T16:47:50.899977+0000","last_undegraded":"2023-11-16T16:47:50.899977+0000","last_fullsized":"2023-11-16T16:47:50.899977+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:02:40.593175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.471677+0000","last_change":"2023-11-16T16:47:26.361786+0000","last_active":"2023-11-16T16:47:45.471677+0000","last_peered":"2023-11-16T16:47:45.471677+0000","last_clean":"2023-11-16T16:47:45.471677+0000","last_became_active":"2023-11-16T16:47:26.361410+0000","last_became_peered":"2023-11-16T16:47:26.361410+0000","last_unstale":"2023-11-16T16:47:45.471677+0000","last_undegraded":"2023-11-16T16:47:45.471677+0000","last_fullsized":"2023-11-16T16:47:45.471677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:14:02.213856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.474814+0000","last_change":"2023-11-16T16:47:45.474123+0000","last_active":"2023-11-16T16:47:45.474814+0000","last_peered":"2023-11-16T16:47:45.474814+0000","last_clean":"2023-11-16T16:47:45.474814+0000","last_became_active":"2023-11-16T16:47:45.473630+0000","last_became_peered":"2023-11-16T16:47:45.473630+0000","last_unstale":"2023-11-16T16:47:45.474814+0000","last_undegraded":"2023-11-16T16:47:45.474814+0000","last_fullsized":"2023-11-16T16:47:45.474814+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:39:24.719048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:56.609089+0000","last_change":"2023-11-16T16:47:25.816793+0000","last_active":"2023-11-16T16:47:56.609089+0000","last_peered":"2023-11-16T16:47:56.609089+0000","last_clean":"2023-11-16T16:47:56.609089+0000","last_became_active":"2023-11-16T16:47:25.815136+0000","last_became_peered":"2023-11-16T16:47:25.815136+0000","last_unstale":"2023-11-16T16:47:56.609089+0000","last_undegraded":"2023-11-16T16:47:56.609089+0000","last_fullsized":"2023-11-16T16:47:56.609089+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:12:25.041301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.412827+0000","last_change":"2023-11-16T16:47:26.379945+0000","last_active":"2023-11-16T16:47:45.412827+0000","last_peered":"2023-11-16T16:47:45.412827+0000","last_clean":"2023-11-16T16:47:45.412827+0000","last_became_active":"2023-11-16T16:47:26.376552+0000","last_became_peered":"2023-11-16T16:47:26.376552+0000","last_unstale":"2023-11-16T16:47:45.412827+0000","last_undegraded":"2023-11-16T16:47:45.412827+0000","last_fullsized":"2023-11-16T16:47:45.412827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:44:18.198024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854429+0000","last_change":"2023-11-16T16:47:30.826281+0000","last_active":"2023-11-16T16:47:44.854429+0000","last_peered":"2023-11-16T16:47:44.854429+0000","last_clean":"2023-11-16T16:47:44.854429+0000","last_became_active":"2023-11-16T16:47:30.825920+0000","last_became_peered":"2023-11-16T16:47:30.825920+0000","last_unstale":"2023-11-16T16:47:44.854429+0000","last_undegraded":"2023-11-16T16:47:44.854429+0000","last_fullsized":"2023-11-16T16:47:44.854429+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:23:25.700645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.899941+0000","last_change":"2023-11-16T16:47:25.815879+0000","last_active":"2023-11-16T16:47:50.899941+0000","last_peered":"2023-11-16T16:47:50.899941+0000","last_clean":"2023-11-16T16:47:50.899941+0000","last_became_active":"2023-11-16T16:47:25.815557+0000","last_became_peered":"2023-11-16T16:47:25.815557+0000","last_unstale":"2023-11-16T16:47:50.899941+0000","last_undegraded":"2023-11-16T16:47:50.899941+0000","last_fullsized":"2023-11-16T16:47:50.899941+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:04:39.245684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858518+0000","last_change":"2023-11-16T16:47:26.365013+0000","last_active":"2023-11-16T16:47:44.858518+0000","last_peered":"2023-11-16T16:47:44.858518+0000","last_clean":"2023-11-16T16:47:44.858518+0000","last_became_active":"2023-11-16T16:47:26.364710+0000","last_became_peered":"2023-11-16T16:47:26.364710+0000","last_unstale":"2023-11-16T16:47:44.858518+0000","last_undegraded":"2023-11-16T16:47:44.858518+0000","last_fullsized":"2023-11-16T16:47:44.858518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:57:21.045225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857432+0000","last_change":"2023-11-16T16:47:30.826880+0000","last_active":"2023-11-16T16:47:44.857432+0000","last_peered":"2023-11-16T16:47:44.857432+0000","last_clean":"2023-11-16T16:47:44.857432+0000","last_became_active":"2023-11-16T16:47:30.826547+0000","last_became_peered":"2023-11-16T16:47:30.826547+0000","last_unstale":"2023-11-16T16:47:44.857432+0000","last_undegraded":"2023-11-16T16:47:44.857432+0000","last_fullsized":"2023-11-16T16:47:44.857432+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:34:34.448480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.908865+0000","last_change":"2023-11-16T16:47:25.825246+0000","last_active":"2023-11-16T16:47:51.908865+0000","last_peered":"2023-11-16T16:47:51.908865+0000","last_clean":"2023-11-16T16:47:51.908865+0000","last_became_active":"2023-11-16T16:47:25.824929+0000","last_became_peered":"2023-11-16T16:47:25.824929+0000","last_unstale":"2023-11-16T16:47:51.908865+0000","last_undegraded":"2023-11-16T16:47:51.908865+0000","last_fullsized":"2023-11-16T16:47:51.908865+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:15:26.168130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.859304+0000","last_change":"2023-11-16T16:47:26.372266+0000","last_active":"2023-11-16T16:47:44.859304+0000","last_peered":"2023-11-16T16:47:44.859304+0000","last_clean":"2023-11-16T16:47:44.859304+0000","last_became_active":"2023-11-16T16:47:26.371857+0000","last_became_peered":"2023-11-16T16:47:26.371857+0000","last_unstale":"2023-11-16T16:47:44.859304+0000","last_undegraded":"2023-11-16T16:47:44.859304+0000","last_fullsized":"2023-11-16T16:47:44.859304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:22:33.979224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.859613+0000","last_change":"2023-11-16T16:47:30.832187+0000","last_active":"2023-11-16T16:47:44.859613+0000","last_peered":"2023-11-16T16:47:44.859613+0000","last_clean":"2023-11-16T16:47:44.859613+0000","last_became_active":"2023-11-16T16:47:30.828115+0000","last_became_peered":"2023-11-16T16:47:30.828115+0000","last_unstale":"2023-11-16T16:47:44.859613+0000","last_undegraded":"2023-11-16T16:47:44.859613+0000","last_fullsized":"2023-11-16T16:47:44.859613+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:05:02.381726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473519+0000","last_change":"2023-11-16T16:47:25.822706+0000","last_active":"2023-11-16T16:47:45.473519+0000","last_peered":"2023-11-16T16:47:45.473519+0000","last_clean":"2023-11-16T16:47:45.473519+0000","last_became_active":"2023-11-16T16:47:25.822400+0000","last_became_peered":"2023-11-16T16:47:25.822400+0000","last_unstale":"2023-11-16T16:47:45.473519+0000","last_undegraded":"2023-11-16T16:47:45.473519+0000","last_fullsized":"2023-11-16T16:47:45.473519+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:32:15.408720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.474751+0000","last_change":"2023-11-16T16:47:45.474751+0000","last_active":"2023-11-16T16:47:45.474751+0000","last_peered":"2023-11-16T16:47:45.474751+0000","last_clean":"2023-11-16T16:47:45.474751+0000","last_became_active":"2023-11-16T16:47:45.474415+0000","last_became_peered":"2023-11-16T16:47:45.474415+0000","last_unstale":"2023-11-16T16:47:45.474751+0000","last_undegraded":"2023-11-16T16:47:45.474751+0000","last_fullsized":"2023-11-16T16:47:45.474751+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:56:14.927411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472751+0000","last_change":"2023-11-16T16:47:30.822549+0000","last_active":"2023-11-16T16:47:45.472751+0000","last_peered":"2023-11-16T16:47:45.472751+0000","last_clean":"2023-11-16T16:47:45.472751+0000","last_became_active":"2023-11-16T16:47:30.822150+0000","last_became_peered":"2023-11-16T16:47:30.822150+0000","last_unstale":"2023-11-16T16:47:45.472751+0000","last_undegraded":"2023-11-16T16:47:45.472751+0000","last_fullsized":"2023-11-16T16:47:45.472751+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:17:45.983962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857636+0000","last_change":"2023-11-16T16:47:25.826947+0000","last_active":"2023-11-16T16:47:44.857636+0000","last_peered":"2023-11-16T16:47:44.857636+0000","last_clean":"2023-11-16T16:47:44.857636+0000","last_became_active":"2023-11-16T16:47:25.826587+0000","last_became_peered":"2023-11-16T16:47:25.826587+0000","last_unstale":"2023-11-16T16:47:44.857636+0000","last_undegraded":"2023-11-16T16:47:44.857636+0000","last_fullsized":"2023-11-16T16:47:44.857636+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:26:31.810188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858841+0000","last_change":"2023-11-16T16:47:26.373914+0000","last_active":"2023-11-16T16:47:44.858841+0000","last_peered":"2023-11-16T16:47:44.858841+0000","last_clean":"2023-11-16T16:47:44.858841+0000","last_became_active":"2023-11-16T16:47:26.373594+0000","last_became_peered":"2023-11-16T16:47:26.373594+0000","last_unstale":"2023-11-16T16:47:44.858841+0000","last_undegraded":"2023-11-16T16:47:44.858841+0000","last_fullsized":"2023-11-16T16:47:44.858841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:18:55.608486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472165+0000","last_change":"2023-11-16T16:47:26.368237+0000","last_active":"2023-11-16T16:47:45.472165+0000","last_peered":"2023-11-16T16:47:45.472165+0000","last_clean":"2023-11-16T16:47:45.472165+0000","last_became_active":"2023-11-16T16:47:26.367957+0000","last_became_peered":"2023-11-16T16:47:26.367957+0000","last_unstale":"2023-11-16T16:47:45.472165+0000","last_undegraded":"2023-11-16T16:47:45.472165+0000","last_fullsized":"2023-11-16T16:47:45.472165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:33:19.976017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854982+0000","last_change":"2023-11-16T16:47:26.375399+0000","last_active":"2023-11-16T16:47:44.854982+0000","last_peered":"2023-11-16T16:47:44.854982+0000","last_clean":"2023-11-16T16:47:44.854982+0000","last_became_active":"2023-11-16T16:47:26.375064+0000","last_became_peered":"2023-11-16T16:47:26.375064+0000","last_unstale":"2023-11-16T16:47:44.854982+0000","last_undegraded":"2023-11-16T16:47:44.854982+0000","last_fullsized":"2023-11-16T16:47:44.854982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:10:23.687848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854908+0000","last_change":"2023-11-16T16:47:26.377080+0000","last_active":"2023-11-16T16:47:44.854908+0000","last_peered":"2023-11-16T16:47:44.854908+0000","last_clean":"2023-11-16T16:47:44.854908+0000","last_became_active":"2023-11-16T16:47:26.376759+0000","last_became_peered":"2023-11-16T16:47:26.376759+0000","last_unstale":"2023-11-16T16:47:44.854908+0000","last_undegraded":"2023-11-16T16:47:44.854908+0000","last_fullsized":"2023-11-16T16:47:44.854908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:51:18.518523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.862174+0000","last_change":"2023-11-16T16:47:26.380619+0000","last_active":"2023-11-16T16:47:44.862174+0000","last_peered":"2023-11-16T16:47:44.862174+0000","last_clean":"2023-11-16T16:47:44.862174+0000","last_became_active":"2023-11-16T16:47:26.378668+0000","last_became_peered":"2023-11-16T16:47:26.378668+0000","last_unstale":"2023-11-16T16:47:44.862174+0000","last_undegraded":"2023-11-16T16:47:44.862174+0000","last_fullsized":"2023-11-16T16:47:44.862174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:59:30.902022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856785+0000","last_change":"2023-11-16T16:47:26.377482+0000","last_active":"2023-11-16T16:47:44.856785+0000","last_peered":"2023-11-16T16:47:44.856785+0000","last_clean":"2023-11-16T16:47:44.856785+0000","last_became_active":"2023-11-16T16:47:26.377077+0000","last_became_peered":"2023-11-16T16:47:26.377077+0000","last_unstale":"2023-11-16T16:47:44.856785+0000","last_undegraded":"2023-11-16T16:47:44.856785+0000","last_fullsized":"2023-11-16T16:47:44.856785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:19:35.960276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472542+0000","last_change":"2023-11-16T16:47:26.368193+0000","last_active":"2023-11-16T16:47:45.472542+0000","last_peered":"2023-11-16T16:47:45.472542+0000","last_clean":"2023-11-16T16:47:45.472542+0000","last_became_active":"2023-11-16T16:47:26.367499+0000","last_became_peered":"2023-11-16T16:47:26.367499+0000","last_unstale":"2023-11-16T16:47:45.472542+0000","last_undegraded":"2023-11-16T16:47:45.472542+0000","last_fullsized":"2023-11-16T16:47:45.472542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:09:17.429110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855675+0000","last_change":"2023-11-16T16:47:26.367340+0000","last_active":"2023-11-16T16:47:44.855675+0000","last_peered":"2023-11-16T16:47:44.855675+0000","last_clean":"2023-11-16T16:47:44.855675+0000","last_became_active":"2023-11-16T16:47:26.367046+0000","last_became_peered":"2023-11-16T16:47:26.367046+0000","last_unstale":"2023-11-16T16:47:44.855675+0000","last_undegraded":"2023-11-16T16:47:44.855675+0000","last_fullsized":"2023-11-16T16:47:44.855675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:14:30.539229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.413517+0000","last_change":"2023-11-16T16:47:26.386174+0000","last_active":"2023-11-16T16:47:45.413517+0000","last_peered":"2023-11-16T16:47:45.413517+0000","last_clean":"2023-11-16T16:47:45.413517+0000","last_became_active":"2023-11-16T16:47:26.385496+0000","last_became_peered":"2023-11-16T16:47:26.385496+0000","last_unstale":"2023-11-16T16:47:45.413517+0000","last_undegraded":"2023-11-16T16:47:45.413517+0000","last_fullsized":"2023-11-16T16:47:45.413517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:52:02.965352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473021+0000","last_change":"2023-11-16T16:47:26.368224+0000","last_active":"2023-11-16T16:47:45.473021+0000","last_peered":"2023-11-16T16:47:45.473021+0000","last_clean":"2023-11-16T16:47:45.473021+0000","last_became_active":"2023-11-16T16:47:26.367909+0000","last_became_peered":"2023-11-16T16:47:26.367909+0000","last_unstale":"2023-11-16T16:47:45.473021+0000","last_undegraded":"2023-11-16T16:47:45.473021+0000","last_fullsized":"2023-11-16T16:47:45.473021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:52:55.966566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.412846+0000","last_change":"2023-11-16T16:47:26.381224+0000","last_active":"2023-11-16T16:47:45.412846+0000","last_peered":"2023-11-16T16:47:45.412846+0000","last_clean":"2023-11-16T16:47:45.412846+0000","last_became_active":"2023-11-16T16:47:26.376586+0000","last_became_peered":"2023-11-16T16:47:26.376586+0000","last_unstale":"2023-11-16T16:47:45.412846+0000","last_undegraded":"2023-11-16T16:47:45.412846+0000","last_fullsized":"2023-11-16T16:47:45.412846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:36:59.850981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858440+0000","last_change":"2023-11-16T16:47:26.385895+0000","last_active":"2023-11-16T16:47:44.858440+0000","last_peered":"2023-11-16T16:47:44.858440+0000","last_clean":"2023-11-16T16:47:44.858440+0000","last_became_active":"2023-11-16T16:47:26.385580+0000","last_became_peered":"2023-11-16T16:47:26.385580+0000","last_unstale":"2023-11-16T16:47:44.858440+0000","last_undegraded":"2023-11-16T16:47:44.858440+0000","last_fullsized":"2023-11-16T16:47:44.858440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:03:39.204040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.856629+0000","last_change":"2023-11-16T16:47:26.373785+0000","last_active":"2023-11-16T16:47:44.856629+0000","last_peered":"2023-11-16T16:47:44.856629+0000","last_clean":"2023-11-16T16:47:44.856629+0000","last_became_active":"2023-11-16T16:47:26.373425+0000","last_became_peered":"2023-11-16T16:47:26.373425+0000","last_unstale":"2023-11-16T16:47:44.856629+0000","last_undegraded":"2023-11-16T16:47:44.856629+0000","last_fullsized":"2023-11-16T16:47:44.856629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:16:02.226398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472287+0000","last_change":"2023-11-16T16:47:26.374722+0000","last_active":"2023-11-16T16:47:45.472287+0000","last_peered":"2023-11-16T16:47:45.472287+0000","last_clean":"2023-11-16T16:47:45.472287+0000","last_became_active":"2023-11-16T16:47:26.374326+0000","last_became_peered":"2023-11-16T16:47:26.374326+0000","last_unstale":"2023-11-16T16:47:45.472287+0000","last_undegraded":"2023-11-16T16:47:45.472287+0000","last_fullsized":"2023-11-16T16:47:45.472287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:52:18.596419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":249745,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":80977,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":338,"num_read_kb":253,"num_write":166,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466944,"data_stored":242931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383708,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6164,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738988,"statfs":{"total":95995035648,"available":95988723712,"internally_reserved":0,"allocated":995328,"data_stored":500317,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55900000000000005,"15min":0.56000000000000005},"min":{"1min":0.41799999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58299999999999996,"15min":0.58399999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.77300000000000002,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.55700000000000005}]},{"osd":1,"last update":"Thu Nov 16 16:48:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.65200000000000002,"15min":0.64600000000000002},"min":{"1min":0.46000000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.85499999999999998,"5min":0.873,"15min":0.873},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57999999999999996,"15min":0.57799999999999996},"min":{"1min":0.46600000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71799999999999997,"5min":0.749,"15min":0.749},"last":0.60999999999999999}]},{"osd":2,"last update":"Thu Nov 16 16:48:37 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52200000000000002,"15min":0.52500000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68200000000000005,"5min":1.159,"15min":1.159},"last":0.441},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.51900000000000002,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78700000000000003,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.46600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":4,"last update":"Thu Nov 16 16:48:37 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.68899999999999995,"15min":0.68300000000000005},"min":{"1min":0.49299999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88700000000000001,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.51600000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81399999999999995,"5min":1.046,"15min":1.046},"last":0.54700000000000004}]},{"osd":5,"last update":"Thu Nov 16 16:48:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.63700000000000001,"15min":0.64100000000000001},"min":{"1min":0.439,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.78200000000000003,"5min":1.345,"15min":1.345},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63,"15min":0.628},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.89200000000000002,"5min":1.02,"15min":1.02},"last":0.64200000000000002}]},{"osd":6,"last update":"Thu Nov 16 16:48:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.627,"15min":0.63},"min":{"1min":0.436,"5min":0.432,"15min":0.432},"max":{"1min":0.91200000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.496},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.76700000000000002,"15min":0.77300000000000002},"min":{"1min":0.59199999999999997,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.91500000000000004,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.71899999999999997}]},{"osd":7,"last update":"Thu Nov 16 16:48:37 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.58099999999999996,"5min":0.496,"15min":0.496},"max":{"1min":0.92000000000000004,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72999999999999998,"15min":0.73499999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.84699999999999998,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.68999999999999995}]},{"osd":8,"last update":"Thu Nov 16 16:48:27 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.82499999999999996}]},{"osd":9,"last update":"Thu Nov 16 16:48:37 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.83399999999999996,"15min":0.83799999999999997},"min":{"1min":0.63200000000000001,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.94499999999999995,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.81499999999999995,"15min":0.81299999999999994},"min":{"1min":0.63700000000000001,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.95899999999999996,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.84899999999999998}]},{"osd":10,"last update":"Thu Nov 16 16:48:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.751,"15min":0.753},"min":{"1min":0.56799999999999995,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0169999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.65800000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.92300000000000004,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.747}]}]},{"osd":10,"up_from":72,"seq":309237645345,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6632,"kb_used_data":1120,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738520,"statfs":{"total":95995035648,"available":95988244480,"internally_reserved":0,"allocated":1146880,"data_stored":1158377,"data_compressed":53789,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:27 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.58299999999999996}]},{"osd":1,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61699999999999999,"5min":0.622,"15min":0.622},"last":0.433},{"interface":"front","average":{"1min":0.441,"5min":0.44500000000000001,"15min":0.44600000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.39800000000000002}]},{"osd":2,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.45800000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61299999999999999,"15min":0.60899999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.67000000000000004}]},{"osd":3,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64200000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.66400000000000003,"15min":0.67200000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.81899999999999995,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.59399999999999997}]},{"osd":4,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.55800000000000005,"15min":0.56000000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77600000000000002,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.375,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.46400000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65400000000000003,"15min":0.65600000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.77000000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67300000000000004,"15min":0.67000000000000004},"min":{"1min":0.47199999999999998,"5min":0.435,"15min":0.435},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.72299999999999998}]},{"osd":6,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.74299999999999999,"15min":0.755},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.92400000000000004,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.72999999999999998,"15min":0.73899999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.87,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.75600000000000001}]},{"osd":7,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.46999999999999997}]},{"osd":8,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.92400000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.73099999999999998,"15min":0.73699999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.68700000000000006}]},{"osd":9,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.79800000000000004,"15min":0.81000000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.94699999999999995,"5min":0.995,"15min":0.995},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.77300000000000002,"15min":0.78700000000000003},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.91300000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.74399999999999999}]},{"osd":11,"last update":"Thu Nov 16 16:48:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.748,"15min":0.746},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.745,"15min":0.74299999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.79400000000000004}]}]},{"osd":9,"up_from":67,"seq":287762808870,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6372,"kb_used_data":988,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":1011712,"data_stored":500553,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.56699999999999995,"15min":0.58599999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.68300000000000005,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.57399999999999995,"15min":0.59099999999999997},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.64200000000000002,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.40699999999999997}]},{"osd":1,"last update":"Thu Nov 16 16:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.67000000000000004,"15min":0.68400000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.70499999999999996,"5min":1.002,"15min":1.002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.79900000000000004,"15min":0.81699999999999995},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.86699999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.52400000000000002}]},{"osd":2,"last update":"Thu Nov 16 16:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.85499999999999998,"15min":0.879},"min":{"1min":0.60599999999999998,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.82499999999999996,"5min":1.401,"15min":1.401},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.82999999999999996,"15min":0.85299999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.90000000000000002,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Nov 16 16:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.78600000000000003,"5min":0.995,"15min":0.995},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.82299999999999995,"15min":0.84999999999999998},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.90800000000000003,"5min":1.343,"15min":1.343},"last":0.61499999999999999}]},{"osd":4,"last update":"Thu Nov 16 16:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.84299999999999997,"15min":0.86699999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.97399999999999998,"5min":1.202,"15min":1.202},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.78900000000000003,"15min":0.80500000000000005},"min":{"1min":0.57399999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.88800000000000001,"5min":1.081,"15min":1.081},"last":0.56699999999999995}]},{"osd":5,"last update":"Thu Nov 16 16:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.92200000000000004,"15min":0.95399999999999996},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.94199999999999995,"5min":1.302,"15min":1.302},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.85999999999999999,"15min":0.88600000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.94899999999999995,"5min":1.196,"15min":1.196},"last":0.68700000000000006}]},{"osd":6,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.629}]},{"osd":7,"last update":"Thu Nov 16 16:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.90300000000000002,"15min":0.92800000000000005},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":0.97399999999999998,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.99399999999999999,"15min":1.022},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.0740000000000001,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.72899999999999998}]},{"osd":8,"last update":"Thu Nov 16 16:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":1.0189999999999999,"15min":1.0469999999999999},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.012,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.82399999999999995,"5min":1.0109999999999999,"15min":1.042},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.0880000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.83599999999999997}]},{"osd":10,"last update":"Thu Nov 16 16:48:19 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.94199999999999995,"15min":0.95399999999999996},"min":{"1min":0.72099999999999997,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.0169999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.93799999999999994,"15min":0.95199999999999996},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.0489999999999999,"5min":1.421,"15min":1.421},"last":0.71599999999999997}]},{"osd":11,"last update":"Thu Nov 16 16:48:31 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.78400000000000003}]}]},{"osd":8,"up_from":59,"seq":253403070506,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6496,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738656,"statfs":{"total":95995035648,"available":95988383744,"internally_reserved":0,"allocated":1007616,"data_stored":500142,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58599999999999997,"15min":0.60199999999999998},"min":{"1min":0.48299999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78300000000000003,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.42599999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.77700000000000002,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.878}]},{"osd":1,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.60899999999999999,"15min":0.59699999999999998},"min":{"1min":0.38300000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.97499999999999998,"5min":1.089,"15min":1.089},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.60599999999999998,"15min":0.59799999999999998},"min":{"1min":0.42599999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.68100000000000005}]},{"osd":2,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.47999999999999998,"15min":0.48099999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64400000000000002,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.48099999999999998,"15min":0.47599999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69099999999999995,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.56000000000000005}]},{"osd":3,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.627,"15min":0.63700000000000001},"min":{"1min":0.46200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81999999999999995,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.66600000000000004,"15min":0.68100000000000005},"min":{"1min":0.499,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93400000000000005,"5min":1.115,"15min":1.115},"last":0.748}]},{"osd":4,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.61399999999999999,"15min":0.59599999999999997},"min":{"1min":0.41399999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.40500000000000003,"5min":0.375,"15min":0.375},"max":{"1min":0.94999999999999996,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.79100000000000004}]},{"osd":5,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59599999999999997,"15min":0.60599999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.84499999999999997,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.59599999999999997,"15min":0.58999999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.75800000000000001}]},{"osd":6,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":0.80400000000000005,"15min":0.77300000000000002},"min":{"1min":0.56399999999999995,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":6.5940000000000003,"5min":6.5940000000000003,"15min":6.5940000000000003},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.75900000000000001,"15min":0.78000000000000003},"min":{"1min":0.57899999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.98099999999999998,"5min":1.093,"15min":1.093},"last":0.80600000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.72699999999999998,"15min":0.72999999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.94499999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.71699999999999997,"15min":0.70199999999999996},"min":{"1min":0.46500000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.81799999999999995}]},{"osd":9,"last update":"Thu Nov 16 16:47:58 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.59199999999999997,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.998,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.79800000000000004,"15min":0.80500000000000005},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.92900000000000005,"5min":1.393,"15min":1.393},"last":0.871}]},{"osd":10,"last update":"Thu Nov 16 16:48:20 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.72699999999999998,"15min":0.71499999999999997},"min":{"1min":0.60499999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.749,"15min":0.73499999999999999},"min":{"1min":0.63100000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.89800000000000002}]},{"osd":11,"last update":"Thu Nov 16 16:48:26 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.92300000000000004}]}]},{"osd":7,"up_from":51,"seq":219043332142,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6620,"kb_used_data":1044,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738532,"statfs":{"total":95995035648,"available":95988256768,"internally_reserved":0,"allocated":1069056,"data_stored":1106673,"data_compressed":53789,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56200000000000006,"15min":0.55500000000000005},"min":{"1min":0.34899999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.56799999999999995,"15min":0.55600000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.48699999999999999}]},{"osd":1,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.42799999999999999,"15min":0.41699999999999998},"min":{"1min":0.314,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65600000000000003,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.46999999999999997,"15min":0.46000000000000002},"min":{"1min":0.34899999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.67900000000000005,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.34300000000000003}]},{"osd":2,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.57699999999999996,"15min":0.57199999999999995},"min":{"1min":0.38600000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58099999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77900000000000003,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.52000000000000002}]},{"osd":3,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61299999999999999,"15min":0.61699999999999999},"min":{"1min":0.47099999999999997,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85399999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60199999999999998,"15min":0.58999999999999997},"min":{"1min":0.497,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72899999999999998,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.47499999999999998}]},{"osd":4,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.51600000000000001,"15min":0.5},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.83199999999999996,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.499},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51500000000000001,"15min":0.51300000000000001},"min":{"1min":0.35199999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.441}]},{"osd":5,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.63200000000000001,"15min":0.61699999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.878,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.66600000000000004}]},{"osd":6,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.74099999999999999,"15min":0.70499999999999996},"min":{"1min":0.52600000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.93100000000000005,"5min":4.8840000000000003,"15min":4.8840000000000003},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68799999999999994,"15min":0.68000000000000005},"min":{"1min":0.54800000000000004,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.90800000000000003,"5min":0.996,"15min":0.996},"last":0.68200000000000005}]},{"osd":8,"last update":"Thu Nov 16 16:48:27 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.70499999999999996,"15min":0.69499999999999995},"min":{"1min":0.47599999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.746,"15min":0.74099999999999999},"min":{"1min":0.55000000000000004,"5min":0.497,"15min":0.497},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.77500000000000002}]},{"osd":9,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.87,"15min":0.84199999999999997},"min":{"1min":0.57199999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":4.9210000000000003,"5min":4.9210000000000003,"15min":4.9210000000000003},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.80900000000000005,"15min":0.78300000000000003},"min":{"1min":0.60799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":4.9000000000000004,"5min":4.9000000000000004,"15min":4.9000000000000004},"last":0.63600000000000001}]},{"osd":10,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.76500000000000001,"15min":0.77400000000000002},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.022,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.752},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.72099999999999997,"15min":0.72499999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.88200000000000001,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.60899999999999999}]},{"osd":11,"last update":"Thu Nov 16 16:48:27 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.79700000000000004}]}]},{"osd":6,"up_from":46,"seq":197568495667,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6480,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738672,"statfs":{"total":95995035648,"available":95988400128,"internally_reserved":0,"allocated":991232,"data_stored":495729,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.49099999999999999,"15min":0.46700000000000003},"min":{"1min":0.312,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.71799999999999997,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55400000000000005,"15min":0.53400000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.81899999999999995,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.379}]},{"osd":1,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.69499999999999995,"15min":0.66000000000000003},"min":{"1min":0.50600000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.032,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.433,"5min":0.378,"15min":0.378},"max":{"1min":0.90000000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.65700000000000003}]},{"osd":2,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.72899999999999998,"15min":0.70999999999999996},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0589999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.751},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.74199999999999999,"15min":0.71999999999999997},"min":{"1min":0.56100000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.008,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.71499999999999997}]},{"osd":3,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.68799999999999994,"15min":0.66800000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.88},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60599999999999998,"15min":0.60199999999999998},"min":{"1min":0.36399999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.90600000000000003,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.54500000000000004}]},{"osd":4,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.73799999999999999},"min":{"1min":0.47799999999999998,"5min":0.44,"15min":0.44},"max":{"1min":1,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.79600000000000004,"15min":0.78100000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.1819999999999999,"5min":1.3,"15min":1.3},"last":0.64000000000000001}]},{"osd":5,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.73199999999999998,"15min":0.73099999999999998},"min":{"1min":0.50800000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.147,"5min":1.206,"15min":1.206},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80600000000000005,"15min":0.78800000000000003},"min":{"1min":0.58299999999999996,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.139,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.86199999999999999}]},{"osd":7,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.74099999999999999,"15min":0.71299999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.82199999999999995,"15min":0.80500000000000005},"min":{"1min":0.61599999999999999,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2030000000000001,"5min":1.246,"15min":1.246},"last":0.497}]},{"osd":8,"last update":"Thu Nov 16 16:48:30 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.91100000000000003,"15min":0.91500000000000004},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.92400000000000004,"15min":0.93000000000000005},"min":{"1min":0.63500000000000001,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.2370000000000001,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.81100000000000005}]},{"osd":9,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.998,"15min":1.026},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.2070000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.98099999999999998,"15min":1.0029999999999999},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.117,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.77100000000000002}]},{"osd":10,"last update":"Thu Nov 16 16:48:17 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.88900000000000001,"15min":0.89300000000000002},"min":{"1min":0.60899999999999999,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.671,"5min":1.671,"15min":1.671},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.004,"15min":1.0129999999999999},"min":{"1min":0.77200000000000002,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.54,"5min":1.54,"15min":1.54},"last":0.82699999999999996}]},{"osd":11,"last update":"Thu Nov 16 16:48:30 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.702,"5min":1.702,"15min":1.702},"last":0.79300000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7576,"kb_used_data":1040,"kb_used_omap":0,"kb_used_meta":6528,"kb_avail":93737576,"statfs":{"total":95995035648,"available":95987277824,"internally_reserved":0,"allocated":1064960,"data_stored":562324,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6684672},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:50 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.53000000000000003,"15min":0.51600000000000001},"min":{"1min":0.34699999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.71099999999999997,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.52300000000000002,"15min":0.498},"min":{"1min":0.35199999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.68400000000000005,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.54900000000000004}]},{"osd":2,"last update":"Thu Nov 16 16:48:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.53900000000000003,"15min":0.51500000000000001},"min":{"1min":0.38700000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.77700000000000002,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.56100000000000005,"15min":0.54700000000000004},"min":{"1min":0.35099999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82799999999999996,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":0.57799999999999996}]},{"osd":3,"last update":"Thu Nov 16 16:48:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58999999999999997,"15min":0.57399999999999995},"min":{"1min":0.441,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94099999999999995,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.625},{"interface":"front","average":{"1min":0.62,"5min":0.61599999999999999,"15min":0.60999999999999999},"min":{"1min":0.44500000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.97199999999999998,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.68100000000000005}]},{"osd":4,"last update":"Thu Nov 16 16:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.41999999999999998,"15min":0.40699999999999997},"min":{"1min":0.32700000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.498,"5min":0.45600000000000002,"15min":0.433},"min":{"1min":0.30599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.66100000000000003}]},{"osd":5,"last update":"Thu Nov 16 16:48:20 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.60699999999999998,"15min":0.56899999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.89300000000000002,"5min":1.04,"15min":1.04},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.63300000000000001,"15min":0.58899999999999997},"min":{"1min":0.42399999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.90000000000000002,"5min":1.577,"15min":1.577},"last":0.95899999999999996}]},{"osd":6,"last update":"Thu Nov 16 16:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66800000000000004,"15min":0.67000000000000004},"min":{"1min":0.45700000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.94499999999999995,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.68899999999999995},"min":{"1min":0.47099999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.91300000000000003,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.76200000000000001}]},{"osd":7,"last update":"Thu Nov 16 16:48:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58299999999999996,"15min":0.57499999999999996},"min":{"1min":0.36599999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.90500000000000003,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58499999999999996,"15min":0.57799999999999996},"min":{"1min":0.35199999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.88800000000000001,"5min":1.48,"15min":1.48},"last":0.67000000000000004}]},{"osd":8,"last update":"Thu Nov 16 16:48:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.65600000000000003,"15min":0.63900000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.85399999999999998,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.745},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69499999999999995,"15min":0.69099999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.89200000000000002,"5min":1,"15min":1},"last":0.78800000000000003}]},{"osd":9,"last update":"Thu Nov 16 16:47:56 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.78500000000000003,"15min":0.79100000000000004},"min":{"1min":0.55200000000000005,"5min":0.504,"15min":0.504},"max":{"1min":0.96699999999999997,"5min":1.651,"15min":1.651},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.53300000000000003,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.02,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.80600000000000005}]},{"osd":10,"last update":"Thu Nov 16 16:48:15 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.92200000000000004,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.754,"15min":0.76100000000000001},"min":{"1min":0.53600000000000003,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.96899999999999997,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":1.1279999999999999}]},{"osd":11,"last update":"Thu Nov 16 16:48:26 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":1.095}]}]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7400,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737752,"statfs":{"total":95995035648,"available":95987458048,"internally_reserved":0,"allocated":1015808,"data_stored":497117,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 16:47:56 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.59799999999999998,"15min":0.53400000000000003},"min":{"1min":0.35099999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.96399999999999997,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.60399999999999998,"15min":0.54200000000000004},"min":{"1min":0.378,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91200000000000003,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.61799999999999999}]},{"osd":2,"last update":"Thu Nov 16 16:48:15 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.61499999999999999,"15min":0.58899999999999997},"min":{"1min":0.46899999999999997,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.83299999999999996,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.746},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65100000000000002,"15min":0.59399999999999997},"min":{"1min":0.48299999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.103,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":0.754}]},{"osd":3,"last update":"Thu Nov 16 16:48:27 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.47899999999999998}]},{"osd":4,"last update":"Thu Nov 16 16:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66400000000000003,"15min":0.621},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.96599999999999997,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.65500000000000003,"15min":0.63300000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.87,"5min":1.268,"15min":1.268},"last":0.78100000000000003}]},{"osd":5,"last update":"Thu Nov 16 16:48:20 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.437,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86699999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.73099999999999998,"15min":0.69999999999999996},"min":{"1min":0.54900000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.137,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":0.83499999999999996}]},{"osd":6,"last update":"Thu Nov 16 16:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.69199999999999995,"15min":0.67400000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73399999999999999,"15min":0.70899999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.72499999999999998}]},{"osd":7,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.871},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.552,"5min":1.552,"15min":1.552},"last":0.93000000000000005}]},{"osd":8,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.82199999999999995,"15min":0.81599999999999995},"min":{"1min":0.59199999999999997,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.851,"5min":2.145,"15min":2.145},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.84499999999999997,"15min":0.82499999999999996},"min":{"1min":0.71499999999999997,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.603,"5min":2.125,"15min":2.125},"last":0.89900000000000002}]},{"osd":9,"last update":"Thu Nov 16 16:47:56 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.81799999999999995,"15min":0.83299999999999996},"min":{"1min":0.48099999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.94699999999999995,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.92400000000000004,"15min":0.94899999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.97799999999999998,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.81899999999999995}]},{"osd":10,"last update":"Thu Nov 16 16:48:20 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.95299999999999996,"15min":0.96199999999999997},"min":{"1min":0.54700000000000004,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.5269999999999999,"5min":2.3159999999999998,"15min":2.3159999999999998},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.83899999999999997,"15min":0.84299999999999997},"min":{"1min":0.56599999999999995,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0209999999999999,"5min":1.381,"15min":1.381},"last":0.88100000000000001}]},{"osd":11,"last update":"Thu Nov 16 16:47:37 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":2.2189999999999999,"5min":2.2189999999999999,"15min":2.2189999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":0.96399999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7664,"kb_used_data":1064,"kb_used_omap":0,"kb_used_meta":6592,"kb_avail":93737488,"statfs":{"total":95995035648,"available":95987187712,"internally_reserved":0,"allocated":1089536,"data_stored":573072,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6750208},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.499,"15min":0.46899999999999997},"min":{"1min":0.38800000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.71399999999999997,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.502,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.69599999999999995}]},{"osd":1,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.52700000000000002,"15min":0.50600000000000001},"min":{"1min":0.35599999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.56000000000000005,"15min":0.51600000000000001},"min":{"1min":0.39500000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.81200000000000006}]},{"osd":3,"last update":"Thu Nov 16 16:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.60099999999999998,"15min":0.56699999999999995},"min":{"1min":0.441,"5min":0.38,"15min":0.38},"max":{"1min":0.85599999999999998,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.60599999999999998,"15min":0.57799999999999996},"min":{"1min":0.432,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.85499999999999998,"5min":1.016,"15min":1.016},"last":0.76200000000000001}]},{"osd":4,"last update":"Thu Nov 16 16:48:25 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.79700000000000004}]},{"osd":5,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.44,"15min":0.41399999999999998},"min":{"1min":0.28399999999999997,"5min":0.245,"15min":0.245},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.51600000000000001,"15min":0.47799999999999998},"min":{"1min":0.39400000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.66400000000000003}]},{"osd":6,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.63800000000000001,"15min":0.61599999999999999},"min":{"1min":0.56599999999999995,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.91100000000000003,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.64400000000000002,"15min":0.63300000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.88500000000000001,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.82199999999999995}]},{"osd":7,"last update":"Thu Nov 16 16:48:31 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.877},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.84899999999999998}]},{"osd":8,"last update":"Thu Nov 16 16:48:31 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.61199999999999999,"15min":0.59399999999999997},"min":{"1min":0.53500000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.93899999999999995,"5min":0.997,"15min":0.997},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.63600000000000001,"15min":0.61599999999999999},"min":{"1min":0.47999999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.86499999999999999}]},{"osd":9,"last update":"Thu Nov 16 16:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.72599999999999998,"15min":0.71999999999999997},"min":{"1min":0.55800000000000005,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0029999999999999,"5min":1.006,"15min":1.006},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.53500000000000003,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.88300000000000001,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.92500000000000004}]},{"osd":10,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.68500000000000005,"15min":0.67000000000000004},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.70299999999999996,"15min":0.68600000000000005},"min":{"1min":0.54300000000000004,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.97399999999999998}]},{"osd":11,"last update":"Thu Nov 16 16:48:25 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.89000000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6748,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738404,"statfs":{"total":95995035648,"available":95988125696,"internally_reserved":0,"allocated":1003520,"data_stored":495923,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.433,"15min":0.40300000000000002},"min":{"1min":0.33900000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.436,"15min":0.41399999999999998},"min":{"1min":0.38300000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.41299999999999998}]},{"osd":1,"last update":"Thu Nov 16 16:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.63400000000000001,"15min":0.60599999999999998},"min":{"1min":0.41899999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.64600000000000002,"15min":0.60799999999999998},"min":{"1min":0.46500000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.92400000000000004,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.61299999999999999}]},{"osd":2,"last update":"Thu Nov 16 16:47:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.625},"min":{"1min":0.42699999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89800000000000002,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.63800000000000001,"15min":0.61499999999999999},"min":{"1min":0.45400000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.81699999999999995,"5min":0.871,"15min":0.871},"last":0.59099999999999997}]},{"osd":4,"last update":"Thu Nov 16 16:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.71599999999999997,"15min":0.71399999999999997},"min":{"1min":0.54900000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.748,"15min":0.74199999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.64300000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:48:23 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.72599999999999998,"15min":0.71299999999999997},"min":{"1min":0.50900000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.71799999999999997,"15min":0.71999999999999997},"min":{"1min":0.56000000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.66600000000000004}]},{"osd":6,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61199999999999999,"15min":0.60999999999999999},"min":{"1min":0.42699999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.495,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.75800000000000001}]},{"osd":7,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.76600000000000001,"15min":0.73699999999999999},"min":{"1min":0.497,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.72299999999999998,"15min":0.71699999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.44900000000000001}]},{"osd":8,"last update":"Thu Nov 16 16:46:26 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.81299999999999994}]},{"osd":9,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80600000000000005,"15min":0.80700000000000005},"min":{"1min":0.59699999999999998,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.036,"5min":1.123,"15min":1.123},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.91900000000000004,"15min":0.92400000000000004},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.72299999999999998}]},{"osd":10,"last update":"Thu Nov 16 16:48:17 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.92100000000000004,"15min":0.92600000000000005},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.86199999999999999,"15min":0.86799999999999999},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.149,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.70099999999999996}]},{"osd":11,"last update":"Thu Nov 16 16:48:27 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.71199999999999997}]}]},{"osd":4,"up_from":33,"seq":141733920829,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6620,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738532,"statfs":{"total":95995035648,"available":95988256768,"internally_reserved":0,"allocated":1003520,"data_stored":500721,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.504,"15min":0.49099999999999999},"min":{"1min":0.41299999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.35299999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.41699999999999998}]},{"osd":1,"last update":"Thu Nov 16 16:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.38500000000000001,"15min":0.35899999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.56599999999999995,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.442},{"interface":"front","average":{"1min":0.436,"5min":0.40699999999999997,"15min":0.38800000000000001},"min":{"1min":0.29699999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64000000000000001,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.46999999999999997}]},{"osd":2,"last update":"Thu Nov 16 16:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56100000000000005,"15min":0.54700000000000004},"min":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.623},{"interface":"front","average":{"1min":0.626,"5min":0.55300000000000005,"15min":0.53400000000000003},"min":{"1min":0.34599999999999997,"5min":0.307,"15min":0.307},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.63700000000000001}]},{"osd":3,"last update":"Thu Nov 16 16:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.57999999999999996,"15min":0.54900000000000004},"min":{"1min":0.44700000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85899999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57799999999999996,"15min":0.55800000000000005},"min":{"1min":0.436,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.57999999999999996}]},{"osd":5,"last update":"Thu Nov 16 16:48:22 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.58199999999999996,"15min":0.54900000000000004},"min":{"1min":0.376,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83099999999999996,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58299999999999996,"15min":0.55200000000000005},"min":{"1min":0.38900000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.67800000000000005}]},{"osd":6,"last update":"Thu Nov 16 16:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61699999999999999,"15min":0.60699999999999998},"min":{"1min":0.47899999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.84399999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66200000000000003,"15min":0.65100000000000002},"min":{"1min":0.49399999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.88500000000000001,"5min":0.995,"15min":0.995},"last":0.59099999999999997}]},{"osd":7,"last update":"Thu Nov 16 16:48:08 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51800000000000002,"15min":0.50900000000000001},"min":{"1min":0.34799999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65700000000000003,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.498},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60899999999999999,"15min":0.59799999999999998},"min":{"1min":0.45300000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.82999999999999996,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.51900000000000002}]},{"osd":8,"last update":"Thu Nov 16 16:48:31 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69599999999999995,"15min":0.69499999999999995},"min":{"1min":0.499,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.68400000000000005,"15min":0.68000000000000005},"min":{"1min":0.49199999999999999,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.751}]},{"osd":9,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.77500000000000002,"15min":0.78000000000000003},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.93000000000000005,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.752,"15min":0.75700000000000001},"min":{"1min":0.54300000000000004,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.71299999999999997}]},{"osd":10,"last update":"Thu Nov 16 16:48:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.72099999999999997,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.56999999999999995,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.83799999999999997}]},{"osd":11,"last update":"Thu Nov 16 16:48:26 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.69299999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757305,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6816,"kb_used_data":1048,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738336,"statfs":{"total":95995035648,"available":95988056064,"internally_reserved":0,"allocated":1073152,"data_stored":1101588,"data_compressed":53789,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:21 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53000000000000003,"15min":0.52500000000000002},"min":{"1min":0.36799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.72199999999999998,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52600000000000002,"15min":0.51800000000000002},"min":{"1min":0.38100000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.69799999999999995,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.435}]},{"osd":1,"last update":"Thu Nov 16 16:48:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.53300000000000003,"15min":0.49299999999999999},"min":{"1min":0.47999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76600000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.54200000000000004,"15min":0.504},"min":{"1min":0.46400000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.499}]},{"osd":2,"last update":"Thu Nov 16 16:48:21 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.41399999999999998,"15min":0.39700000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.57199999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.43099999999999999,"15min":0.40699999999999997},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.64100000000000001,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.377}]},{"osd":3,"last update":"Thu Nov 16 16:48:21 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56399999999999995,"15min":0.55200000000000005},"min":{"1min":0.40000000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.79600000000000004,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57099999999999995,"15min":0.55600000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72099999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.48399999999999999}]},{"osd":4,"last update":"Thu Nov 16 16:48:21 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.59999999999999998,"15min":0.56000000000000005},"min":{"1min":0.48299999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83499999999999996,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.61599999999999999,"15min":0.57399999999999995},"min":{"1min":0.51300000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.84299999999999997,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.65100000000000002}]},{"osd":6,"last update":"Thu Nov 16 16:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.626,"15min":0.60999999999999999},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.82699999999999996,"5min":1.052,"15min":1.052},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.63100000000000001,"15min":0.61699999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.86899999999999999,"5min":1.117,"15min":1.117},"last":0.55800000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:48:09 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.60699999999999998,"15min":0.59099999999999997},"min":{"1min":0.52500000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83999999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.62,"15min":0.60599999999999998},"min":{"1min":0.55600000000000005,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.85799999999999998,"5min":1.022,"15min":1.022},"last":0.59499999999999997}]},{"osd":8,"last update":"Thu Nov 16 16:48:34 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.59699999999999998,"15min":0.58399999999999996},"min":{"1min":0.39100000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.90700000000000003,"5min":0.995,"15min":0.995},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.63600000000000001,"15min":0.621},"min":{"1min":0.48999999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.88500000000000001,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.60699999999999998}]},{"osd":9,"last update":"Thu Nov 16 16:47:52 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.73999999999999999,"15min":0.73799999999999999},"min":{"1min":0.53100000000000003,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.94499999999999995,"5min":1.145,"15min":1.145},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.74399999999999999,"15min":0.745},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.89900000000000002,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.66900000000000004}]},{"osd":10,"last update":"Thu Nov 16 16:48:16 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.69099999999999995,"15min":0.67800000000000005},"min":{"1min":0.60799999999999998,"5min":0.437,"15min":0.437},"max":{"1min":0.92000000000000004,"5min":1.097,"15min":1.097},"last":0.629},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.71399999999999997,"15min":0.70199999999999996},"min":{"1min":0.61299999999999999,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.93799999999999994,"5min":1.272,"15min":1.272},"last":0.63900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34505,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34505,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":66685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":78375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":11816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":12474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":68605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T16:48:45.701 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph pg dump --format=json 2023-11-16T16:48:45.764 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:45 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2023-11-16T16:48:45.765 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:45 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2023-11-16T16:48:45.765 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:45 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2023-11-16T16:48:45.765 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:45 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2023-11-16T16:48:45.765 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:45 smithi002 ceph-mon[128712]: osdmap e92: 12 total, 12 up, 12 in 2023-11-16T16:48:45.765 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:45 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:48:45.765 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:45 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:48:45.765 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:45 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:48:45.765 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:45 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:45.765 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:45 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:48:45.765 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:45 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:48:45.765 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:45 smithi002 ceph-mon[128712]: Service mds.all has no related daemon already placed. 2023-11-16T16:48:45.765 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:45 smithi002 ceph-mon[128712]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:48:45.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:45 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2023-11-16T16:48:45.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:45 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2023-11-16T16:48:45.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:45 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2023-11-16T16:48:45.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:45 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2023-11-16T16:48:45.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:45 smithi141 ceph-mon[131184]: osdmap e92: 12 total, 12 up, 12 in 2023-11-16T16:48:45.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:45 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:48:45.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:45 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:48:45.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:45 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:48:45.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:45 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:45.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:45 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:48:45.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:45 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:48:45.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:45 smithi141 ceph-mon[131184]: Service mds.all has no related daemon already placed. 2023-11-16T16:48:45.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:45 smithi141 ceph-mon[131184]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:48:45.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:45 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2023-11-16T16:48:45.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:45 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2023-11-16T16:48:45.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:45 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2023-11-16T16:48:45.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:45 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2023-11-16T16:48:45.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:45 smithi155 ceph-mon[131607]: osdmap e92: 12 total, 12 up, 12 in 2023-11-16T16:48:45.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:45 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:48:45.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:45 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:48:45.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:45 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:48:45.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:45 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' 2023-11-16T16:48:45.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:45 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:48:45.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:45 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:48:45.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:45 smithi155 ceph-mon[131607]: Service mds.all has no related daemon already placed. 2023-11-16T16:48:45.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:45 smithi155 ceph-mon[131607]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:48:46.567 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:46.764 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:46 smithi141 ceph-mon[131184]: osdmap e93: 12 total, 12 up, 12 in 2023-11-16T16:48:46.764 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:46 smithi141 ceph-mon[131184]: pgmap v324: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-11-16T16:48:46.850 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:46 smithi002 ceph-mon[128712]: osdmap e93: 12 total, 12 up, 12 in 2023-11-16T16:48:46.850 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:46 smithi002 ceph-mon[128712]: pgmap v324: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-11-16T16:48:46.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:46 smithi155 ceph-mon[131607]: osdmap e93: 12 total, 12 up, 12 in 2023-11-16T16:48:46.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:46 smithi155 ceph-mon[131607]: pgmap v324: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-11-16T16:48:48.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.108+0000 7f10d068e700 1 -- 172.21.15.2:0/1189229321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10c8100ee0 msgr2=0x7f10c8104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:48.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.108+0000 7f10d068e700 1 --2- 172.21.15.2:0/1189229321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10c8100ee0 0x7f10c8104eb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f10b80099c0 tx=0x7f10b8009cd0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.108+0000 7f10d068e700 1 -- 172.21.15.2:0/1189229321 shutdown_connections 2023-11-16T16:48:48.110 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.108+0000 7f10d068e700 1 --2- 172.21.15.2:0/1189229321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10c8100540 0x7f10c8100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.108+0000 7f10d068e700 1 --2- 172.21.15.2:0/1189229321 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10c81055e0 0x7f10c810d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.108+0000 7f10d068e700 1 --2- 172.21.15.2:0/1189229321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10c8100ee0 0x7f10c8104eb0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.108+0000 7f10d068e700 1 -- 172.21.15.2:0/1189229321 >> 172.21.15.2:0/1189229321 conn(0x7f10c80fbdc0 msgr2=0x7f10c80fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:48.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.108+0000 7f10d068e700 1 -- 172.21.15.2:0/1189229321 shutdown_connections 2023-11-16T16:48:48.111 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.108+0000 7f10d068e700 1 -- 172.21.15.2:0/1189229321 wait complete. 2023-11-16T16:48:48.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.109+0000 7f10d068e700 1 Processor -- start 2023-11-16T16:48:48.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.110+0000 7f10d068e700 1 -- start start 2023-11-16T16:48:48.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.110+0000 7f10d068e700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10c8100540 0x7f10c8072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:48.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.110+0000 7f10d068e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10c8100ee0 0x7f10c8073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:48.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.110+0000 7f10d068e700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10c81055e0 0x7f10c80738b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:48.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.110+0000 7f10d068e700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c81a4810 con 0x7f10c81055e0 2023-11-16T16:48:48.112 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.110+0000 7f10d068e700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c81a4980 con 0x7f10c8100540 2023-11-16T16:48:48.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.110+0000 7f10d068e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c81a4b40 con 0x7f10c8100ee0 2023-11-16T16:48:48.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.110+0000 7f10ce42a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10c8100540 0x7f10c8072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:48.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.110+0000 7f10cdc29700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10c8100ee0 0x7f10c8073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:48.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.110+0000 7f10ce42a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10c8100540 0x7f10c8072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:47476/0 (socket says 172.21.15.2:47476) 2023-11-16T16:48:48.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.110+0000 7f10ce42a700 1 -- 172.21.15.2:0/4276004222 learned_addr learned my addr 172.21.15.2:0/4276004222 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:48.113 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.110+0000 7f10cec2b700 1 --2- 172.21.15.2:0/4276004222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10c81055e0 0x7f10c80738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:48.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.111+0000 7f10ce42a700 1 -- 172.21.15.2:0/4276004222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10c8100ee0 msgr2=0x7f10c8073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:48.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.111+0000 7f10ce42a700 1 --2- 172.21.15.2:0/4276004222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10c8100ee0 0x7f10c8073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.111+0000 7f10ce42a700 1 -- 172.21.15.2:0/4276004222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10c81055e0 msgr2=0x7f10c80738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:48.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.111+0000 7f10ce42a700 1 --2- 172.21.15.2:0/4276004222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10c81055e0 0x7f10c80738b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.114 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.111+0000 7f10ce42a700 1 -- 172.21.15.2:0/4276004222 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b80095f0 con 0x7f10c8100540 2023-11-16T16:48:48.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.111+0000 7f10ce42a700 1 --2- 172.21.15.2:0/4276004222 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10c8100540 0x7f10c8072df0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f10c400e9a0 tx=0x7f10c400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:48.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.111+0000 7f10bf7fe700 1 -- 172.21.15.2:0/4276004222 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c4009790 con 0x7f10c8100540 2023-11-16T16:48:48.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.111+0000 7f10d068e700 1 -- 172.21.15.2:0/4276004222 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c8110310 con 0x7f10c8100540 2023-11-16T16:48:48.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.111+0000 7f10bf7fe700 1 -- 172.21.15.2:0/4276004222 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f10c4010430 con 0x7f10c8100540 2023-11-16T16:48:48.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.112+0000 7f10d068e700 1 -- 172.21.15.2:0/4276004222 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c8110180 con 0x7f10c8100540 2023-11-16T16:48:48.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.112+0000 7f10bf7fe700 1 -- 172.21.15.2:0/4276004222 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c4018680 con 0x7f10c8100540 2023-11-16T16:48:48.115 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.112+0000 7f10d068e700 1 -- 172.21.15.2:0/4276004222 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10ac005320 con 0x7f10c8100540 2023-11-16T16:48:48.116 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.115+0000 7f10bf7fe700 1 -- 172.21.15.2:0/4276004222 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f10c400ca10 con 0x7f10c8100540 2023-11-16T16:48:48.118 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.116+0000 7f10bf7fe700 1 --2- 172.21.15.2:0/4276004222 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f10b40a1260 0x7f10b40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:48.118 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.116+0000 7f10bf7fe700 1 -- 172.21.15.2:0/4276004222 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f10c4014070 con 0x7f10c8100540 2023-11-16T16:48:48.118 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.116+0000 7f10cdc29700 1 --2- 172.21.15.2:0/4276004222 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f10b40a1260 0x7f10b40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:48.118 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.117+0000 7f10cdc29700 1 --2- 172.21.15.2:0/4276004222 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f10b40a1260 0x7f10b40a3710 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f10b8009fd0 tx=0x7f10b8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.118+0000 7f10bf7fe700 1 -- 172.21.15.2:0/4276004222 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f10c40963d0 con 0x7f10c8100540 2023-11-16T16:48:48.323 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.321+0000 7f10d068e700 1 -- 172.21.15.2:0/4276004222 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f10ac000bf0 con 0x7f10b40a1260 2023-11-16T16:48:48.336 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.334+0000 7f10bf7fe700 1 -- 172.21.15.2:0/4276004222 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427352 (secure 0 0 0) 0x7f10ac000bf0 con 0x7f10b40a1260 2023-11-16T16:48:48.343 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:48:48.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.341+0000 7f10d068e700 1 -- 172.21.15.2:0/4276004222 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f10b40a1260 msgr2=0x7f10b40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:48.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.341+0000 7f10d068e700 1 --2- 172.21.15.2:0/4276004222 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f10b40a1260 0x7f10b40a3710 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f10b8009fd0 tx=0x7f10b8005fb0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.341+0000 7f10d068e700 1 -- 172.21.15.2:0/4276004222 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10c8100540 msgr2=0x7f10c8072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:48.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.341+0000 7f10d068e700 1 --2- 172.21.15.2:0/4276004222 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10c8100540 0x7f10c8072df0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f10c400e9a0 tx=0x7f10c400ecb0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.341+0000 7f10d068e700 1 -- 172.21.15.2:0/4276004222 shutdown_connections 2023-11-16T16:48:48.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.341+0000 7f10d068e700 1 --2- 172.21.15.2:0/4276004222 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f10b40a1260 0x7f10b40a3710 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.341+0000 7f10d068e700 1 --2- 172.21.15.2:0/4276004222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10c81055e0 0x7f10c80738b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.341+0000 7f10d068e700 1 --2- 172.21.15.2:0/4276004222 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f10c8100540 0x7f10c8072df0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.341+0000 7f10d068e700 1 --2- 172.21.15.2:0/4276004222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10c8100ee0 0x7f10c8073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:48.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.341+0000 7f10d068e700 1 -- 172.21.15.2:0/4276004222 >> 172.21.15.2:0/4276004222 conn(0x7f10c80fbdc0 msgr2=0x7f10c80fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:48.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.341+0000 7f10d068e700 1 -- 172.21.15.2:0/4276004222 shutdown_connections 2023-11-16T16:48:48.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:48.341+0000 7f10d068e700 1 -- 172.21.15.2:0/4276004222 wait complete. 2023-11-16T16:48:48.348 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2023-11-16T16:48:48.867 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2023-11-16T16:48:47.807742+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":671591,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":479,"num_read_kb":377,"num_write":349,"num_write_kb":3074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180770,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":284,"ondisk_log_size":284,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":421,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":81892,"kb_used_data":12292,"kb_used_omap":0,"kb_used_meta":69504,"kb_avail":1124859932,"statfs":{"total":1151940427776,"available":1151856570368,"internally_reserved":0,"allocated":12587008,"data_stored":8080344,"data_compressed":346938,"data_compressed_allocated":466944,"data_compressed_original":2433024,"omap_allocated":0,"internal_metadata":71172096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":34,"num_keys_recovered":17,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001845"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.500989+0000","last_change":"2023-11-16T16:47:26.367598+0000","last_active":"2023-11-16T16:48:45.500989+0000","last_peered":"2023-11-16T16:48:45.500989+0000","last_clean":"2023-11-16T16:48:45.500989+0000","last_became_active":"2023-11-16T16:47:26.366233+0000","last_became_peered":"2023-11-16T16:47:26.366233+0000","last_unstale":"2023-11-16T16:48:45.500989+0000","last_undegraded":"2023-11-16T16:48:45.500989+0000","last_fullsized":"2023-11-16T16:48:45.500989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:17:03.187194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.845705+0000","last_change":"2023-11-16T16:47:26.371984+0000","last_active":"2023-11-16T16:48:45.845705+0000","last_peered":"2023-11-16T16:48:45.845705+0000","last_clean":"2023-11-16T16:48:45.845705+0000","last_became_active":"2023-11-16T16:47:26.371628+0000","last_became_peered":"2023-11-16T16:47:26.371628+0000","last_unstale":"2023-11-16T16:48:45.845705+0000","last_undegraded":"2023-11-16T16:48:45.845705+0000","last_fullsized":"2023-11-16T16:48:45.845705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:04:13.543055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.845177+0000","last_change":"2023-11-16T16:47:26.374109+0000","last_active":"2023-11-16T16:48:45.845177+0000","last_peered":"2023-11-16T16:48:45.845177+0000","last_clean":"2023-11-16T16:48:45.845177+0000","last_became_active":"2023-11-16T16:47:26.373759+0000","last_became_peered":"2023-11-16T16:47:26.373759+0000","last_unstale":"2023-11-16T16:48:45.845177+0000","last_undegraded":"2023-11-16T16:48:45.845177+0000","last_fullsized":"2023-11-16T16:48:45.845177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:26.099966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855484+0000","last_change":"2023-11-16T16:47:26.378404+0000","last_active":"2023-11-16T16:47:44.855484+0000","last_peered":"2023-11-16T16:47:44.855484+0000","last_clean":"2023-11-16T16:47:44.855484+0000","last_became_active":"2023-11-16T16:47:26.378067+0000","last_became_peered":"2023-11-16T16:47:26.378067+0000","last_unstale":"2023-11-16T16:47:44.855484+0000","last_undegraded":"2023-11-16T16:47:44.855484+0000","last_fullsized":"2023-11-16T16:47:44.855484+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:58:13.078678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.502720+0000","last_change":"2023-11-16T16:47:26.370066+0000","last_active":"2023-11-16T16:48:45.502720+0000","last_peered":"2023-11-16T16:48:45.502720+0000","last_clean":"2023-11-16T16:48:45.502720+0000","last_became_active":"2023-11-16T16:47:26.369789+0000","last_became_peered":"2023-11-16T16:47:26.369789+0000","last_unstale":"2023-11-16T16:48:45.502720+0000","last_undegraded":"2023-11-16T16:48:45.502720+0000","last_fullsized":"2023-11-16T16:48:45.502720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:44:47.174516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472727+0000","last_change":"2023-11-16T16:47:26.379403+0000","last_active":"2023-11-16T16:47:45.472727+0000","last_peered":"2023-11-16T16:47:45.472727+0000","last_clean":"2023-11-16T16:47:45.472727+0000","last_became_active":"2023-11-16T16:47:26.379086+0000","last_became_peered":"2023-11-16T16:47:26.379086+0000","last_unstale":"2023-11-16T16:47:45.472727+0000","last_undegraded":"2023-11-16T16:47:45.472727+0000","last_fullsized":"2023-11-16T16:47:45.472727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:20:51.896082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857929+0000","last_change":"2023-11-16T16:47:26.377555+0000","last_active":"2023-11-16T16:47:44.857929+0000","last_peered":"2023-11-16T16:47:44.857929+0000","last_clean":"2023-11-16T16:47:44.857929+0000","last_became_active":"2023-11-16T16:47:26.377274+0000","last_became_peered":"2023-11-16T16:47:26.377274+0000","last_unstale":"2023-11-16T16:47:44.857929+0000","last_undegraded":"2023-11-16T16:47:44.857929+0000","last_fullsized":"2023-11-16T16:47:44.857929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:40:11.254785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.502414+0000","last_change":"2023-11-16T16:47:26.380842+0000","last_active":"2023-11-16T16:48:45.502414+0000","last_peered":"2023-11-16T16:48:45.502414+0000","last_clean":"2023-11-16T16:48:45.502414+0000","last_became_active":"2023-11-16T16:47:26.380527+0000","last_became_peered":"2023-11-16T16:47:26.380527+0000","last_unstale":"2023-11-16T16:48:45.502414+0000","last_undegraded":"2023-11-16T16:48:45.502414+0000","last_fullsized":"2023-11-16T16:48:45.502414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:20:51.448163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858099+0000","last_change":"2023-11-16T16:47:26.371416+0000","last_active":"2023-11-16T16:47:44.858099+0000","last_peered":"2023-11-16T16:47:44.858099+0000","last_clean":"2023-11-16T16:47:44.858099+0000","last_became_active":"2023-11-16T16:47:26.371096+0000","last_became_peered":"2023-11-16T16:47:26.371096+0000","last_unstale":"2023-11-16T16:47:44.858099+0000","last_undegraded":"2023-11-16T16:47:44.858099+0000","last_fullsized":"2023-11-16T16:47:44.858099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:35:21.284705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473525+0000","last_change":"2023-11-16T16:47:26.380828+0000","last_active":"2023-11-16T16:47:45.473525+0000","last_peered":"2023-11-16T16:47:45.473525+0000","last_clean":"2023-11-16T16:47:45.473525+0000","last_became_active":"2023-11-16T16:47:26.377495+0000","last_became_peered":"2023-11-16T16:47:26.377495+0000","last_unstale":"2023-11-16T16:47:45.473525+0000","last_undegraded":"2023-11-16T16:47:45.473525+0000","last_fullsized":"2023-11-16T16:47:45.473525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:58:53.606878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.859142+0000","last_change":"2023-11-16T16:47:26.374127+0000","last_active":"2023-11-16T16:47:44.859142+0000","last_peered":"2023-11-16T16:47:44.859142+0000","last_clean":"2023-11-16T16:47:44.859142+0000","last_became_active":"2023-11-16T16:47:26.373794+0000","last_became_peered":"2023-11-16T16:47:26.373794+0000","last_unstale":"2023-11-16T16:47:44.859142+0000","last_undegraded":"2023-11-16T16:47:44.859142+0000","last_fullsized":"2023-11-16T16:47:44.859142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:39:54.776347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855500+0000","last_change":"2023-11-16T16:47:26.382530+0000","last_active":"2023-11-16T16:47:44.855500+0000","last_peered":"2023-11-16T16:47:44.855500+0000","last_clean":"2023-11-16T16:47:44.855500+0000","last_became_active":"2023-11-16T16:47:26.382229+0000","last_became_peered":"2023-11-16T16:47:26.382229+0000","last_unstale":"2023-11-16T16:47:44.855500+0000","last_undegraded":"2023-11-16T16:47:44.855500+0000","last_fullsized":"2023-11-16T16:47:44.855500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:28:54.780902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.495706+0000","last_change":"2023-11-16T16:47:45.473174+0000","last_active":"2023-11-16T16:48:45.495706+0000","last_peered":"2023-11-16T16:48:45.495706+0000","last_clean":"2023-11-16T16:48:45.495706+0000","last_became_active":"2023-11-16T16:47:45.472853+0000","last_became_peered":"2023-11-16T16:47:45.472853+0000","last_unstale":"2023-11-16T16:48:45.495706+0000","last_undegraded":"2023-11-16T16:48:45.495706+0000","last_fullsized":"2023-11-16T16:48:45.495706+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:02:29.336535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.845594+0000","last_change":"2023-11-16T16:47:26.375546+0000","last_active":"2023-11-16T16:48:45.845594+0000","last_peered":"2023-11-16T16:48:45.845594+0000","last_clean":"2023-11-16T16:48:45.845594+0000","last_became_active":"2023-11-16T16:47:26.375218+0000","last_became_peered":"2023-11-16T16:47:26.375218+0000","last_unstale":"2023-11-16T16:48:45.845594+0000","last_undegraded":"2023-11-16T16:48:45.845594+0000","last_fullsized":"2023-11-16T16:48:45.845594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:13:08.766342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.844897+0000","last_change":"2023-11-16T16:47:44.860204+0000","last_active":"2023-11-16T16:48:45.844897+0000","last_peered":"2023-11-16T16:48:45.844897+0000","last_clean":"2023-11-16T16:48:45.844897+0000","last_became_active":"2023-11-16T16:47:44.859473+0000","last_became_peered":"2023-11-16T16:47:44.859473+0000","last_unstale":"2023-11-16T16:48:45.844897+0000","last_undegraded":"2023-11-16T16:48:45.844897+0000","last_fullsized":"2023-11-16T16:48:45.844897+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:50:30.600530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857085+0000","last_change":"2023-11-16T16:47:26.379604+0000","last_active":"2023-11-16T16:47:44.857085+0000","last_peered":"2023-11-16T16:47:44.857085+0000","last_clean":"2023-11-16T16:47:44.857085+0000","last_became_active":"2023-11-16T16:47:26.379225+0000","last_became_peered":"2023-11-16T16:47:26.379225+0000","last_unstale":"2023-11-16T16:47:44.857085+0000","last_undegraded":"2023-11-16T16:47:44.857085+0000","last_fullsized":"2023-11-16T16:47:44.857085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:26:54.929229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.501181+0000","last_change":"2023-11-16T16:47:26.367373+0000","last_active":"2023-11-16T16:48:45.501181+0000","last_peered":"2023-11-16T16:48:45.501181+0000","last_clean":"2023-11-16T16:48:45.501181+0000","last_became_active":"2023-11-16T16:47:26.366549+0000","last_became_peered":"2023-11-16T16:47:26.366549+0000","last_unstale":"2023-11-16T16:48:45.501181+0000","last_undegraded":"2023-11-16T16:48:45.501181+0000","last_fullsized":"2023-11-16T16:48:45.501181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:33:00.398860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854006+0000","last_change":"2023-11-16T16:47:26.376291+0000","last_active":"2023-11-16T16:47:44.854006+0000","last_peered":"2023-11-16T16:47:44.854006+0000","last_clean":"2023-11-16T16:47:44.854006+0000","last_became_active":"2023-11-16T16:47:26.374608+0000","last_became_peered":"2023-11-16T16:47:26.374608+0000","last_unstale":"2023-11-16T16:47:44.854006+0000","last_undegraded":"2023-11-16T16:47:44.854006+0000","last_fullsized":"2023-11-16T16:47:44.854006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:17:25.971629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855424+0000","last_change":"2023-11-16T16:47:26.382444+0000","last_active":"2023-11-16T16:47:44.855424+0000","last_peered":"2023-11-16T16:47:44.855424+0000","last_clean":"2023-11-16T16:47:44.855424+0000","last_became_active":"2023-11-16T16:47:26.382130+0000","last_became_peered":"2023-11-16T16:47:26.382130+0000","last_unstale":"2023-11-16T16:47:44.855424+0000","last_undegraded":"2023-11-16T16:47:44.855424+0000","last_fullsized":"2023-11-16T16:47:44.855424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:32:34.208412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854988+0000","last_change":"2023-11-16T16:47:30.838569+0000","last_active":"2023-11-16T16:47:44.854988+0000","last_peered":"2023-11-16T16:47:44.854988+0000","last_clean":"2023-11-16T16:47:44.854988+0000","last_became_active":"2023-11-16T16:47:30.838282+0000","last_became_peered":"2023-11-16T16:47:30.838282+0000","last_unstale":"2023-11-16T16:47:44.854988+0000","last_undegraded":"2023-11-16T16:47:44.854988+0000","last_fullsized":"2023-11-16T16:47:44.854988+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:06:09.584975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.907867+0000","last_change":"2023-11-16T16:47:25.820254+0000","last_active":"2023-11-16T16:47:51.907867+0000","last_peered":"2023-11-16T16:47:51.907867+0000","last_clean":"2023-11-16T16:47:51.907867+0000","last_became_active":"2023-11-16T16:47:25.819924+0000","last_became_peered":"2023-11-16T16:47:25.819924+0000","last_unstale":"2023-11-16T16:47:51.907867+0000","last_undegraded":"2023-11-16T16:47:51.907867+0000","last_fullsized":"2023-11-16T16:47:51.907867+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:50:37.572954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.503110+0000","last_change":"2023-11-16T16:47:26.374560+0000","last_active":"2023-11-16T16:48:45.503110+0000","last_peered":"2023-11-16T16:48:45.503110+0000","last_clean":"2023-11-16T16:48:45.503110+0000","last_became_active":"2023-11-16T16:47:26.373554+0000","last_became_peered":"2023-11-16T16:47:26.373554+0000","last_unstale":"2023-11-16T16:48:45.503110+0000","last_undegraded":"2023-11-16T16:48:45.503110+0000","last_fullsized":"2023-11-16T16:48:45.503110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:38:15.241991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854083+0000","last_change":"2023-11-16T16:47:30.826047+0000","last_active":"2023-11-16T16:47:44.854083+0000","last_peered":"2023-11-16T16:47:44.854083+0000","last_clean":"2023-11-16T16:47:44.854083+0000","last_became_active":"2023-11-16T16:47:30.825696+0000","last_became_peered":"2023-11-16T16:47:30.825696+0000","last_unstale":"2023-11-16T16:47:44.854083+0000","last_undegraded":"2023-11-16T16:47:44.854083+0000","last_fullsized":"2023-11-16T16:47:44.854083+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:31:58.343449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.503181+0000","last_change":"2023-11-16T16:47:25.830763+0000","last_active":"2023-11-16T16:48:45.503181+0000","last_peered":"2023-11-16T16:48:45.503181+0000","last_clean":"2023-11-16T16:48:45.503181+0000","last_became_active":"2023-11-16T16:47:25.830405+0000","last_became_peered":"2023-11-16T16:47:25.830405+0000","last_unstale":"2023-11-16T16:48:45.503181+0000","last_undegraded":"2023-11-16T16:48:45.503181+0000","last_fullsized":"2023-11-16T16:48:45.503181+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:22:11.766366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.845473+0000","last_change":"2023-11-16T16:47:26.385897+0000","last_active":"2023-11-16T16:48:45.845473+0000","last_peered":"2023-11-16T16:48:45.845473+0000","last_clean":"2023-11-16T16:48:45.845473+0000","last_became_active":"2023-11-16T16:47:26.382557+0000","last_became_peered":"2023-11-16T16:47:26.382557+0000","last_unstale":"2023-11-16T16:48:45.845473+0000","last_undegraded":"2023-11-16T16:48:45.845473+0000","last_fullsized":"2023-11-16T16:48:45.845473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:29:19.195510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854309+0000","last_change":"2023-11-16T16:47:30.828521+0000","last_active":"2023-11-16T16:47:44.854309+0000","last_peered":"2023-11-16T16:47:44.854309+0000","last_clean":"2023-11-16T16:47:44.854309+0000","last_became_active":"2023-11-16T16:47:30.828135+0000","last_became_peered":"2023-11-16T16:47:30.828135+0000","last_unstale":"2023-11-16T16:47:44.854309+0000","last_undegraded":"2023-11-16T16:47:44.854309+0000","last_fullsized":"2023-11-16T16:47:44.854309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:19:57.561985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.502761+0000","last_change":"2023-11-16T16:47:25.819576+0000","last_active":"2023-11-16T16:48:45.502761+0000","last_peered":"2023-11-16T16:48:45.502761+0000","last_clean":"2023-11-16T16:48:45.502761+0000","last_became_active":"2023-11-16T16:47:25.818354+0000","last_became_peered":"2023-11-16T16:47:25.818354+0000","last_unstale":"2023-11-16T16:48:45.502761+0000","last_undegraded":"2023-11-16T16:48:45.502761+0000","last_fullsized":"2023-11-16T16:48:45.502761+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:58:46.834637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857839+0000","last_change":"2023-11-16T16:47:26.364632+0000","last_active":"2023-11-16T16:47:44.857839+0000","last_peered":"2023-11-16T16:47:44.857839+0000","last_clean":"2023-11-16T16:47:44.857839+0000","last_became_active":"2023-11-16T16:47:26.364288+0000","last_became_peered":"2023-11-16T16:47:26.364288+0000","last_unstale":"2023-11-16T16:47:44.857839+0000","last_undegraded":"2023-11-16T16:47:44.857839+0000","last_fullsized":"2023-11-16T16:47:44.857839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:17:52.757571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854072+0000","last_change":"2023-11-16T16:47:30.828722+0000","last_active":"2023-11-16T16:47:44.854072+0000","last_peered":"2023-11-16T16:47:44.854072+0000","last_clean":"2023-11-16T16:47:44.854072+0000","last_became_active":"2023-11-16T16:47:30.828299+0000","last_became_peered":"2023-11-16T16:47:30.828299+0000","last_unstale":"2023-11-16T16:47:44.854072+0000","last_undegraded":"2023-11-16T16:47:44.854072+0000","last_fullsized":"2023-11-16T16:47:44.854072+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:33:38.194438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.502238+0000","last_change":"2023-11-16T16:47:25.817030+0000","last_active":"2023-11-16T16:48:45.502238+0000","last_peered":"2023-11-16T16:48:45.502238+0000","last_clean":"2023-11-16T16:48:45.502238+0000","last_became_active":"2023-11-16T16:47:25.816710+0000","last_became_peered":"2023-11-16T16:47:25.816710+0000","last_unstale":"2023-11-16T16:48:45.502238+0000","last_undegraded":"2023-11-16T16:48:45.502238+0000","last_fullsized":"2023-11-16T16:48:45.502238+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:55:39.364696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473673+0000","last_change":"2023-11-16T16:47:26.371226+0000","last_active":"2023-11-16T16:47:45.473673+0000","last_peered":"2023-11-16T16:47:45.473673+0000","last_clean":"2023-11-16T16:47:45.473673+0000","last_became_active":"2023-11-16T16:47:26.370925+0000","last_became_peered":"2023-11-16T16:47:26.370925+0000","last_unstale":"2023-11-16T16:47:45.473673+0000","last_undegraded":"2023-11-16T16:47:45.473673+0000","last_fullsized":"2023-11-16T16:47:45.473673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:48:12.574242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855221+0000","last_change":"2023-11-16T16:47:30.821364+0000","last_active":"2023-11-16T16:47:44.855221+0000","last_peered":"2023-11-16T16:47:44.855221+0000","last_clean":"2023-11-16T16:47:44.855221+0000","last_became_active":"2023-11-16T16:47:30.820938+0000","last_became_peered":"2023-11-16T16:47:30.820938+0000","last_unstale":"2023-11-16T16:47:44.855221+0000","last_undegraded":"2023-11-16T16:47:44.855221+0000","last_fullsized":"2023-11-16T16:47:44.855221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:13:28.692462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'9","reported_seq":91,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:41.989059+0000","last_change":"2023-11-16T16:47:25.818612+0000","last_active":"2023-11-16T16:48:41.989059+0000","last_peered":"2023-11-16T16:48:41.989059+0000","last_clean":"2023-11-16T16:48:41.989059+0000","last_became_active":"2023-11-16T16:47:25.818272+0000","last_became_peered":"2023-11-16T16:47:25.818272+0000","last_unstale":"2023-11-16T16:48:41.989059+0000","last_undegraded":"2023-11-16T16:48:41.989059+0000","last_fullsized":"2023-11-16T16:48:41.989059+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:00:33.166271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10962,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":14,"num_write":6,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.502234+0000","last_change":"2023-11-16T16:47:26.372245+0000","last_active":"2023-11-16T16:48:45.502234+0000","last_peered":"2023-11-16T16:48:45.502234+0000","last_clean":"2023-11-16T16:48:45.502234+0000","last_became_active":"2023-11-16T16:47:26.371942+0000","last_became_peered":"2023-11-16T16:47:26.371942+0000","last_unstale":"2023-11-16T16:48:45.502234+0000","last_undegraded":"2023-11-16T16:48:45.502234+0000","last_fullsized":"2023-11-16T16:48:45.502234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:49:37.947569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855696+0000","last_change":"2023-11-16T16:47:30.839669+0000","last_active":"2023-11-16T16:47:44.855696+0000","last_peered":"2023-11-16T16:47:44.855696+0000","last_clean":"2023-11-16T16:47:44.855696+0000","last_became_active":"2023-11-16T16:47:30.839165+0000","last_became_peered":"2023-11-16T16:47:30.839165+0000","last_unstale":"2023-11-16T16:47:44.855696+0000","last_undegraded":"2023-11-16T16:47:44.855696+0000","last_fullsized":"2023-11-16T16:47:44.855696+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:26:12.940109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.905121+0000","last_change":"2023-11-16T16:47:25.819867+0000","last_active":"2023-11-16T16:47:51.905121+0000","last_peered":"2023-11-16T16:47:51.905121+0000","last_clean":"2023-11-16T16:47:51.905121+0000","last_became_active":"2023-11-16T16:47:25.819516+0000","last_became_peered":"2023-11-16T16:47:25.819516+0000","last_unstale":"2023-11-16T16:47:51.905121+0000","last_undegraded":"2023-11-16T16:47:51.905121+0000","last_fullsized":"2023-11-16T16:47:51.905121+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:23:15.640449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473687+0000","last_change":"2023-11-16T16:47:26.383679+0000","last_active":"2023-11-16T16:47:45.473687+0000","last_peered":"2023-11-16T16:47:45.473687+0000","last_clean":"2023-11-16T16:47:45.473687+0000","last_became_active":"2023-11-16T16:47:26.383370+0000","last_became_peered":"2023-11-16T16:47:26.383370+0000","last_unstale":"2023-11-16T16:47:45.473687+0000","last_undegraded":"2023-11-16T16:47:45.473687+0000","last_fullsized":"2023-11-16T16:47:45.473687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:56:40.615856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.500643+0000","last_change":"2023-11-16T16:47:30.832845+0000","last_active":"2023-11-16T16:48:45.500643+0000","last_peered":"2023-11-16T16:48:45.500643+0000","last_clean":"2023-11-16T16:48:45.500643+0000","last_became_active":"2023-11-16T16:47:30.832364+0000","last_became_peered":"2023-11-16T16:47:30.832364+0000","last_unstale":"2023-11-16T16:48:45.500643+0000","last_undegraded":"2023-11-16T16:48:45.500643+0000","last_fullsized":"2023-11-16T16:48:45.500643+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:44:33.296292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.495224+0000","last_change":"2023-11-16T16:47:25.817420+0000","last_active":"2023-11-16T16:48:45.495224+0000","last_peered":"2023-11-16T16:48:45.495224+0000","last_clean":"2023-11-16T16:48:45.495224+0000","last_became_active":"2023-11-16T16:47:25.816160+0000","last_became_peered":"2023-11-16T16:47:25.816160+0000","last_unstale":"2023-11-16T16:48:45.495224+0000","last_undegraded":"2023-11-16T16:48:45.495224+0000","last_fullsized":"2023-11-16T16:48:45.495224+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:47:25.768025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.502654+0000","last_change":"2023-11-16T16:47:26.370729+0000","last_active":"2023-11-16T16:48:45.502654+0000","last_peered":"2023-11-16T16:48:45.502654+0000","last_clean":"2023-11-16T16:48:45.502654+0000","last_became_active":"2023-11-16T16:47:26.370435+0000","last_became_peered":"2023-11-16T16:47:26.370435+0000","last_unstale":"2023-11-16T16:48:45.502654+0000","last_undegraded":"2023-11-16T16:48:45.502654+0000","last_fullsized":"2023-11-16T16:48:45.502654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:33:23.983330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.845567+0000","last_change":"2023-11-16T16:47:30.832895+0000","last_active":"2023-11-16T16:48:45.845567+0000","last_peered":"2023-11-16T16:48:45.845567+0000","last_clean":"2023-11-16T16:48:45.845567+0000","last_became_active":"2023-11-16T16:47:30.832192+0000","last_became_peered":"2023-11-16T16:47:30.832192+0000","last_unstale":"2023-11-16T16:48:45.845567+0000","last_undegraded":"2023-11-16T16:48:45.845567+0000","last_fullsized":"2023-11-16T16:48:45.845567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:56:20.002988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473152+0000","last_change":"2023-11-16T16:47:25.815673+0000","last_active":"2023-11-16T16:47:45.473152+0000","last_peered":"2023-11-16T16:47:45.473152+0000","last_clean":"2023-11-16T16:47:45.473152+0000","last_became_active":"2023-11-16T16:47:25.815362+0000","last_became_peered":"2023-11-16T16:47:25.815362+0000","last_unstale":"2023-11-16T16:47:45.473152+0000","last_undegraded":"2023-11-16T16:47:45.473152+0000","last_fullsized":"2023-11-16T16:47:45.473152+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:50:06.110785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.853033+0000","last_change":"2023-11-16T16:47:26.383483+0000","last_active":"2023-11-16T16:47:44.853033+0000","last_peered":"2023-11-16T16:47:44.853033+0000","last_clean":"2023-11-16T16:47:44.853033+0000","last_became_active":"2023-11-16T16:47:26.383177+0000","last_became_peered":"2023-11-16T16:47:26.383177+0000","last_unstale":"2023-11-16T16:47:44.853033+0000","last_undegraded":"2023-11-16T16:47:44.853033+0000","last_fullsized":"2023-11-16T16:47:44.853033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:27:38.101289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.502908+0000","last_change":"2023-11-16T16:47:30.829044+0000","last_active":"2023-11-16T16:48:45.502908+0000","last_peered":"2023-11-16T16:48:45.502908+0000","last_clean":"2023-11-16T16:48:45.502908+0000","last_became_active":"2023-11-16T16:47:30.827398+0000","last_became_peered":"2023-11-16T16:47:30.827398+0000","last_unstale":"2023-11-16T16:48:45.502908+0000","last_undegraded":"2023-11-16T16:48:45.502908+0000","last_fullsized":"2023-11-16T16:48:45.502908+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:26:40.727203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:42.990133+0000","last_change":"2023-11-16T16:47:25.817461+0000","last_active":"2023-11-16T16:48:42.990133+0000","last_peered":"2023-11-16T16:48:42.990133+0000","last_clean":"2023-11-16T16:48:42.990133+0000","last_became_active":"2023-11-16T16:47:25.817135+0000","last_became_peered":"2023-11-16T16:47:25.817135+0000","last_unstale":"2023-11-16T16:48:42.990133+0000","last_undegraded":"2023-11-16T16:48:42.990133+0000","last_fullsized":"2023-11-16T16:48:42.990133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:00:21.140226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858098+0000","last_change":"2023-11-16T16:47:26.380218+0000","last_active":"2023-11-16T16:47:44.858098+0000","last_peered":"2023-11-16T16:47:44.858098+0000","last_clean":"2023-11-16T16:47:44.858098+0000","last_became_active":"2023-11-16T16:47:26.379849+0000","last_became_peered":"2023-11-16T16:47:26.379849+0000","last_unstale":"2023-11-16T16:47:44.858098+0000","last_undegraded":"2023-11-16T16:47:44.858098+0000","last_fullsized":"2023-11-16T16:47:44.858098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:16:44.031031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.852960+0000","last_change":"2023-11-16T16:47:30.836464+0000","last_active":"2023-11-16T16:47:44.852960+0000","last_peered":"2023-11-16T16:47:44.852960+0000","last_clean":"2023-11-16T16:47:44.852960+0000","last_became_active":"2023-11-16T16:47:30.835185+0000","last_became_peered":"2023-11-16T16:47:30.835185+0000","last_unstale":"2023-11-16T16:47:44.852960+0000","last_undegraded":"2023-11-16T16:47:44.852960+0000","last_fullsized":"2023-11-16T16:47:44.852960+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:00:34.542077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.898666+0000","last_change":"2023-11-16T16:47:25.816583+0000","last_active":"2023-11-16T16:47:50.898666+0000","last_peered":"2023-11-16T16:47:50.898666+0000","last_clean":"2023-11-16T16:47:50.898666+0000","last_became_active":"2023-11-16T16:47:25.816286+0000","last_became_peered":"2023-11-16T16:47:25.816286+0000","last_unstale":"2023-11-16T16:47:50.898666+0000","last_undegraded":"2023-11-16T16:47:50.898666+0000","last_fullsized":"2023-11-16T16:47:50.898666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:55:47.335416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.845754+0000","last_change":"2023-11-16T16:47:26.379964+0000","last_active":"2023-11-16T16:48:45.845754+0000","last_peered":"2023-11-16T16:48:45.845754+0000","last_clean":"2023-11-16T16:48:45.845754+0000","last_became_active":"2023-11-16T16:47:26.376857+0000","last_became_peered":"2023-11-16T16:47:26.376857+0000","last_unstale":"2023-11-16T16:48:45.845754+0000","last_undegraded":"2023-11-16T16:48:45.845754+0000","last_fullsized":"2023-11-16T16:48:45.845754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:31:29.957221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854625+0000","last_change":"2023-11-16T16:47:30.832572+0000","last_active":"2023-11-16T16:47:44.854625+0000","last_peered":"2023-11-16T16:47:44.854625+0000","last_clean":"2023-11-16T16:47:44.854625+0000","last_became_active":"2023-11-16T16:47:30.832138+0000","last_became_peered":"2023-11-16T16:47:30.832138+0000","last_unstale":"2023-11-16T16:47:44.854625+0000","last_undegraded":"2023-11-16T16:47:44.854625+0000","last_fullsized":"2023-11-16T16:47:44.854625+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:25:20.590832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.900690+0000","last_change":"2023-11-16T16:47:25.826848+0000","last_active":"2023-11-16T16:47:50.900690+0000","last_peered":"2023-11-16T16:47:50.900690+0000","last_clean":"2023-11-16T16:47:50.900690+0000","last_became_active":"2023-11-16T16:47:25.826532+0000","last_became_peered":"2023-11-16T16:47:25.826532+0000","last_unstale":"2023-11-16T16:47:50.900690+0000","last_undegraded":"2023-11-16T16:47:50.900690+0000","last_fullsized":"2023-11-16T16:47:50.900690+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:34:36.051545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.863649+0000","last_change":"2023-11-16T16:47:44.863649+0000","last_active":"2023-11-16T16:47:44.863649+0000","last_peered":"2023-11-16T16:47:44.863649+0000","last_clean":"2023-11-16T16:47:44.863649+0000","last_became_active":"2023-11-16T16:47:44.863350+0000","last_became_peered":"2023-11-16T16:47:44.863350+0000","last_unstale":"2023-11-16T16:47:44.863649+0000","last_undegraded":"2023-11-16T16:47:44.863649+0000","last_fullsized":"2023-11-16T16:47:44.863649+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:45:32.381225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.853019+0000","last_change":"2023-11-16T16:47:30.829074+0000","last_active":"2023-11-16T16:47:44.853019+0000","last_peered":"2023-11-16T16:47:44.853019+0000","last_clean":"2023-11-16T16:47:44.853019+0000","last_became_active":"2023-11-16T16:47:30.828688+0000","last_became_peered":"2023-11-16T16:47:30.828688+0000","last_unstale":"2023-11-16T16:47:44.853019+0000","last_undegraded":"2023-11-16T16:47:44.853019+0000","last_fullsized":"2023-11-16T16:47:44.853019+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:56:56.319865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'35","reported_seq":173,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:43.991507+0000","last_change":"2023-11-16T16:47:25.816469+0000","last_active":"2023-11-16T16:48:43.991507+0000","last_peered":"2023-11-16T16:48:43.991507+0000","last_clean":"2023-11-16T16:48:43.991507+0000","last_became_active":"2023-11-16T16:47:25.816115+0000","last_became_peered":"2023-11-16T16:47:25.816115+0000","last_unstale":"2023-11-16T16:48:43.991507+0000","last_undegraded":"2023-11-16T16:48:43.991507+0000","last_fullsized":"2023-11-16T16:48:43.991507+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:33:32.434023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66685,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":112,"num_read_kb":121,"num_write":36,"num_write_kb":83,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.502399+0000","last_change":"2023-11-16T16:47:26.373382+0000","last_active":"2023-11-16T16:48:45.502399+0000","last_peered":"2023-11-16T16:48:45.502399+0000","last_clean":"2023-11-16T16:48:45.502399+0000","last_became_active":"2023-11-16T16:47:26.372951+0000","last_became_peered":"2023-11-16T16:47:26.372951+0000","last_unstale":"2023-11-16T16:48:45.502399+0000","last_undegraded":"2023-11-16T16:48:45.502399+0000","last_fullsized":"2023-11-16T16:48:45.502399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:40:50.759648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473826+0000","last_change":"2023-11-16T16:47:30.833890+0000","last_active":"2023-11-16T16:47:45.473826+0000","last_peered":"2023-11-16T16:47:45.473826+0000","last_clean":"2023-11-16T16:47:45.473826+0000","last_became_active":"2023-11-16T16:47:30.833489+0000","last_became_peered":"2023-11-16T16:47:30.833489+0000","last_unstale":"2023-11-16T16:47:45.473826+0000","last_undegraded":"2023-11-16T16:47:45.473826+0000","last_fullsized":"2023-11-16T16:47:45.473826+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:21:14.794417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.516253+0000","last_change":"2023-11-16T16:48:45.515866+0000","last_active":"2023-11-16T16:48:45.516253+0000","last_peered":"2023-11-16T16:48:45.516253+0000","last_clean":"2023-11-16T16:48:45.516253+0000","last_became_active":"2023-11-16T16:48:45.505352+0000","last_became_peered":"2023-11-16T16:48:45.505352+0000","last_unstale":"2023-11-16T16:48:45.516253+0000","last_undegraded":"2023-11-16T16:48:45.516253+0000","last_fullsized":"2023-11-16T16:48:45.516253+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:04:16.458976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.495489+0000","last_change":"2023-11-16T16:47:26.378190+0000","last_active":"2023-11-16T16:48:45.495489+0000","last_peered":"2023-11-16T16:48:45.495489+0000","last_clean":"2023-11-16T16:48:45.495489+0000","last_became_active":"2023-11-16T16:47:26.374581+0000","last_became_peered":"2023-11-16T16:47:26.374581+0000","last_unstale":"2023-11-16T16:48:45.495489+0000","last_undegraded":"2023-11-16T16:48:45.495489+0000","last_fullsized":"2023-11-16T16:48:45.495489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:44:54.825977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.499984+0000","last_change":"2023-11-16T16:47:30.826637+0000","last_active":"2023-11-16T16:48:45.499984+0000","last_peered":"2023-11-16T16:48:45.499984+0000","last_clean":"2023-11-16T16:48:45.499984+0000","last_became_active":"2023-11-16T16:47:30.826156+0000","last_became_peered":"2023-11-16T16:47:30.826156+0000","last_unstale":"2023-11-16T16:48:45.499984+0000","last_undegraded":"2023-11-16T16:48:45.499984+0000","last_fullsized":"2023-11-16T16:48:45.499984+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:41:17.744276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.901236+0000","last_change":"2023-11-16T16:47:25.822749+0000","last_active":"2023-11-16T16:47:50.901236+0000","last_peered":"2023-11-16T16:47:50.901236+0000","last_clean":"2023-11-16T16:47:50.901236+0000","last_became_active":"2023-11-16T16:47:25.822441+0000","last_became_peered":"2023-11-16T16:47:25.822441+0000","last_unstale":"2023-11-16T16:47:50.901236+0000","last_undegraded":"2023-11-16T16:47:50.901236+0000","last_fullsized":"2023-11-16T16:47:50.901236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:42:27.484194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858444+0000","last_change":"2023-11-16T16:47:26.375131+0000","last_active":"2023-11-16T16:47:44.858444+0000","last_peered":"2023-11-16T16:47:44.858444+0000","last_clean":"2023-11-16T16:47:44.858444+0000","last_became_active":"2023-11-16T16:47:26.374740+0000","last_became_peered":"2023-11-16T16:47:26.374740+0000","last_unstale":"2023-11-16T16:47:44.858444+0000","last_undegraded":"2023-11-16T16:47:44.858444+0000","last_fullsized":"2023-11-16T16:47:44.858444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:58:23.761900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.853864+0000","last_change":"2023-11-16T16:47:30.836806+0000","last_active":"2023-11-16T16:47:44.853864+0000","last_peered":"2023-11-16T16:47:44.853864+0000","last_clean":"2023-11-16T16:47:44.853864+0000","last_became_active":"2023-11-16T16:47:30.835299+0000","last_became_peered":"2023-11-16T16:47:30.835299+0000","last_unstale":"2023-11-16T16:47:44.853864+0000","last_undegraded":"2023-11-16T16:47:44.853864+0000","last_fullsized":"2023-11-16T16:47:44.853864+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:53:26.843235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":85,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:42.990174+0000","last_change":"2023-11-16T16:47:25.816793+0000","last_active":"2023-11-16T16:48:42.990174+0000","last_peered":"2023-11-16T16:48:42.990174+0000","last_clean":"2023-11-16T16:48:42.990174+0000","last_became_active":"2023-11-16T16:47:25.816433+0000","last_became_peered":"2023-11-16T16:47:25.816433+0000","last_unstale":"2023-11-16T16:48:42.990174+0000","last_undegraded":"2023-11-16T16:48:42.990174+0000","last_fullsized":"2023-11-16T16:48:42.990174+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:30:24.755449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858132+0000","last_change":"2023-11-16T16:47:26.375383+0000","last_active":"2023-11-16T16:47:44.858132+0000","last_peered":"2023-11-16T16:47:44.858132+0000","last_clean":"2023-11-16T16:47:44.858132+0000","last_became_active":"2023-11-16T16:47:26.375026+0000","last_became_peered":"2023-11-16T16:47:26.375026+0000","last_unstale":"2023-11-16T16:47:44.858132+0000","last_undegraded":"2023-11-16T16:47:44.858132+0000","last_fullsized":"2023-11-16T16:47:44.858132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:07:01.547781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.495903+0000","last_change":"2023-11-16T16:47:45.475115+0000","last_active":"2023-11-16T16:48:45.495903+0000","last_peered":"2023-11-16T16:48:45.495903+0000","last_clean":"2023-11-16T16:48:45.495903+0000","last_became_active":"2023-11-16T16:47:45.474475+0000","last_became_peered":"2023-11-16T16:47:45.474475+0000","last_unstale":"2023-11-16T16:48:45.495903+0000","last_undegraded":"2023-11-16T16:48:45.495903+0000","last_fullsized":"2023-11-16T16:48:45.495903+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:44:05.426707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.502126+0000","last_change":"2023-11-16T16:47:25.826911+0000","last_active":"2023-11-16T16:48:45.502126+0000","last_peered":"2023-11-16T16:48:45.502126+0000","last_clean":"2023-11-16T16:48:45.502126+0000","last_became_active":"2023-11-16T16:47:25.826605+0000","last_became_peered":"2023-11-16T16:47:25.826605+0000","last_unstale":"2023-11-16T16:48:45.502126+0000","last_undegraded":"2023-11-16T16:48:45.502126+0000","last_fullsized":"2023-11-16T16:48:45.502126+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:44:18.287021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472956+0000","last_change":"2023-11-16T16:47:26.376072+0000","last_active":"2023-11-16T16:47:45.472956+0000","last_peered":"2023-11-16T16:47:45.472956+0000","last_clean":"2023-11-16T16:47:45.472956+0000","last_became_active":"2023-11-16T16:47:26.375585+0000","last_became_peered":"2023-11-16T16:47:26.375585+0000","last_unstale":"2023-11-16T16:47:45.472956+0000","last_undegraded":"2023-11-16T16:47:45.472956+0000","last_fullsized":"2023-11-16T16:47:45.472956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:30:44.970814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472838+0000","last_change":"2023-11-16T16:47:30.834558+0000","last_active":"2023-11-16T16:47:45.472838+0000","last_peered":"2023-11-16T16:47:45.472838+0000","last_clean":"2023-11-16T16:47:45.472838+0000","last_became_active":"2023-11-16T16:47:30.833989+0000","last_became_peered":"2023-11-16T16:47:30.833989+0000","last_unstale":"2023-11-16T16:47:45.472838+0000","last_undegraded":"2023-11-16T16:47:45.472838+0000","last_fullsized":"2023-11-16T16:47:45.472838+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:54:42.983565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.503184+0000","last_change":"2023-11-16T16:47:25.817016+0000","last_active":"2023-11-16T16:48:45.503184+0000","last_peered":"2023-11-16T16:48:45.503184+0000","last_clean":"2023-11-16T16:48:45.503184+0000","last_became_active":"2023-11-16T16:47:25.816752+0000","last_became_peered":"2023-11-16T16:47:25.816752+0000","last_unstale":"2023-11-16T16:48:45.503184+0000","last_undegraded":"2023-11-16T16:48:45.503184+0000","last_fullsized":"2023-11-16T16:48:45.503184+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:43:55.777893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.500689+0000","last_change":"2023-11-16T16:47:26.374947+0000","last_active":"2023-11-16T16:48:45.500689+0000","last_peered":"2023-11-16T16:48:45.500689+0000","last_clean":"2023-11-16T16:48:45.500689+0000","last_became_active":"2023-11-16T16:47:26.374533+0000","last_became_peered":"2023-11-16T16:47:26.374533+0000","last_unstale":"2023-11-16T16:48:45.500689+0000","last_undegraded":"2023-11-16T16:48:45.500689+0000","last_fullsized":"2023-11-16T16:48:45.500689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:38:17.935054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.502129+0000","last_change":"2023-11-16T16:47:30.830506+0000","last_active":"2023-11-16T16:48:45.502129+0000","last_peered":"2023-11-16T16:48:45.502129+0000","last_clean":"2023-11-16T16:48:45.502129+0000","last_became_active":"2023-11-16T16:47:30.830086+0000","last_became_peered":"2023-11-16T16:47:30.830086+0000","last_unstale":"2023-11-16T16:48:45.502129+0000","last_undegraded":"2023-11-16T16:48:45.502129+0000","last_fullsized":"2023-11-16T16:48:45.502129+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:31:53.418225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.853657+0000","last_change":"2023-11-16T16:47:25.830104+0000","last_active":"2023-11-16T16:47:44.853657+0000","last_peered":"2023-11-16T16:47:44.853657+0000","last_clean":"2023-11-16T16:47:44.853657+0000","last_became_active":"2023-11-16T16:47:25.829232+0000","last_became_peered":"2023-11-16T16:47:25.829232+0000","last_unstale":"2023-11-16T16:47:44.853657+0000","last_undegraded":"2023-11-16T16:47:44.853657+0000","last_fullsized":"2023-11-16T16:47:44.853657+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:29:15.724324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473108+0000","last_change":"2023-11-16T16:47:26.368918+0000","last_active":"2023-11-16T16:47:45.473108+0000","last_peered":"2023-11-16T16:47:45.473108+0000","last_clean":"2023-11-16T16:47:45.473108+0000","last_became_active":"2023-11-16T16:47:26.368592+0000","last_became_peered":"2023-11-16T16:47:26.368592+0000","last_unstale":"2023-11-16T16:47:45.473108+0000","last_undegraded":"2023-11-16T16:47:45.473108+0000","last_fullsized":"2023-11-16T16:47:45.473108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:09:36.266152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.495810+0000","last_change":"2023-11-16T16:47:30.830558+0000","last_active":"2023-11-16T16:48:45.495810+0000","last_peered":"2023-11-16T16:48:45.495810+0000","last_clean":"2023-11-16T16:48:45.495810+0000","last_became_active":"2023-11-16T16:47:30.829777+0000","last_became_peered":"2023-11-16T16:47:30.829777+0000","last_unstale":"2023-11-16T16:48:45.495810+0000","last_undegraded":"2023-11-16T16:48:45.495810+0000","last_fullsized":"2023-11-16T16:48:45.495810+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:12:59.549507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858408+0000","last_change":"2023-11-16T16:47:25.820315+0000","last_active":"2023-11-16T16:47:44.858408+0000","last_peered":"2023-11-16T16:47:44.858408+0000","last_clean":"2023-11-16T16:47:44.858408+0000","last_became_active":"2023-11-16T16:47:25.820032+0000","last_became_peered":"2023-11-16T16:47:25.820032+0000","last_unstale":"2023-11-16T16:47:44.858408+0000","last_undegraded":"2023-11-16T16:47:44.858408+0000","last_fullsized":"2023-11-16T16:47:44.858408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:36:13.332902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472754+0000","last_change":"2023-11-16T16:47:26.379841+0000","last_active":"2023-11-16T16:47:45.472754+0000","last_peered":"2023-11-16T16:47:45.472754+0000","last_clean":"2023-11-16T16:47:45.472754+0000","last_became_active":"2023-11-16T16:47:26.377401+0000","last_became_peered":"2023-11-16T16:47:26.377401+0000","last_unstale":"2023-11-16T16:47:45.472754+0000","last_undegraded":"2023-11-16T16:47:45.472754+0000","last_fullsized":"2023-11-16T16:47:45.472754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:21:55.430094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.495877+0000","last_change":"2023-11-16T16:47:45.473595+0000","last_active":"2023-11-16T16:48:45.495877+0000","last_peered":"2023-11-16T16:48:45.495877+0000","last_clean":"2023-11-16T16:48:45.495877+0000","last_became_active":"2023-11-16T16:47:45.473100+0000","last_became_peered":"2023-11-16T16:47:45.473100+0000","last_unstale":"2023-11-16T16:48:45.495877+0000","last_undegraded":"2023-11-16T16:48:45.495877+0000","last_fullsized":"2023-11-16T16:48:45.495877+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:29:41.041024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473340+0000","last_change":"2023-11-16T16:47:26.364112+0000","last_active":"2023-11-16T16:47:45.473340+0000","last_peered":"2023-11-16T16:47:45.473340+0000","last_clean":"2023-11-16T16:47:45.473340+0000","last_became_active":"2023-11-16T16:47:26.363767+0000","last_became_peered":"2023-11-16T16:47:26.363767+0000","last_unstale":"2023-11-16T16:47:45.473340+0000","last_undegraded":"2023-11-16T16:47:45.473340+0000","last_fullsized":"2023-11-16T16:47:45.473340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:01:40.809234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'4","reported_seq":88,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.495158+0000","last_change":"2023-11-16T16:47:25.830012+0000","last_active":"2023-11-16T16:48:45.495158+0000","last_peered":"2023-11-16T16:48:45.495158+0000","last_clean":"2023-11-16T16:48:45.495158+0000","last_became_active":"2023-11-16T16:47:25.829474+0000","last_became_peered":"2023-11-16T16:47:25.829474+0000","last_unstale":"2023-11-16T16:48:45.495158+0000","last_undegraded":"2023-11-16T16:48:45.495158+0000","last_fullsized":"2023-11-16T16:48:45.495158+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:24:50.530669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":54,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858885+0000","last_change":"2023-11-16T16:47:30.832724+0000","last_active":"2023-11-16T16:47:44.858885+0000","last_peered":"2023-11-16T16:47:44.858885+0000","last_clean":"2023-11-16T16:47:44.858885+0000","last_became_active":"2023-11-16T16:47:30.832226+0000","last_became_peered":"2023-11-16T16:47:30.832226+0000","last_unstale":"2023-11-16T16:47:44.858885+0000","last_undegraded":"2023-11-16T16:47:44.858885+0000","last_fullsized":"2023-11-16T16:47:44.858885+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:11:15.234460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.476312+0000","last_change":"2023-11-16T16:47:45.474859+0000","last_active":"2023-11-16T16:47:45.476312+0000","last_peered":"2023-11-16T16:47:45.476312+0000","last_clean":"2023-11-16T16:47:45.476312+0000","last_became_active":"2023-11-16T16:47:45.474439+0000","last_became_peered":"2023-11-16T16:47:45.474439+0000","last_unstale":"2023-11-16T16:47:45.476312+0000","last_undegraded":"2023-11-16T16:47:45.476312+0000","last_fullsized":"2023-11-16T16:47:45.476312+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:05:04.346537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:52.910430+0000","last_change":"2023-11-16T16:47:25.813894+0000","last_active":"2023-11-16T16:47:52.910430+0000","last_peered":"2023-11-16T16:47:52.910430+0000","last_clean":"2023-11-16T16:47:52.910430+0000","last_became_active":"2023-11-16T16:47:25.813526+0000","last_became_peered":"2023-11-16T16:47:25.813526+0000","last_unstale":"2023-11-16T16:47:52.910430+0000","last_undegraded":"2023-11-16T16:47:52.910430+0000","last_fullsized":"2023-11-16T16:47:52.910430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:26:19.570859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":359,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857425+0000","last_change":"2023-11-16T16:46:18.016268+0000","last_active":"2023-11-16T16:47:44.857425+0000","last_peered":"2023-11-16T16:47:44.857425+0000","last_clean":"2023-11-16T16:47:44.857425+0000","last_became_active":"2023-11-16T16:46:18.015772+0000","last_became_peered":"2023-11-16T16:46:18.015772+0000","last_unstale":"2023-11-16T16:47:44.857425+0000","last_undegraded":"2023-11-16T16:47:44.857425+0000","last_fullsized":"2023-11-16T16:47:44.857425+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:43:04.489460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:43:04.489460+0000","last_clean_scrub_stamp":"2023-11-16T16:43:04.489460+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:22:57.779720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473045+0000","last_change":"2023-11-16T16:47:30.830574+0000","last_active":"2023-11-16T16:47:45.473045+0000","last_peered":"2023-11-16T16:47:45.473045+0000","last_clean":"2023-11-16T16:47:45.473045+0000","last_became_active":"2023-11-16T16:47:30.830040+0000","last_became_peered":"2023-11-16T16:47:30.830040+0000","last_unstale":"2023-11-16T16:47:45.473045+0000","last_undegraded":"2023-11-16T16:47:45.473045+0000","last_fullsized":"2023-11-16T16:47:45.473045+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:29:55.893689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.495108+0000","last_change":"2023-11-16T16:47:26.378278+0000","last_active":"2023-11-16T16:48:45.495108+0000","last_peered":"2023-11-16T16:48:45.495108+0000","last_clean":"2023-11-16T16:48:45.495108+0000","last_became_active":"2023-11-16T16:47:26.377139+0000","last_became_peered":"2023-11-16T16:47:26.377139+0000","last_unstale":"2023-11-16T16:48:45.495108+0000","last_undegraded":"2023-11-16T16:48:45.495108+0000","last_fullsized":"2023-11-16T16:48:45.495108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:35:37.986780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.906018+0000","last_change":"2023-11-16T16:47:25.815714+0000","last_active":"2023-11-16T16:47:51.906018+0000","last_peered":"2023-11-16T16:47:51.906018+0000","last_clean":"2023-11-16T16:47:51.906018+0000","last_became_active":"2023-11-16T16:47:25.815452+0000","last_became_peered":"2023-11-16T16:47:25.815452+0000","last_unstale":"2023-11-16T16:47:51.906018+0000","last_undegraded":"2023-11-16T16:47:51.906018+0000","last_fullsized":"2023-11-16T16:47:51.906018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:33:18.235656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.500008+0000","last_change":"2023-11-16T16:47:30.826690+0000","last_active":"2023-11-16T16:48:45.500008+0000","last_peered":"2023-11-16T16:48:45.500008+0000","last_clean":"2023-11-16T16:48:45.500008+0000","last_became_active":"2023-11-16T16:47:30.826216+0000","last_became_peered":"2023-11-16T16:47:30.826216+0000","last_unstale":"2023-11-16T16:48:45.500008+0000","last_undegraded":"2023-11-16T16:48:45.500008+0000","last_fullsized":"2023-11-16T16:48:45.500008+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:55:34.457094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.502850+0000","last_change":"2023-11-16T16:47:25.826960+0000","last_active":"2023-11-16T16:48:45.502850+0000","last_peered":"2023-11-16T16:48:45.502850+0000","last_clean":"2023-11-16T16:48:45.502850+0000","last_became_active":"2023-11-16T16:47:25.826668+0000","last_became_peered":"2023-11-16T16:47:25.826668+0000","last_unstale":"2023-11-16T16:48:45.502850+0000","last_undegraded":"2023-11-16T16:48:45.502850+0000","last_fullsized":"2023-11-16T16:48:45.502850+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:53:26.902488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.495682+0000","last_change":"2023-11-16T16:47:26.370684+0000","last_active":"2023-11-16T16:48:45.495682+0000","last_peered":"2023-11-16T16:48:45.495682+0000","last_clean":"2023-11-16T16:48:45.495682+0000","last_became_active":"2023-11-16T16:47:26.370323+0000","last_became_peered":"2023-11-16T16:47:26.370323+0000","last_unstale":"2023-11-16T16:48:45.495682+0000","last_undegraded":"2023-11-16T16:48:45.495682+0000","last_fullsized":"2023-11-16T16:48:45.495682+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:56:19.756023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.502602+0000","last_change":"2023-11-16T16:47:30.837921+0000","last_active":"2023-11-16T16:48:45.502602+0000","last_peered":"2023-11-16T16:48:45.502602+0000","last_clean":"2023-11-16T16:48:45.502602+0000","last_became_active":"2023-11-16T16:47:30.831491+0000","last_became_peered":"2023-11-16T16:47:30.831491+0000","last_unstale":"2023-11-16T16:48:45.502602+0000","last_undegraded":"2023-11-16T16:48:45.502602+0000","last_fullsized":"2023-11-16T16:48:45.502602+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:18:33.595231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:48:00.309552+0000","last_change":"2023-11-16T16:47:25.816416+0000","last_active":"2023-11-16T16:48:00.309552+0000","last_peered":"2023-11-16T16:48:00.309552+0000","last_clean":"2023-11-16T16:48:00.309552+0000","last_became_active":"2023-11-16T16:47:25.815875+0000","last_became_peered":"2023-11-16T16:47:25.815875+0000","last_unstale":"2023-11-16T16:48:00.309552+0000","last_undegraded":"2023-11-16T16:48:00.309552+0000","last_fullsized":"2023-11-16T16:48:00.309552+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:34:15.271617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.502871+0000","last_change":"2023-11-16T16:47:26.386831+0000","last_active":"2023-11-16T16:48:45.502871+0000","last_peered":"2023-11-16T16:48:45.502871+0000","last_clean":"2023-11-16T16:48:45.502871+0000","last_became_active":"2023-11-16T16:47:26.386518+0000","last_became_peered":"2023-11-16T16:47:26.386518+0000","last_unstale":"2023-11-16T16:48:45.502871+0000","last_undegraded":"2023-11-16T16:48:45.502871+0000","last_fullsized":"2023-11-16T16:48:45.502871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:07:53.209712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854593+0000","last_change":"2023-11-16T16:47:30.823733+0000","last_active":"2023-11-16T16:47:44.854593+0000","last_peered":"2023-11-16T16:47:44.854593+0000","last_clean":"2023-11-16T16:47:44.854593+0000","last_became_active":"2023-11-16T16:47:30.823293+0000","last_became_peered":"2023-11-16T16:47:30.823293+0000","last_unstale":"2023-11-16T16:47:44.854593+0000","last_undegraded":"2023-11-16T16:47:44.854593+0000","last_fullsized":"2023-11-16T16:47:44.854593+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:30:09.995445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.899469+0000","last_change":"2023-11-16T16:47:25.818137+0000","last_active":"2023-11-16T16:47:50.899469+0000","last_peered":"2023-11-16T16:47:50.899469+0000","last_clean":"2023-11-16T16:47:50.899469+0000","last_became_active":"2023-11-16T16:47:25.817661+0000","last_became_peered":"2023-11-16T16:47:25.817661+0000","last_unstale":"2023-11-16T16:47:50.899469+0000","last_undegraded":"2023-11-16T16:47:50.899469+0000","last_fullsized":"2023-11-16T16:47:50.899469+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:17:33.467718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.845828+0000","last_change":"2023-11-16T16:47:45.415106+0000","last_active":"2023-11-16T16:48:45.845828+0000","last_peered":"2023-11-16T16:48:45.845828+0000","last_clean":"2023-11-16T16:48:45.845828+0000","last_became_active":"2023-11-16T16:47:45.414554+0000","last_became_peered":"2023-11-16T16:47:45.414554+0000","last_unstale":"2023-11-16T16:48:45.845828+0000","last_undegraded":"2023-11-16T16:48:45.845828+0000","last_fullsized":"2023-11-16T16:48:45.845828+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:05:11.725246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,8],"acting":[0,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854667+0000","last_change":"2023-11-16T16:47:30.820944+0000","last_active":"2023-11-16T16:47:44.854667+0000","last_peered":"2023-11-16T16:47:44.854667+0000","last_clean":"2023-11-16T16:47:44.854667+0000","last_became_active":"2023-11-16T16:47:30.820514+0000","last_became_peered":"2023-11-16T16:47:30.820514+0000","last_unstale":"2023-11-16T16:47:44.854667+0000","last_undegraded":"2023-11-16T16:47:44.854667+0000","last_fullsized":"2023-11-16T16:47:44.854667+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:18:55.421306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:50.899895+0000","last_change":"2023-11-16T16:47:25.826960+0000","last_active":"2023-11-16T16:47:50.899895+0000","last_peered":"2023-11-16T16:47:50.899895+0000","last_clean":"2023-11-16T16:47:50.899895+0000","last_became_active":"2023-11-16T16:47:25.826671+0000","last_became_peered":"2023-11-16T16:47:25.826671+0000","last_unstale":"2023-11-16T16:47:50.899895+0000","last_undegraded":"2023-11-16T16:47:50.899895+0000","last_fullsized":"2023-11-16T16:47:50.899895+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:41:34.201037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.502378+0000","last_change":"2023-11-16T16:47:26.364697+0000","last_active":"2023-11-16T16:48:45.502378+0000","last_peered":"2023-11-16T16:48:45.502378+0000","last_clean":"2023-11-16T16:48:45.502378+0000","last_became_active":"2023-11-16T16:47:26.364418+0000","last_became_peered":"2023-11-16T16:47:26.364418+0000","last_unstale":"2023-11-16T16:48:45.502378+0000","last_undegraded":"2023-11-16T16:48:45.502378+0000","last_fullsized":"2023-11-16T16:48:45.502378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:20:23.913142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.503027+0000","last_change":"2023-11-16T16:47:45.474571+0000","last_active":"2023-11-16T16:48:45.503027+0000","last_peered":"2023-11-16T16:48:45.503027+0000","last_clean":"2023-11-16T16:48:45.503027+0000","last_became_active":"2023-11-16T16:47:45.474093+0000","last_became_peered":"2023-11-16T16:47:45.474093+0000","last_unstale":"2023-11-16T16:48:45.503027+0000","last_undegraded":"2023-11-16T16:48:45.503027+0000","last_fullsized":"2023-11-16T16:48:45.503027+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:56:12.976730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.502944+0000","last_change":"2023-11-16T16:47:25.826208+0000","last_active":"2023-11-16T16:48:45.502944+0000","last_peered":"2023-11-16T16:48:45.502944+0000","last_clean":"2023-11-16T16:48:45.502944+0000","last_became_active":"2023-11-16T16:47:25.825853+0000","last_became_peered":"2023-11-16T16:47:25.825853+0000","last_unstale":"2023-11-16T16:48:45.502944+0000","last_undegraded":"2023-11-16T16:48:45.502944+0000","last_fullsized":"2023-11-16T16:48:45.502944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:15:45.867733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.499831+0000","last_change":"2023-11-16T16:47:26.361786+0000","last_active":"2023-11-16T16:48:45.499831+0000","last_peered":"2023-11-16T16:48:45.499831+0000","last_clean":"2023-11-16T16:48:45.499831+0000","last_became_active":"2023-11-16T16:47:26.361410+0000","last_became_peered":"2023-11-16T16:47:26.361410+0000","last_unstale":"2023-11-16T16:48:45.499831+0000","last_undegraded":"2023-11-16T16:48:45.499831+0000","last_fullsized":"2023-11-16T16:48:45.499831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:00:44.942183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.474814+0000","last_change":"2023-11-16T16:47:45.474123+0000","last_active":"2023-11-16T16:47:45.474814+0000","last_peered":"2023-11-16T16:47:45.474814+0000","last_clean":"2023-11-16T16:47:45.474814+0000","last_became_active":"2023-11-16T16:47:45.473630+0000","last_became_peered":"2023-11-16T16:47:45.473630+0000","last_unstale":"2023-11-16T16:47:45.474814+0000","last_undegraded":"2023-11-16T16:47:45.474814+0000","last_fullsized":"2023-11-16T16:47:45.474814+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:39:24.719048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'2","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.509295+0000","last_change":"2023-11-16T16:48:45.508825+0000","last_active":"2023-11-16T16:48:45.509295+0000","last_peered":"2023-11-16T16:48:45.509295+0000","last_clean":"2023-11-16T16:48:45.509295+0000","last_became_active":"2023-11-16T16:48:45.504751+0000","last_became_peered":"2023-11-16T16:48:45.504751+0000","last_unstale":"2023-11-16T16:48:45.509295+0000","last_undegraded":"2023-11-16T16:48:45.509295+0000","last_fullsized":"2023-11-16T16:48:45.509295+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:37:45.458725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.502688+0000","last_change":"2023-11-16T16:47:26.379945+0000","last_active":"2023-11-16T16:48:45.502688+0000","last_peered":"2023-11-16T16:48:45.502688+0000","last_clean":"2023-11-16T16:48:45.502688+0000","last_became_active":"2023-11-16T16:47:26.376552+0000","last_became_peered":"2023-11-16T16:47:26.376552+0000","last_unstale":"2023-11-16T16:48:45.502688+0000","last_undegraded":"2023-11-16T16:48:45.502688+0000","last_fullsized":"2023-11-16T16:48:45.502688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:44:15.712067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854429+0000","last_change":"2023-11-16T16:47:30.826281+0000","last_active":"2023-11-16T16:47:44.854429+0000","last_peered":"2023-11-16T16:47:44.854429+0000","last_clean":"2023-11-16T16:47:44.854429+0000","last_became_active":"2023-11-16T16:47:30.825920+0000","last_became_peered":"2023-11-16T16:47:30.825920+0000","last_unstale":"2023-11-16T16:47:44.854429+0000","last_undegraded":"2023-11-16T16:47:44.854429+0000","last_fullsized":"2023-11-16T16:47:44.854429+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T02:23:25.700645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.502509+0000","last_change":"2023-11-16T16:47:25.815879+0000","last_active":"2023-11-16T16:48:45.502509+0000","last_peered":"2023-11-16T16:48:45.502509+0000","last_clean":"2023-11-16T16:48:45.502509+0000","last_became_active":"2023-11-16T16:47:25.815557+0000","last_became_peered":"2023-11-16T16:47:25.815557+0000","last_unstale":"2023-11-16T16:48:45.502509+0000","last_undegraded":"2023-11-16T16:48:45.502509+0000","last_fullsized":"2023-11-16T16:48:45.502509+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:44:32.860681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858518+0000","last_change":"2023-11-16T16:47:26.365013+0000","last_active":"2023-11-16T16:47:44.858518+0000","last_peered":"2023-11-16T16:47:44.858518+0000","last_clean":"2023-11-16T16:47:44.858518+0000","last_became_active":"2023-11-16T16:47:26.364710+0000","last_became_peered":"2023-11-16T16:47:26.364710+0000","last_unstale":"2023-11-16T16:47:44.858518+0000","last_undegraded":"2023-11-16T16:47:44.858518+0000","last_fullsized":"2023-11-16T16:47:44.858518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:57:21.045225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.857432+0000","last_change":"2023-11-16T16:47:30.826880+0000","last_active":"2023-11-16T16:47:44.857432+0000","last_peered":"2023-11-16T16:47:44.857432+0000","last_clean":"2023-11-16T16:47:44.857432+0000","last_became_active":"2023-11-16T16:47:30.826547+0000","last_became_peered":"2023-11-16T16:47:30.826547+0000","last_unstale":"2023-11-16T16:47:44.857432+0000","last_undegraded":"2023-11-16T16:47:44.857432+0000","last_fullsized":"2023-11-16T16:47:44.857432+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:34:34.448480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:51.908865+0000","last_change":"2023-11-16T16:47:25.825246+0000","last_active":"2023-11-16T16:47:51.908865+0000","last_peered":"2023-11-16T16:47:51.908865+0000","last_clean":"2023-11-16T16:47:51.908865+0000","last_became_active":"2023-11-16T16:47:25.824929+0000","last_became_peered":"2023-11-16T16:47:25.824929+0000","last_unstale":"2023-11-16T16:47:51.908865+0000","last_undegraded":"2023-11-16T16:47:51.908865+0000","last_fullsized":"2023-11-16T16:47:51.908865+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:15:26.168130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.859304+0000","last_change":"2023-11-16T16:47:26.372266+0000","last_active":"2023-11-16T16:47:44.859304+0000","last_peered":"2023-11-16T16:47:44.859304+0000","last_clean":"2023-11-16T16:47:44.859304+0000","last_became_active":"2023-11-16T16:47:26.371857+0000","last_became_peered":"2023-11-16T16:47:26.371857+0000","last_unstale":"2023-11-16T16:47:44.859304+0000","last_undegraded":"2023-11-16T16:47:44.859304+0000","last_fullsized":"2023-11-16T16:47:44.859304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:22:33.979224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.845734+0000","last_change":"2023-11-16T16:47:30.832187+0000","last_active":"2023-11-16T16:48:45.845734+0000","last_peered":"2023-11-16T16:48:45.845734+0000","last_clean":"2023-11-16T16:48:45.845734+0000","last_became_active":"2023-11-16T16:47:30.828115+0000","last_became_peered":"2023-11-16T16:47:30.828115+0000","last_unstale":"2023-11-16T16:48:45.845734+0000","last_undegraded":"2023-11-16T16:48:45.845734+0000","last_fullsized":"2023-11-16T16:48:45.845734+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:15:43.103300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473519+0000","last_change":"2023-11-16T16:47:25.822706+0000","last_active":"2023-11-16T16:47:45.473519+0000","last_peered":"2023-11-16T16:47:45.473519+0000","last_clean":"2023-11-16T16:47:45.473519+0000","last_became_active":"2023-11-16T16:47:25.822400+0000","last_became_peered":"2023-11-16T16:47:25.822400+0000","last_unstale":"2023-11-16T16:47:45.473519+0000","last_undegraded":"2023-11-16T16:47:45.473519+0000","last_fullsized":"2023-11-16T16:47:45.473519+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:32:15.408720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.474751+0000","last_change":"2023-11-16T16:47:45.474751+0000","last_active":"2023-11-16T16:47:45.474751+0000","last_peered":"2023-11-16T16:47:45.474751+0000","last_clean":"2023-11-16T16:47:45.474751+0000","last_became_active":"2023-11-16T16:47:45.474415+0000","last_became_peered":"2023-11-16T16:47:45.474415+0000","last_unstale":"2023-11-16T16:47:45.474751+0000","last_undegraded":"2023-11-16T16:47:45.474751+0000","last_fullsized":"2023-11-16T16:47:45.474751+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:56:14.927411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.472751+0000","last_change":"2023-11-16T16:47:30.822549+0000","last_active":"2023-11-16T16:47:45.472751+0000","last_peered":"2023-11-16T16:47:45.472751+0000","last_clean":"2023-11-16T16:47:45.472751+0000","last_became_active":"2023-11-16T16:47:30.822150+0000","last_became_peered":"2023-11-16T16:47:30.822150+0000","last_unstale":"2023-11-16T16:47:45.472751+0000","last_undegraded":"2023-11-16T16:47:45.472751+0000","last_fullsized":"2023-11-16T16:47:45.472751+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:29.796954+0000","last_clean_scrub_stamp":"2023-11-16T16:47:29.796954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:17:45.983962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.495851+0000","last_change":"2023-11-16T16:47:25.826947+0000","last_active":"2023-11-16T16:48:45.495851+0000","last_peered":"2023-11-16T16:48:45.495851+0000","last_clean":"2023-11-16T16:48:45.495851+0000","last_became_active":"2023-11-16T16:47:25.826587+0000","last_became_peered":"2023-11-16T16:47:25.826587+0000","last_unstale":"2023-11-16T16:48:45.495851+0000","last_undegraded":"2023-11-16T16:48:45.495851+0000","last_fullsized":"2023-11-16T16:48:45.495851+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:24.783822+0000","last_clean_scrub_stamp":"2023-11-16T16:47:24.783822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T23:16:18.739244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858841+0000","last_change":"2023-11-16T16:47:26.373914+0000","last_active":"2023-11-16T16:47:44.858841+0000","last_peered":"2023-11-16T16:47:44.858841+0000","last_clean":"2023-11-16T16:47:44.858841+0000","last_became_active":"2023-11-16T16:47:26.373594+0000","last_became_peered":"2023-11-16T16:47:26.373594+0000","last_unstale":"2023-11-16T16:47:44.858841+0000","last_undegraded":"2023-11-16T16:47:44.858841+0000","last_fullsized":"2023-11-16T16:47:44.858841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:18:55.608486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.499889+0000","last_change":"2023-11-16T16:47:26.368237+0000","last_active":"2023-11-16T16:48:45.499889+0000","last_peered":"2023-11-16T16:48:45.499889+0000","last_clean":"2023-11-16T16:48:45.499889+0000","last_became_active":"2023-11-16T16:47:26.367957+0000","last_became_peered":"2023-11-16T16:47:26.367957+0000","last_unstale":"2023-11-16T16:48:45.499889+0000","last_undegraded":"2023-11-16T16:48:45.499889+0000","last_fullsized":"2023-11-16T16:48:45.499889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:00:05.806876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854982+0000","last_change":"2023-11-16T16:47:26.375399+0000","last_active":"2023-11-16T16:47:44.854982+0000","last_peered":"2023-11-16T16:47:44.854982+0000","last_clean":"2023-11-16T16:47:44.854982+0000","last_became_active":"2023-11-16T16:47:26.375064+0000","last_became_peered":"2023-11-16T16:47:26.375064+0000","last_unstale":"2023-11-16T16:47:44.854982+0000","last_undegraded":"2023-11-16T16:47:44.854982+0000","last_fullsized":"2023-11-16T16:47:44.854982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T18:10:23.687848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.854908+0000","last_change":"2023-11-16T16:47:26.377080+0000","last_active":"2023-11-16T16:47:44.854908+0000","last_peered":"2023-11-16T16:47:44.854908+0000","last_clean":"2023-11-16T16:47:44.854908+0000","last_became_active":"2023-11-16T16:47:26.376759+0000","last_became_peered":"2023-11-16T16:47:26.376759+0000","last_unstale":"2023-11-16T16:47:44.854908+0000","last_undegraded":"2023-11-16T16:47:44.854908+0000","last_fullsized":"2023-11-16T16:47:44.854908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:51:18.518523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.846304+0000","last_change":"2023-11-16T16:47:26.380619+0000","last_active":"2023-11-16T16:48:45.846304+0000","last_peered":"2023-11-16T16:48:45.846304+0000","last_clean":"2023-11-16T16:48:45.846304+0000","last_became_active":"2023-11-16T16:47:26.378668+0000","last_became_peered":"2023-11-16T16:47:26.378668+0000","last_unstale":"2023-11-16T16:48:45.846304+0000","last_undegraded":"2023-11-16T16:48:45.846304+0000","last_fullsized":"2023-11-16T16:48:45.846304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T21:36:46.240566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.495624+0000","last_change":"2023-11-16T16:47:26.377482+0000","last_active":"2023-11-16T16:48:45.495624+0000","last_peered":"2023-11-16T16:48:45.495624+0000","last_clean":"2023-11-16T16:48:45.495624+0000","last_became_active":"2023-11-16T16:47:26.377077+0000","last_became_peered":"2023-11-16T16:47:26.377077+0000","last_unstale":"2023-11-16T16:48:45.495624+0000","last_undegraded":"2023-11-16T16:48:45.495624+0000","last_fullsized":"2023-11-16T16:48:45.495624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T04:05:44.614692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.500705+0000","last_change":"2023-11-16T16:47:26.368193+0000","last_active":"2023-11-16T16:48:45.500705+0000","last_peered":"2023-11-16T16:48:45.500705+0000","last_clean":"2023-11-16T16:48:45.500705+0000","last_became_active":"2023-11-16T16:47:26.367499+0000","last_became_peered":"2023-11-16T16:47:26.367499+0000","last_unstale":"2023-11-16T16:48:45.500705+0000","last_undegraded":"2023-11-16T16:48:45.500705+0000","last_fullsized":"2023-11-16T16:48:45.500705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T00:04:53.812540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.855675+0000","last_change":"2023-11-16T16:47:26.367340+0000","last_active":"2023-11-16T16:47:44.855675+0000","last_peered":"2023-11-16T16:47:44.855675+0000","last_clean":"2023-11-16T16:47:44.855675+0000","last_became_active":"2023-11-16T16:47:26.367046+0000","last_became_peered":"2023-11-16T16:47:26.367046+0000","last_unstale":"2023-11-16T16:47:44.855675+0000","last_undegraded":"2023-11-16T16:47:44.855675+0000","last_fullsized":"2023-11-16T16:47:44.855675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T22:14:30.539229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.503000+0000","last_change":"2023-11-16T16:47:26.386174+0000","last_active":"2023-11-16T16:48:45.503000+0000","last_peered":"2023-11-16T16:48:45.503000+0000","last_clean":"2023-11-16T16:48:45.503000+0000","last_became_active":"2023-11-16T16:47:26.385496+0000","last_became_peered":"2023-11-16T16:47:26.385496+0000","last_unstale":"2023-11-16T16:48:45.503000+0000","last_undegraded":"2023-11-16T16:48:45.503000+0000","last_fullsized":"2023-11-16T16:48:45.503000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T17:35:09.298672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:45.473021+0000","last_change":"2023-11-16T16:47:26.368224+0000","last_active":"2023-11-16T16:47:45.473021+0000","last_peered":"2023-11-16T16:47:45.473021+0000","last_clean":"2023-11-16T16:47:45.473021+0000","last_became_active":"2023-11-16T16:47:26.367909+0000","last_became_peered":"2023-11-16T16:47:26.367909+0000","last_unstale":"2023-11-16T16:47:45.473021+0000","last_undegraded":"2023-11-16T16:47:45.473021+0000","last_fullsized":"2023-11-16T16:47:45.473021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T16:52:55.966566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.502659+0000","last_change":"2023-11-16T16:47:26.381224+0000","last_active":"2023-11-16T16:48:45.502659+0000","last_peered":"2023-11-16T16:48:45.502659+0000","last_clean":"2023-11-16T16:48:45.502659+0000","last_became_active":"2023-11-16T16:47:26.376586+0000","last_became_peered":"2023-11-16T16:47:26.376586+0000","last_unstale":"2023-11-16T16:48:45.502659+0000","last_undegraded":"2023-11-16T16:48:45.502659+0000","last_fullsized":"2023-11-16T16:48:45.502659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T03:06:27.961904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-11-16T16:47:44.858440+0000","last_change":"2023-11-16T16:47:26.385895+0000","last_active":"2023-11-16T16:47:44.858440+0000","last_peered":"2023-11-16T16:47:44.858440+0000","last_clean":"2023-11-16T16:47:44.858440+0000","last_became_active":"2023-11-16T16:47:26.385580+0000","last_became_peered":"2023-11-16T16:47:26.385580+0000","last_unstale":"2023-11-16T16:47:44.858440+0000","last_undegraded":"2023-11-16T16:47:44.858440+0000","last_fullsized":"2023-11-16T16:47:44.858440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T20:03:39.204040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.495631+0000","last_change":"2023-11-16T16:47:26.373785+0000","last_active":"2023-11-16T16:48:45.495631+0000","last_peered":"2023-11-16T16:48:45.495631+0000","last_clean":"2023-11-16T16:48:45.495631+0000","last_became_active":"2023-11-16T16:47:26.373425+0000","last_became_peered":"2023-11-16T16:47:26.373425+0000","last_unstale":"2023-11-16T16:48:45.495631+0000","last_undegraded":"2023-11-16T16:48:45.495631+0000","last_fullsized":"2023-11-16T16:48:45.495631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-17T19:39:58.541860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-11-16T16:48:45.500148+0000","last_change":"2023-11-16T16:47:26.374722+0000","last_active":"2023-11-16T16:48:45.500148+0000","last_peered":"2023-11-16T16:48:45.500148+0000","last_clean":"2023-11-16T16:48:45.500148+0000","last_became_active":"2023-11-16T16:47:26.374326+0000","last_became_peered":"2023-11-16T16:47:26.374326+0000","last_unstale":"2023-11-16T16:48:45.500148+0000","last_undegraded":"2023-11-16T16:48:45.500148+0000","last_fullsized":"2023-11-16T16:48:45.500148+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-16T16:47:25.794046+0000","last_clean_scrub_stamp":"2023-11-16T16:47:25.794046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-18T01:05:47.772747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":249745,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":80977,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":343,"num_read_kb":258,"num_write":166,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":34,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":242965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383709,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6256,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738896,"statfs":{"total":95995035648,"available":95988629504,"internally_reserved":0,"allocated":1024000,"data_stored":519859,"data_compressed":23596,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55900000000000005,"15min":0.56000000000000005},"min":{"1min":0.41799999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.5},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58299999999999996,"15min":0.58399999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.77300000000000002,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.67000000000000004}]},{"osd":1,"last update":"Thu Nov 16 16:48:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.65200000000000002,"15min":0.64600000000000002},"min":{"1min":0.46000000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.85499999999999998,"5min":0.873,"15min":0.873},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57999999999999996,"15min":0.57799999999999996},"min":{"1min":0.46600000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71799999999999997,"5min":0.749,"15min":0.749},"last":0.68400000000000005}]},{"osd":2,"last update":"Thu Nov 16 16:48:37 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52200000000000002,"15min":0.52500000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68200000000000005,"5min":1.159,"15min":1.159},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.51900000000000002,"15min":0.51100000000000001},"min":{"1min":0.39400000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78700000000000003,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.65200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":4,"last update":"Thu Nov 16 16:48:37 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.68899999999999995,"15min":0.68300000000000005},"min":{"1min":0.49299999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88700000000000001,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.51600000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81399999999999995,"5min":1.046,"15min":1.046},"last":0.51800000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:48:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.63700000000000001,"15min":0.64100000000000001},"min":{"1min":0.439,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.78200000000000003,"5min":1.345,"15min":1.345},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63,"15min":0.628},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.89200000000000002,"5min":1.02,"15min":1.02},"last":0.73799999999999999}]},{"osd":6,"last update":"Thu Nov 16 16:48:37 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.627,"15min":0.63},"min":{"1min":0.436,"5min":0.432,"15min":0.432},"max":{"1min":0.91200000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.76700000000000002,"15min":0.77300000000000002},"min":{"1min":0.59199999999999997,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.91500000000000004,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Nov 16 16:48:37 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.58099999999999996,"5min":0.496,"15min":0.496},"max":{"1min":0.92000000000000004,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72999999999999998,"15min":0.73499999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.84699999999999998,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.63}]},{"osd":8,"last update":"Thu Nov 16 16:48:27 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.69999999999999996}]},{"osd":9,"last update":"Thu Nov 16 16:48:37 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.83399999999999996,"15min":0.83799999999999997},"min":{"1min":0.63200000000000001,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.94499999999999995,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.81499999999999995,"15min":0.81299999999999994},"min":{"1min":0.63700000000000001,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.95899999999999996,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.79700000000000004}]},{"osd":10,"last update":"Thu Nov 16 16:48:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.751,"15min":0.753},"min":{"1min":0.56799999999999995,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0169999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.65800000000000003,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.92300000000000004,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.75}]}]},{"osd":10,"up_from":72,"seq":309237645345,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6632,"kb_used_data":1120,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738520,"statfs":{"total":95995035648,"available":95988244480,"internally_reserved":0,"allocated":1146880,"data_stored":1158377,"data_compressed":53789,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:27 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.58299999999999996}]},{"osd":1,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61699999999999999,"5min":0.622,"15min":0.622},"last":0.433},{"interface":"front","average":{"1min":0.441,"5min":0.44500000000000001,"15min":0.44600000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.39800000000000002}]},{"osd":2,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.45800000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61299999999999999,"15min":0.60899999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.67000000000000004}]},{"osd":3,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64200000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.66400000000000003,"15min":0.67200000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.81899999999999995,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.59399999999999997}]},{"osd":4,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.55800000000000005,"15min":0.56000000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77600000000000002,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.375,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.46400000000000002}]},{"osd":5,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65400000000000003,"15min":0.65600000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.77000000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67300000000000004,"15min":0.67000000000000004},"min":{"1min":0.47199999999999998,"5min":0.435,"15min":0.435},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.72299999999999998}]},{"osd":6,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.74299999999999999,"15min":0.755},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.92400000000000004,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.72999999999999998,"15min":0.73899999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.87,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.75600000000000001}]},{"osd":7,"last update":"Thu Nov 16 16:47:14 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.46999999999999997}]},{"osd":8,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.92400000000000004,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.73099999999999998,"15min":0.73699999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.68700000000000006}]},{"osd":9,"last update":"Thu Nov 16 16:48:14 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.79800000000000004,"15min":0.81000000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.94699999999999995,"5min":0.995,"15min":0.995},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.77300000000000002,"15min":0.78700000000000003},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.91300000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.74399999999999999}]},{"osd":11,"last update":"Thu Nov 16 16:48:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.748,"15min":0.746},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.745,"15min":0.74299999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.79400000000000004}]}]},{"osd":9,"up_from":67,"seq":287762808870,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6372,"kb_used_data":988,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":1011712,"data_stored":500553,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.56699999999999995,"15min":0.58599999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.68300000000000005,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.57399999999999995,"15min":0.59099999999999997},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.64200000000000002,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.40699999999999997}]},{"osd":1,"last update":"Thu Nov 16 16:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.67000000000000004,"15min":0.68400000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.70499999999999996,"5min":1.002,"15min":1.002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.79900000000000004,"15min":0.81699999999999995},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.86699999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.52400000000000002}]},{"osd":2,"last update":"Thu Nov 16 16:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.85499999999999998,"15min":0.879},"min":{"1min":0.60599999999999998,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.82499999999999996,"5min":1.401,"15min":1.401},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.82999999999999996,"15min":0.85299999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.90000000000000002,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Nov 16 16:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.78600000000000003,"5min":0.995,"15min":0.995},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.82299999999999995,"15min":0.84999999999999998},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.90800000000000003,"5min":1.343,"15min":1.343},"last":0.61499999999999999}]},{"osd":4,"last update":"Thu Nov 16 16:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.84299999999999997,"15min":0.86699999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.97399999999999998,"5min":1.202,"15min":1.202},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.78900000000000003,"15min":0.80500000000000005},"min":{"1min":0.57399999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.88800000000000001,"5min":1.081,"15min":1.081},"last":0.56699999999999995}]},{"osd":5,"last update":"Thu Nov 16 16:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.92200000000000004,"15min":0.95399999999999996},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.94199999999999995,"5min":1.302,"15min":1.302},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.85999999999999999,"15min":0.88600000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.94899999999999995,"5min":1.196,"15min":1.196},"last":0.68700000000000006}]},{"osd":6,"last update":"Thu Nov 16 16:46:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.629}]},{"osd":7,"last update":"Thu Nov 16 16:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.90300000000000002,"15min":0.92800000000000005},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":0.97399999999999998,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.99399999999999999,"15min":1.022},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.0740000000000001,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.72899999999999998}]},{"osd":8,"last update":"Thu Nov 16 16:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":1.0189999999999999,"15min":1.0469999999999999},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.012,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.82399999999999995,"5min":1.0109999999999999,"15min":1.042},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.0880000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.83599999999999997}]},{"osd":10,"last update":"Thu Nov 16 16:48:19 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.94199999999999995,"15min":0.95399999999999996},"min":{"1min":0.72099999999999997,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.0169999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.93799999999999994,"15min":0.95199999999999996},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.0489999999999999,"5min":1.421,"15min":1.421},"last":0.71599999999999997}]},{"osd":11,"last update":"Thu Nov 16 16:48:31 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.78400000000000003}]}]},{"osd":8,"up_from":59,"seq":253403070507,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6584,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738568,"statfs":{"total":95995035648,"available":95988293632,"internally_reserved":0,"allocated":1032192,"data_stored":519650,"data_compressed":23596,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58599999999999997,"15min":0.60199999999999998},"min":{"1min":0.48299999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78300000000000003,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.42599999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.77700000000000002,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.47999999999999998}]},{"osd":1,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.60899999999999999,"15min":0.59699999999999998},"min":{"1min":0.38300000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.97499999999999998,"5min":1.089,"15min":1.089},"last":0.629},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.60599999999999998,"15min":0.59799999999999998},"min":{"1min":0.42599999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.64400000000000002}]},{"osd":2,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.47999999999999998,"15min":0.48099999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64400000000000002,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.48099999999999998,"15min":0.47599999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69099999999999995,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.435}]},{"osd":3,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.627,"15min":0.63700000000000001},"min":{"1min":0.46200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81999999999999995,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.66600000000000004,"15min":0.68100000000000005},"min":{"1min":0.499,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93400000000000005,"5min":1.115,"15min":1.115},"last":0.54500000000000004}]},{"osd":4,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.61399999999999999,"15min":0.59599999999999997},"min":{"1min":0.41399999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.40500000000000003,"5min":0.375,"15min":0.375},"max":{"1min":0.94999999999999996,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.78400000000000003}]},{"osd":5,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59599999999999997,"15min":0.60599999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.84499999999999997,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.59599999999999997,"15min":0.58999999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.58799999999999997}]},{"osd":6,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":0.80400000000000005,"15min":0.77300000000000002},"min":{"1min":0.56399999999999995,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":6.5940000000000003,"5min":6.5940000000000003,"15min":6.5940000000000003},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.753,"5min":0.75900000000000001,"15min":0.78000000000000003},"min":{"1min":0.57899999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.98099999999999998,"5min":1.093,"15min":1.093},"last":0.74199999999999999}]},{"osd":7,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.72699999999999998,"15min":0.72999999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.94499999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.71699999999999997,"15min":0.70199999999999996},"min":{"1min":0.46500000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.79800000000000004}]},{"osd":9,"last update":"Thu Nov 16 16:47:58 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.59199999999999997,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.998,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.871},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.79800000000000004,"15min":0.80500000000000005},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.92900000000000005,"5min":1.393,"15min":1.393},"last":0.82899999999999996}]},{"osd":10,"last update":"Thu Nov 16 16:48:20 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.72699999999999998,"15min":0.71499999999999997},"min":{"1min":0.60499999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.749,"15min":0.73499999999999999},"min":{"1min":0.63100000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.85699999999999998}]},{"osd":11,"last update":"Thu Nov 16 16:48:26 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.72999999999999998}]}]},{"osd":7,"up_from":51,"seq":219043332142,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6620,"kb_used_data":1044,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738532,"statfs":{"total":95995035648,"available":95988256768,"internally_reserved":0,"allocated":1069056,"data_stored":1106673,"data_compressed":53789,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56200000000000006,"15min":0.55500000000000005},"min":{"1min":0.34899999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.56799999999999995,"15min":0.55600000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.48699999999999999}]},{"osd":1,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.42799999999999999,"15min":0.41699999999999998},"min":{"1min":0.314,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65600000000000003,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.46999999999999997,"15min":0.46000000000000002},"min":{"1min":0.34899999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.67900000000000005,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.34300000000000003}]},{"osd":2,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.57699999999999996,"15min":0.57199999999999995},"min":{"1min":0.38600000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58099999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77900000000000003,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.52000000000000002}]},{"osd":3,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61299999999999999,"15min":0.61699999999999999},"min":{"1min":0.47099999999999997,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85399999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60199999999999998,"15min":0.58999999999999997},"min":{"1min":0.497,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72899999999999998,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.47499999999999998}]},{"osd":4,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.51600000000000001,"15min":0.5},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.83199999999999996,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.499},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51500000000000001,"15min":0.51300000000000001},"min":{"1min":0.35199999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.441}]},{"osd":5,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.63200000000000001,"15min":0.61699999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.878,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.66600000000000004}]},{"osd":6,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.74099999999999999,"15min":0.70499999999999996},"min":{"1min":0.52600000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.93100000000000005,"5min":4.8840000000000003,"15min":4.8840000000000003},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68799999999999994,"15min":0.68000000000000005},"min":{"1min":0.54800000000000004,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.90800000000000003,"5min":0.996,"15min":0.996},"last":0.68200000000000005}]},{"osd":8,"last update":"Thu Nov 16 16:48:27 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.70499999999999996,"15min":0.69499999999999995},"min":{"1min":0.47599999999999998,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.746,"15min":0.74099999999999999},"min":{"1min":0.55000000000000004,"5min":0.497,"15min":0.497},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.77500000000000002}]},{"osd":9,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.87,"15min":0.84199999999999997},"min":{"1min":0.57199999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":4.9210000000000003,"5min":4.9210000000000003,"15min":4.9210000000000003},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.80900000000000005,"15min":0.78300000000000003},"min":{"1min":0.60799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":4.9000000000000004,"5min":4.9000000000000004,"15min":4.9000000000000004},"last":0.63600000000000001}]},{"osd":10,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.76500000000000001,"15min":0.77400000000000002},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.022,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.752},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.72099999999999997,"15min":0.72499999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.88200000000000001,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.60899999999999999}]},{"osd":11,"last update":"Thu Nov 16 16:48:27 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.79700000000000004}]}]},{"osd":6,"up_from":46,"seq":197568495668,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6504,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738648,"statfs":{"total":95995035648,"available":95988375552,"internally_reserved":0,"allocated":1015808,"data_stored":515237,"data_compressed":23596,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.49099999999999999,"15min":0.46700000000000003},"min":{"1min":0.312,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.71799999999999997,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55400000000000005,"15min":0.53400000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.81899999999999995,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.46999999999999997}]},{"osd":1,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.69499999999999995,"15min":0.66000000000000003},"min":{"1min":0.50600000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.032,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.433,"5min":0.378,"15min":0.378},"max":{"1min":0.90000000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.80500000000000005}]},{"osd":2,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.72899999999999998,"15min":0.70999999999999996},"min":{"1min":0.57899999999999996,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0589999999999999,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.74199999999999999,"15min":0.71999999999999997},"min":{"1min":0.56100000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.008,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.76200000000000001}]},{"osd":3,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.68799999999999994,"15min":0.66800000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60599999999999998,"15min":0.60199999999999998},"min":{"1min":0.36399999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.90600000000000003,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.55200000000000005}]},{"osd":4,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.73799999999999999},"min":{"1min":0.47799999999999998,"5min":0.44,"15min":0.44},"max":{"1min":1,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.79600000000000004,"15min":0.78100000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.1819999999999999,"5min":1.3,"15min":1.3},"last":1.0169999999999999}]},{"osd":5,"last update":"Thu Nov 16 16:47:47 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.73199999999999998,"15min":0.73099999999999998},"min":{"1min":0.50800000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.147,"5min":1.206,"15min":1.206},"last":0.745},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.80600000000000005,"15min":0.78800000000000003},"min":{"1min":0.58299999999999996,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.139,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.91700000000000004}]},{"osd":7,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.74099999999999999,"15min":0.71299999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.82199999999999995,"15min":0.80500000000000005},"min":{"1min":0.61599999999999999,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2030000000000001,"5min":1.246,"15min":1.246},"last":1.032}]},{"osd":8,"last update":"Thu Nov 16 16:48:30 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.91100000000000003,"15min":0.91500000000000004},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.92400000000000004,"15min":0.93000000000000005},"min":{"1min":0.63500000000000001,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.2370000000000001,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.95499999999999996}]},{"osd":9,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.998,"15min":1.026},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.2070000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.996},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.98099999999999998,"15min":1.0029999999999999},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.117,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.97699999999999998}]},{"osd":10,"last update":"Thu Nov 16 16:48:17 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.88900000000000001,"15min":0.89300000000000002},"min":{"1min":0.60899999999999999,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.671,"5min":1.671,"15min":1.671},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.004,"15min":1.0129999999999999},"min":{"1min":0.77200000000000002,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.54,"5min":1.54,"15min":1.54},"last":1.0600000000000001}]},{"osd":11,"last update":"Thu Nov 16 16:48:30 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":1.0820000000000001},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.702,"5min":1.702,"15min":1.702},"last":0.89900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7576,"kb_used_data":1040,"kb_used_omap":0,"kb_used_meta":6528,"kb_avail":93737576,"statfs":{"total":95995035648,"available":95987277824,"internally_reserved":0,"allocated":1064960,"data_stored":562324,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6684672},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:50 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.53000000000000003,"15min":0.51600000000000001},"min":{"1min":0.34699999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.71099999999999997,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.52300000000000002,"15min":0.498},"min":{"1min":0.35199999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.68400000000000005,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.54900000000000004}]},{"osd":2,"last update":"Thu Nov 16 16:48:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.53900000000000003,"15min":0.51500000000000001},"min":{"1min":0.38700000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.77700000000000002,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.56100000000000005,"15min":0.54700000000000004},"min":{"1min":0.35099999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82799999999999996,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":0.57799999999999996}]},{"osd":3,"last update":"Thu Nov 16 16:48:36 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58999999999999997,"15min":0.57399999999999995},"min":{"1min":0.441,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94099999999999995,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.625},{"interface":"front","average":{"1min":0.62,"5min":0.61599999999999999,"15min":0.60999999999999999},"min":{"1min":0.44500000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.97199999999999998,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.68100000000000005}]},{"osd":4,"last update":"Thu Nov 16 16:47:00 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.41999999999999998,"15min":0.40699999999999997},"min":{"1min":0.32700000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.498,"5min":0.45600000000000002,"15min":0.433},"min":{"1min":0.30599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.66100000000000003}]},{"osd":5,"last update":"Thu Nov 16 16:48:20 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.60699999999999998,"15min":0.56899999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.89300000000000002,"5min":1.04,"15min":1.04},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.63300000000000001,"15min":0.58899999999999997},"min":{"1min":0.42399999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.90000000000000002,"5min":1.577,"15min":1.577},"last":0.95899999999999996}]},{"osd":6,"last update":"Thu Nov 16 16:47:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.66800000000000004,"15min":0.67000000000000004},"min":{"1min":0.45700000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.94499999999999995,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.68899999999999995},"min":{"1min":0.47099999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.91300000000000003,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.76200000000000001}]},{"osd":7,"last update":"Thu Nov 16 16:48:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58299999999999996,"15min":0.57499999999999996},"min":{"1min":0.36599999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.90500000000000003,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58499999999999996,"15min":0.57799999999999996},"min":{"1min":0.35199999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.88800000000000001,"5min":1.48,"15min":1.48},"last":0.67000000000000004}]},{"osd":8,"last update":"Thu Nov 16 16:48:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.65600000000000003,"15min":0.63900000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.85399999999999998,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.745},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69499999999999995,"15min":0.69099999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.89200000000000002,"5min":1,"15min":1},"last":0.78800000000000003}]},{"osd":9,"last update":"Thu Nov 16 16:47:56 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.78500000000000003,"15min":0.79100000000000004},"min":{"1min":0.55200000000000005,"5min":0.504,"15min":0.504},"max":{"1min":0.96699999999999997,"5min":1.651,"15min":1.651},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.53300000000000003,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.02,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.80600000000000005}]},{"osd":10,"last update":"Thu Nov 16 16:48:15 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.92200000000000004,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.754,"15min":0.76100000000000001},"min":{"1min":0.53600000000000003,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.96899999999999997,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":1.1279999999999999}]},{"osd":11,"last update":"Thu Nov 16 16:48:26 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":1.095}]}]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7412,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737740,"statfs":{"total":95995035648,"available":95987445760,"internally_reserved":0,"allocated":1028096,"data_stored":506859,"data_compressed":20619,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Nov 16 16:47:56 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.59799999999999998,"15min":0.53400000000000003},"min":{"1min":0.35099999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.96399999999999997,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.60399999999999998,"15min":0.54200000000000004},"min":{"1min":0.378,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91200000000000003,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.77200000000000002}]},{"osd":2,"last update":"Thu Nov 16 16:48:15 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.61499999999999999,"15min":0.58899999999999997},"min":{"1min":0.46899999999999997,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.83299999999999996,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65100000000000002,"15min":0.59399999999999997},"min":{"1min":0.48299999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.103,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":0.93400000000000005}]},{"osd":3,"last update":"Thu Nov 16 16:48:27 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.65300000000000002}]},{"osd":4,"last update":"Thu Nov 16 16:47:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66400000000000003,"15min":0.621},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.96599999999999997,"5min":2.1709999999999998,"15min":2.1709999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.65500000000000003,"15min":0.63300000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.87,"5min":1.268,"15min":1.268},"last":0.56699999999999995}]},{"osd":5,"last update":"Thu Nov 16 16:48:20 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.437,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86699999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.73099999999999998,"15min":0.69999999999999996},"min":{"1min":0.54900000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.137,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":0.83399999999999996}]},{"osd":6,"last update":"Thu Nov 16 16:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.69199999999999995,"15min":0.67400000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73399999999999999,"15min":0.70899999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.71699999999999997}]},{"osd":7,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.552,"5min":1.552,"15min":1.552},"last":0.51900000000000002}]},{"osd":8,"last update":"Thu Nov 16 16:48:32 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.82199999999999995,"15min":0.81599999999999995},"min":{"1min":0.59199999999999997,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.851,"5min":2.145,"15min":2.145},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.84499999999999997,"15min":0.82499999999999996},"min":{"1min":0.71499999999999997,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.603,"5min":2.125,"15min":2.125},"last":0.78900000000000003}]},{"osd":9,"last update":"Thu Nov 16 16:47:56 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.81799999999999995,"15min":0.83299999999999996},"min":{"1min":0.48099999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.94699999999999995,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.92400000000000004,"15min":0.94899999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.97799999999999998,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.86899999999999999}]},{"osd":10,"last update":"Thu Nov 16 16:48:20 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.95299999999999996,"15min":0.96199999999999997},"min":{"1min":0.54700000000000004,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.5269999999999999,"5min":2.3159999999999998,"15min":2.3159999999999998},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.83899999999999997,"15min":0.84299999999999997},"min":{"1min":0.56599999999999995,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0209999999999999,"5min":1.381,"15min":1.381},"last":0.80000000000000004}]},{"osd":11,"last update":"Thu Nov 16 16:48:41 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.89400000000000002,"15min":0.876},"min":{"1min":0.71199999999999997,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.929,"5min":2.2189999999999999,"15min":2.2189999999999999},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.004,"5min":0.93200000000000005,"15min":0.92000000000000004},"min":{"1min":0.66200000000000003,"5min":0.63,"15min":0.63},"max":{"1min":1.629,"5min":2.21,"15min":2.21},"last":0.90100000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7664,"kb_used_data":1064,"kb_used_omap":0,"kb_used_meta":6592,"kb_avail":93737488,"statfs":{"total":95995035648,"available":95987187712,"internally_reserved":0,"allocated":1089536,"data_stored":573072,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6750208},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.499,"15min":0.46899999999999997},"min":{"1min":0.38800000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.71399999999999997,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.502,"15min":0.48399999999999999},"min":{"1min":0.376,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.69599999999999995}]},{"osd":1,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.52700000000000002,"15min":0.50600000000000001},"min":{"1min":0.35599999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.56000000000000005,"15min":0.51600000000000001},"min":{"1min":0.39500000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.81200000000000006}]},{"osd":3,"last update":"Thu Nov 16 16:48:41 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.60099999999999998,"15min":0.56699999999999995},"min":{"1min":0.441,"5min":0.38,"15min":0.38},"max":{"1min":0.85599999999999998,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.60599999999999998,"15min":0.57799999999999996},"min":{"1min":0.432,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.85499999999999998,"5min":1.016,"15min":1.016},"last":0.76200000000000001}]},{"osd":4,"last update":"Thu Nov 16 16:48:25 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.79700000000000004}]},{"osd":5,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.44,"15min":0.41399999999999998},"min":{"1min":0.28399999999999997,"5min":0.245,"15min":0.245},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.51600000000000001,"15min":0.47799999999999998},"min":{"1min":0.39400000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.66400000000000003}]},{"osd":6,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.63800000000000001,"15min":0.61599999999999999},"min":{"1min":0.56599999999999995,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.91100000000000003,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.64400000000000002,"15min":0.63300000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.88500000000000001,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.82199999999999995}]},{"osd":7,"last update":"Thu Nov 16 16:48:31 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.877},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.84899999999999998}]},{"osd":8,"last update":"Thu Nov 16 16:48:31 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.61199999999999999,"15min":0.59399999999999997},"min":{"1min":0.53500000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.93899999999999995,"5min":0.997,"15min":0.997},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.63600000000000001,"15min":0.61599999999999999},"min":{"1min":0.47999999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.86499999999999999}]},{"osd":9,"last update":"Thu Nov 16 16:47:57 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.72599999999999998,"15min":0.71999999999999997},"min":{"1min":0.55800000000000005,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0029999999999999,"5min":1.006,"15min":1.006},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.53500000000000003,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.88300000000000001,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.92500000000000004}]},{"osd":10,"last update":"Thu Nov 16 16:48:18 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.68500000000000005,"15min":0.67000000000000004},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.70299999999999996,"15min":0.68600000000000005},"min":{"1min":0.54300000000000004,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.97399999999999998}]},{"osd":11,"last update":"Thu Nov 16 16:48:25 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.89000000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6836,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738316,"statfs":{"total":95995035648,"available":95988035584,"internally_reserved":0,"allocated":1028096,"data_stored":515431,"data_compressed":23596,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.44900000000000001,"15min":0.40799999999999997},"min":{"1min":0.34599999999999997,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.90000000000000002,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.46400000000000002,"15min":0.42299999999999999},"min":{"1min":0.32500000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.53300000000000003}]},{"osd":1,"last update":"Thu Nov 16 16:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.65500000000000003,"15min":0.61299999999999999},"min":{"1min":0.46200000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94199999999999995,"5min":1.012,"15min":1.012},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.68300000000000005,"15min":0.621},"min":{"1min":0.52100000000000002,"5min":0.379,"15min":0.379},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.77700000000000002}]},{"osd":2,"last update":"Thu Nov 16 16:48:42 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.44800000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.621},{"interface":"front","average":{"1min":0.749,"5min":0.66700000000000004,"15min":0.625},"min":{"1min":0.50900000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.65100000000000002}]},{"osd":4,"last update":"Thu Nov 16 16:48:04 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.71599999999999997,"15min":0.71399999999999997},"min":{"1min":0.54900000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.748,"15min":0.74199999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.67700000000000005}]},{"osd":5,"last update":"Thu Nov 16 16:48:23 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.72599999999999998,"15min":0.71299999999999997},"min":{"1min":0.50900000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.71799999999999997,"15min":0.71999999999999997},"min":{"1min":0.56000000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.752}]},{"osd":6,"last update":"Thu Nov 16 16:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61199999999999999,"15min":0.60999999999999999},"min":{"1min":0.42699999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.495,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.64200000000000002}]},{"osd":7,"last update":"Thu Nov 16 16:48:11 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.76600000000000001,"15min":0.73699999999999999},"min":{"1min":0.497,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.72299999999999998,"15min":0.71699999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.80400000000000005}]},{"osd":8,"last update":"Thu Nov 16 16:46:26 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.81299999999999994}]},{"osd":9,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80600000000000005,"15min":0.80700000000000005},"min":{"1min":0.59699999999999998,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.036,"5min":1.123,"15min":1.123},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.91900000000000004,"15min":0.92400000000000004},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.82099999999999995}]},{"osd":10,"last update":"Thu Nov 16 16:48:17 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.92100000000000004,"15min":0.92600000000000005},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.879},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.86199999999999999,"15min":0.86799999999999999},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.149,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.89800000000000002}]},{"osd":11,"last update":"Thu Nov 16 16:48:27 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.84199999999999997}]}]},{"osd":4,"up_from":33,"seq":141733920829,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6620,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738532,"statfs":{"total":95995035648,"available":95988256768,"internally_reserved":0,"allocated":1003520,"data_stored":500721,"data_compressed":17642,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:47:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.504,"15min":0.49099999999999999},"min":{"1min":0.41299999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.53200000000000003,"15min":0.52600000000000002},"min":{"1min":0.35299999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.41699999999999998}]},{"osd":1,"last update":"Thu Nov 16 16:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.38500000000000001,"15min":0.35899999999999999},"min":{"1min":0.27700000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.56599999999999995,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.442},{"interface":"front","average":{"1min":0.436,"5min":0.40699999999999997,"15min":0.38800000000000001},"min":{"1min":0.29699999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64000000000000001,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.46999999999999997}]},{"osd":2,"last update":"Thu Nov 16 16:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.56100000000000005,"15min":0.54700000000000004},"min":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.623},{"interface":"front","average":{"1min":0.626,"5min":0.55300000000000005,"15min":0.53400000000000003},"min":{"1min":0.34599999999999997,"5min":0.307,"15min":0.307},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.63700000000000001}]},{"osd":3,"last update":"Thu Nov 16 16:48:05 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.57999999999999996,"15min":0.54900000000000004},"min":{"1min":0.44700000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85899999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57799999999999996,"15min":0.55800000000000005},"min":{"1min":0.436,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.57999999999999996}]},{"osd":5,"last update":"Thu Nov 16 16:48:22 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.58199999999999996,"15min":0.54900000000000004},"min":{"1min":0.376,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83099999999999996,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58299999999999996,"15min":0.55200000000000005},"min":{"1min":0.38900000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.67800000000000005}]},{"osd":6,"last update":"Thu Nov 16 16:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.61699999999999999,"15min":0.60699999999999998},"min":{"1min":0.47899999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.84399999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.66200000000000003,"15min":0.65100000000000002},"min":{"1min":0.49399999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.88500000000000001,"5min":0.995,"15min":0.995},"last":0.59099999999999997}]},{"osd":7,"last update":"Thu Nov 16 16:48:08 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51800000000000002,"15min":0.50900000000000001},"min":{"1min":0.34799999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65700000000000003,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.498},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60899999999999999,"15min":0.59799999999999998},"min":{"1min":0.45300000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.82999999999999996,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.51900000000000002}]},{"osd":8,"last update":"Thu Nov 16 16:48:31 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69599999999999995,"15min":0.69499999999999995},"min":{"1min":0.499,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.68400000000000005,"15min":0.68000000000000005},"min":{"1min":0.49199999999999999,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.751}]},{"osd":9,"last update":"Thu Nov 16 16:47:55 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.77500000000000002,"15min":0.78000000000000003},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.93000000000000005,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.752,"15min":0.75700000000000001},"min":{"1min":0.54300000000000004,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.71299999999999997}]},{"osd":10,"last update":"Thu Nov 16 16:48:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.72099999999999997,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.56999999999999995,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.83799999999999997}]},{"osd":11,"last update":"Thu Nov 16 16:48:26 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.69299999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757305,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6816,"kb_used_data":1048,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738336,"statfs":{"total":95995035648,"available":95988056064,"internally_reserved":0,"allocated":1073152,"data_stored":1101588,"data_compressed":53789,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Nov 16 16:48:21 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53000000000000003,"15min":0.52500000000000002},"min":{"1min":0.36799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.72199999999999998,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52600000000000002,"15min":0.51800000000000002},"min":{"1min":0.38100000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.69799999999999995,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.435}]},{"osd":1,"last update":"Thu Nov 16 16:48:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.53300000000000003,"15min":0.49299999999999999},"min":{"1min":0.47999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76600000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.54200000000000004,"15min":0.504},"min":{"1min":0.46400000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.499}]},{"osd":2,"last update":"Thu Nov 16 16:48:21 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.41399999999999998,"15min":0.39700000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.57199999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.43099999999999999,"15min":0.40699999999999997},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.64100000000000001,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.377}]},{"osd":3,"last update":"Thu Nov 16 16:48:21 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56399999999999995,"15min":0.55200000000000005},"min":{"1min":0.40000000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.79600000000000004,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57099999999999995,"15min":0.55600000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72099999999999997,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.48399999999999999}]},{"osd":4,"last update":"Thu Nov 16 16:48:21 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.59999999999999998,"15min":0.56000000000000005},"min":{"1min":0.48299999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83499999999999996,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.61599999999999999,"15min":0.57399999999999995},"min":{"1min":0.51300000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.84299999999999997,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.65100000000000002}]},{"osd":6,"last update":"Thu Nov 16 16:47:48 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.626,"15min":0.60999999999999999},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.82699999999999996,"5min":1.052,"15min":1.052},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.63100000000000001,"15min":0.61699999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.86899999999999999,"5min":1.117,"15min":1.117},"last":0.55800000000000005}]},{"osd":7,"last update":"Thu Nov 16 16:48:09 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.60699999999999998,"15min":0.59099999999999997},"min":{"1min":0.52500000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83999999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.62,"15min":0.60599999999999998},"min":{"1min":0.55600000000000005,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.85799999999999998,"5min":1.022,"15min":1.022},"last":0.59499999999999997}]},{"osd":8,"last update":"Thu Nov 16 16:48:34 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.59699999999999998,"15min":0.58399999999999996},"min":{"1min":0.39100000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.90700000000000003,"5min":0.995,"15min":0.995},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.63600000000000001,"15min":0.621},"min":{"1min":0.48999999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.88500000000000001,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.60699999999999998}]},{"osd":9,"last update":"Thu Nov 16 16:47:52 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.73999999999999999,"15min":0.73799999999999999},"min":{"1min":0.53100000000000003,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.94499999999999995,"5min":1.145,"15min":1.145},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.74399999999999999,"15min":0.745},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.89900000000000002,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.66900000000000004}]},{"osd":10,"last update":"Thu Nov 16 16:48:16 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.69099999999999995,"15min":0.67800000000000005},"min":{"1min":0.60799999999999998,"5min":0.437,"15min":0.437},"max":{"1min":0.92000000000000004,"5min":1.097,"15min":1.097},"last":0.629},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.71399999999999997,"15min":0.70199999999999996},"min":{"1min":0.61299999999999999,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.93799999999999994,"5min":1.272,"15min":1.272},"last":0.63900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34505,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34505,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36147,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":66685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":78375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":11816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":12474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":68605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-16T16:48:48.875 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-11-16T16:48:48.875 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-16T16:48:48.875 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-11-16T16:48:48.875 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph health --format=json 2023-11-16T16:48:49.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:48 smithi155 ceph-mon[131607]: pgmap v325: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 4 B/s, 2 keys/s, 0 objects/s recovering 2023-11-16T16:48:49.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:48 smithi002 ceph-mon[128712]: pgmap v325: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 4 B/s, 2 keys/s, 0 objects/s recovering 2023-11-16T16:48:49.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:48 smithi141 ceph-mon[131184]: pgmap v325: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 4 B/s, 2 keys/s, 0 objects/s recovering 2023-11-16T16:48:49.644 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:48:50.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:49 smithi155 ceph-mon[131607]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:48:50.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:49 smithi002 ceph-mon[128712]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:48:50.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:49 smithi141 ceph-mon[131184]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-16T16:48:50.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:50 smithi002 ceph-mon[128712]: pgmap v326: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 8.1 KiB/s, 2 keys/s, 1 objects/s recovering 2023-11-16T16:48:51.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:50 smithi155 ceph-mon[131607]: pgmap v326: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 8.1 KiB/s, 2 keys/s, 1 objects/s recovering 2023-11-16T16:48:51.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:50 smithi141 ceph-mon[131184]: pgmap v326: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 8.1 KiB/s, 2 keys/s, 1 objects/s recovering 2023-11-16T16:48:51.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.299+0000 7f770d85a700 1 -- 172.21.15.2:0/2718463197 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7708100ee0 msgr2=0x7f7708104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:51.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.299+0000 7f770d85a700 1 --2- 172.21.15.2:0/2718463197 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7708100ee0 0x7f7708104eb0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f76fc0099c0 tx=0x7f76fc009cd0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.299+0000 7f770d85a700 1 -- 172.21.15.2:0/2718463197 shutdown_connections 2023-11-16T16:48:51.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.299+0000 7f770d85a700 1 --2- 172.21.15.2:0/2718463197 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7708100540 0x7f7708100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.299+0000 7f770d85a700 1 --2- 172.21.15.2:0/2718463197 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f77081055e0 0x7f770810d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.299+0000 7f770d85a700 1 --2- 172.21.15.2:0/2718463197 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7708100ee0 0x7f7708104eb0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.299+0000 7f770d85a700 1 -- 172.21.15.2:0/2718463197 >> 172.21.15.2:0/2718463197 conn(0x7f77080fbdc0 msgr2=0x7f77080fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:51.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.299+0000 7f770d85a700 1 -- 172.21.15.2:0/2718463197 shutdown_connections 2023-11-16T16:48:51.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.299+0000 7f770d85a700 1 -- 172.21.15.2:0/2718463197 wait complete. 2023-11-16T16:48:51.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.300+0000 7f770d85a700 1 Processor -- start 2023-11-16T16:48:51.302 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.300+0000 7f770d85a700 1 -- start start 2023-11-16T16:48:51.303 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.300+0000 7f770d85a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7708100540 0x7f7708072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:51.303 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.300+0000 7f770d85a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7708100ee0 0x7f7708073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:51.303 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.300+0000 7f770d85a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77081055e0 0x7f77080738b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:51.303 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.300+0000 7f770d85a700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77081a4840 con 0x7f7708100540 2023-11-16T16:48:51.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.300+0000 7f770d85a700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77081a49b0 con 0x7f7708100ee0 2023-11-16T16:48:51.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.300+0000 7f770d85a700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77081a4b70 con 0x7f77081055e0 2023-11-16T16:48:51.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.301+0000 7f7706ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7708100540 0x7f7708072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:51.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.301+0000 7f7706ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7708100540 0x7f7708072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53354/0 (socket says 172.21.15.2:53354) 2023-11-16T16:48:51.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.301+0000 7f7706ffd700 1 -- 172.21.15.2:0/667637960 learned_addr learned my addr 172.21.15.2:0/667637960 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:51.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.301+0000 7f77067fc700 1 --2- 172.21.15.2:0/667637960 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7708100ee0 0x7f7708073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:51.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.301+0000 7f77077fe700 1 --2- 172.21.15.2:0/667637960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77081055e0 0x7f77080738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:51.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.301+0000 7f7706ffd700 1 -- 172.21.15.2:0/667637960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77081055e0 msgr2=0x7f77080738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:51.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.301+0000 7f7706ffd700 1 --2- 172.21.15.2:0/667637960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77081055e0 0x7f77080738b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.301+0000 7f7706ffd700 1 -- 172.21.15.2:0/667637960 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7708100ee0 msgr2=0x7f7708073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:51.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.301+0000 7f7706ffd700 1 --2- 172.21.15.2:0/667637960 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7708100ee0 0x7f7708073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.301+0000 7f7706ffd700 1 -- 172.21.15.2:0/667637960 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76fc0095f0 con 0x7f7708100540 2023-11-16T16:48:51.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.301+0000 7f77077fe700 1 --2- 172.21.15.2:0/667637960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77081055e0 0x7f77080738b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:48:51.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.301+0000 7f7706ffd700 1 --2- 172.21.15.2:0/667637960 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7708100540 0x7f7708072df0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f76f800e9a0 tx=0x7f76f800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:51.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.302+0000 7f770c858700 1 -- 172.21.15.2:0/667637960 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f8009790 con 0x7f7708100540 2023-11-16T16:48:51.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.302+0000 7f770c858700 1 -- 172.21.15.2:0/667637960 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f76f800cbd0 con 0x7f7708100540 2023-11-16T16:48:51.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.302+0000 7f770d85a700 1 -- 172.21.15.2:0/667637960 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7708110360 con 0x7f7708100540 2023-11-16T16:48:51.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.302+0000 7f770c858700 1 -- 172.21.15.2:0/667637960 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f8018680 con 0x7f7708100540 2023-11-16T16:48:51.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.302+0000 7f770d85a700 1 -- 172.21.15.2:0/667637960 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77081101d0 con 0x7f7708100540 2023-11-16T16:48:51.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.303+0000 7f770d85a700 1 -- 172.21.15.2:0/667637960 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77081104f0 con 0x7f7708100540 2023-11-16T16:48:51.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.305+0000 7f770c858700 1 -- 172.21.15.2:0/667637960 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f76f800cd40 con 0x7f7708100540 2023-11-16T16:48:51.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.306+0000 7f770c858700 1 --2- 172.21.15.2:0/667637960 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f76f00a1260 0x7f76f00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:51.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.306+0000 7f770c858700 1 -- 172.21.15.2:0/667637960 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f76f8014070 con 0x7f7708100540 2023-11-16T16:48:51.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.306+0000 7f77067fc700 1 --2- 172.21.15.2:0/667637960 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f76f00a1260 0x7f76f00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:51.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.307+0000 7f77067fc700 1 --2- 172.21.15.2:0/667637960 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f76f00a1260 0x7f76f00a3710 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f76fc0098e0 tx=0x7f76fc000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:51.312 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.310+0000 7f770c858700 1 -- 172.21.15.2:0/667637960 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f76f8051730 con 0x7f7708100540 2023-11-16T16:48:51.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.561+0000 7f770d85a700 1 -- 172.21.15.2:0/667637960 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7708066ea0 con 0x7f7708100540 2023-11-16T16:48:51.564 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.562+0000 7f770c858700 1 -- 172.21.15.2:0/667637960 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f76f80962b0 con 0x7f7708100540 2023-11-16T16:48:51.564 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:48:51.564 INFO:teuthology.orchestra.run.smithi002.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-16T16:48:51.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.564+0000 7f770d85a700 1 -- 172.21.15.2:0/667637960 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f76f00a1260 msgr2=0x7f76f00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:51.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.564+0000 7f770d85a700 1 --2- 172.21.15.2:0/667637960 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f76f00a1260 0x7f76f00a3710 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f76fc0098e0 tx=0x7f76fc000bc0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.564+0000 7f770d85a700 1 -- 172.21.15.2:0/667637960 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7708100540 msgr2=0x7f7708072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:51.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.564+0000 7f770d85a700 1 --2- 172.21.15.2:0/667637960 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7708100540 0x7f7708072df0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f76f800e9a0 tx=0x7f76f800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.564+0000 7f770d85a700 1 -- 172.21.15.2:0/667637960 shutdown_connections 2023-11-16T16:48:51.566 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.564+0000 7f770d85a700 1 --2- 172.21.15.2:0/667637960 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f76f00a1260 0x7f76f00a3710 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.564+0000 7f770d85a700 1 --2- 172.21.15.2:0/667637960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77081055e0 0x7f77080738b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.564+0000 7f770d85a700 1 --2- 172.21.15.2:0/667637960 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7708100540 0x7f7708072df0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.564+0000 7f770d85a700 1 --2- 172.21.15.2:0/667637960 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7708100ee0 0x7f7708073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:51.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.564+0000 7f770d85a700 1 -- 172.21.15.2:0/667637960 >> 172.21.15.2:0/667637960 conn(0x7f77080fbdc0 msgr2=0x7f77080fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:51.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.564+0000 7f770d85a700 1 -- 172.21.15.2:0/667637960 shutdown_connections 2023-11-16T16:48:51.567 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:51.564+0000 7f770d85a700 1 -- 172.21.15.2:0/667637960 wait complete. 2023-11-16T16:48:52.146 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:51 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/667637960' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-11-16T16:48:52.149 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-11-16T16:48:52.149 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:48:52.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:51 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/667637960' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-11-16T16:48:52.357 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.355+0000 7f28545ff700 1 -- 172.21.15.2:0/617987986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f284c100290 msgr2=0x7f284c1006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.357 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.355+0000 7f28545ff700 1 --2- 172.21.15.2:0/617987986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f284c100290 0x7f284c1006a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2848009990 tx=0x7f2848009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.357 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.355+0000 7f28545ff700 1 -- 172.21.15.2:0/617987986 shutdown_connections 2023-11-16T16:48:52.357 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.355+0000 7f28545ff700 1 --2- 172.21.15.2:0/617987986 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f284c101e10 0x7f284c1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.355+0000 7f28545ff700 1 --2- 172.21.15.2:0/617987986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f284c100290 0x7f284c1006a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.355+0000 7f28545ff700 1 --2- 172.21.15.2:0/617987986 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f284c101480 0x7f284c1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.355+0000 7f28545ff700 1 -- 172.21.15.2:0/617987986 >> 172.21.15.2:0/617987986 conn(0x7f284c0fb870 msgr2=0x7f284c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:52.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.356+0000 7f28545ff700 1 -- 172.21.15.2:0/617987986 shutdown_connections 2023-11-16T16:48:52.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.356+0000 7f28545ff700 1 -- 172.21.15.2:0/617987986 wait complete. 2023-11-16T16:48:52.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.357+0000 7f28545ff700 1 Processor -- start 2023-11-16T16:48:52.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.357+0000 7f28545ff700 1 -- start start 2023-11-16T16:48:52.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.357+0000 7f28545ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f284c100290 0x7f284c1981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.357+0000 7f28545ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f284c101480 0x7f284c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.357+0000 7f28545ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f284c101e10 0x7f284c19f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.357+0000 7f28545ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c19fd20 con 0x7f284c100290 2023-11-16T16:48:52.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.357+0000 7f28545ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c19fe60 con 0x7f284c101e10 2023-11-16T16:48:52.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.357+0000 7f28545ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c19fff0 con 0x7f284c101480 2023-11-16T16:48:52.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.357+0000 7f285239b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f284c100290 0x7f284c1981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.357+0000 7f285239b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f284c100290 0x7f284c1981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53376/0 (socket says 172.21.15.2:53376) 2023-11-16T16:48:52.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.357+0000 7f285239b700 1 -- 172.21.15.2:0/1590335660 learned_addr learned my addr 172.21.15.2:0/1590335660 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:52.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.357+0000 7f2851b9a700 1 --2- 172.21.15.2:0/1590335660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f284c101480 0x7f284c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.358+0000 7f2852b9c700 1 --2- 172.21.15.2:0/1590335660 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f284c101e10 0x7f284c19f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.358+0000 7f285239b700 1 -- 172.21.15.2:0/1590335660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f284c101480 msgr2=0x7f284c1986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.358+0000 7f285239b700 1 --2- 172.21.15.2:0/1590335660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f284c101480 0x7f284c1986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.358+0000 7f285239b700 1 -- 172.21.15.2:0/1590335660 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f284c101e10 msgr2=0x7f284c19f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.358+0000 7f285239b700 1 --2- 172.21.15.2:0/1590335660 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f284c101e10 0x7f284c19f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.358+0000 7f285239b700 1 -- 172.21.15.2:0/1590335660 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28480095f0 con 0x7f284c100290 2023-11-16T16:48:52.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.358+0000 7f285239b700 1 --2- 172.21.15.2:0/1590335660 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f284c100290 0x7f284c1981a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f284800b620 tx=0x7f2848005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:52.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.358+0000 7f28437fe700 1 -- 172.21.15.2:0/1590335660 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284801d070 con 0x7f284c100290 2023-11-16T16:48:52.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.358+0000 7f28437fe700 1 -- 172.21.15.2:0/1590335660 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2848021410 con 0x7f284c100290 2023-11-16T16:48:52.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.358+0000 7f28545ff700 1 -- 172.21.15.2:0/1590335660 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f284c10bc40 con 0x7f284c100290 2023-11-16T16:48:52.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.358+0000 7f28437fe700 1 -- 172.21.15.2:0/1590335660 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2848017650 con 0x7f284c100290 2023-11-16T16:48:52.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.358+0000 7f28545ff700 1 -- 172.21.15.2:0/1590335660 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f284c10bab0 con 0x7f284c100290 2023-11-16T16:48:52.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.360+0000 7f28545ff700 1 -- 172.21.15.2:0/1590335660 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f284c1a05f0 con 0x7f284c100290 2023-11-16T16:48:52.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.362+0000 7f28437fe700 1 -- 172.21.15.2:0/1590335660 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f2848021580 con 0x7f284c100290 2023-11-16T16:48:52.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.364+0000 7f28437fe700 1 --2- 172.21.15.2:0/1590335660 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f28380a1260 0x7f28380a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.364+0000 7f28437fe700 1 -- 172.21.15.2:0/1590335660 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f28480ce300 con 0x7f284c100290 2023-11-16T16:48:52.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.364+0000 7f2851b9a700 1 --2- 172.21.15.2:0/1590335660 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f28380a1260 0x7f28380a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.365+0000 7f2851b9a700 1 --2- 172.21.15.2:0/1590335660 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f28380a1260 0x7f28380a3710 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f283c005950 tx=0x7f283c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:52.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:51 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/667637960' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-11-16T16:48:52.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.368+0000 7f28437fe700 1 -- 172.21.15.2:0/1590335660 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f28480975c0 con 0x7f284c100290 2023-11-16T16:48:52.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.580+0000 7f28545ff700 1 -- 172.21.15.2:0/1590335660 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f284c1a0c90 con 0x7f284c100290 2023-11-16T16:48:52.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.582+0000 7f28437fe700 1 -- 172.21.15.2:0/1590335660 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24567 (secure 0 0 0) 0x7f284809abe0 con 0x7f284c100290 2023-11-16T16:48:52.584 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:48:52.584 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":93,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:48:45.491739+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:24.458452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:25.786005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:29.313181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:48:52.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.584+0000 7f28545ff700 1 -- 172.21.15.2:0/1590335660 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f28380a1260 msgr2=0x7f28380a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.584+0000 7f28545ff700 1 --2- 172.21.15.2:0/1590335660 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f28380a1260 0x7f28380a3710 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f283c005950 tx=0x7f283c00b410 comp rx=0 tx=0).stop 2023-11-16T16:48:52.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.584+0000 7f28545ff700 1 -- 172.21.15.2:0/1590335660 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f284c100290 msgr2=0x7f284c1981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.584+0000 7f28545ff700 1 --2- 172.21.15.2:0/1590335660 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f284c100290 0x7f284c1981a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f284800b620 tx=0x7f2848005740 comp rx=0 tx=0).stop 2023-11-16T16:48:52.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.584+0000 7f28545ff700 1 -- 172.21.15.2:0/1590335660 shutdown_connections 2023-11-16T16:48:52.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.584+0000 7f28545ff700 1 --2- 172.21.15.2:0/1590335660 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f28380a1260 0x7f28380a3710 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.586 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.584+0000 7f28545ff700 1 --2- 172.21.15.2:0/1590335660 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f284c101e10 0x7f284c19f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.584+0000 7f28545ff700 1 --2- 172.21.15.2:0/1590335660 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f284c100290 0x7f284c1981a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.584+0000 7f28545ff700 1 --2- 172.21.15.2:0/1590335660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f284c101480 0x7f284c1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.584+0000 7f28545ff700 1 -- 172.21.15.2:0/1590335660 >> 172.21.15.2:0/1590335660 conn(0x7f284c0fb870 msgr2=0x7f284c0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:52.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.584+0000 7f28545ff700 1 -- 172.21.15.2:0/1590335660 shutdown_connections 2023-11-16T16:48:52.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.584+0000 7f28545ff700 1 -- 172.21.15.2:0/1590335660 wait complete. 2023-11-16T16:48:52.605 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:04.306584+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T16:47:24.458452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T16:47:25.786005+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T16:47:29.313181+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T16:48:52.605 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:48:52.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.811+0000 7ff18a3ff700 1 -- 172.21.15.2:0/737301416 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1841002c0 msgr2=0x7ff1841006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.811+0000 7ff18a3ff700 1 --2- 172.21.15.2:0/737301416 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1841002c0 0x7ff1841006d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff174009990 tx=0x7ff174009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.811+0000 7ff18a3ff700 1 -- 172.21.15.2:0/737301416 shutdown_connections 2023-11-16T16:48:52.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.811+0000 7ff18a3ff700 1 --2- 172.21.15.2:0/737301416 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff184101e40 0x7ff184108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.811+0000 7ff18a3ff700 1 --2- 172.21.15.2:0/737301416 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1841002c0 0x7ff1841006d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.811+0000 7ff18a3ff700 1 --2- 172.21.15.2:0/737301416 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1841014b0 0x7ff184101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.815 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.811+0000 7ff18a3ff700 1 -- 172.21.15.2:0/737301416 >> 172.21.15.2:0/737301416 conn(0x7ff1840fb8a0 msgr2=0x7ff1840fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:52.815 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.811+0000 7ff18a3ff700 1 -- 172.21.15.2:0/737301416 shutdown_connections 2023-11-16T16:48:52.815 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.811+0000 7ff18a3ff700 1 -- 172.21.15.2:0/737301416 wait complete. 2023-11-16T16:48:52.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.812+0000 7ff18a3ff700 1 Processor -- start 2023-11-16T16:48:52.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.813+0000 7ff18a3ff700 1 -- start start 2023-11-16T16:48:52.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.813+0000 7ff18a3ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1841002c0 0x7ff184198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.813+0000 7ff18a3ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1841014b0 0x7ff1841987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.813+0000 7ff18a3ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff184101e40 0x7ff18419f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.813+0000 7ff18a3ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18419fe00 con 0x7ff1841002c0 2023-11-16T16:48:52.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.813+0000 7ff18a3ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18419ff40 con 0x7ff1841014b0 2023-11-16T16:48:52.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.813+0000 7ff18a3ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1841a00d0 con 0x7ff184101e40 2023-11-16T16:48:52.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.813+0000 7ff183fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1841002c0 0x7ff184198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.813+0000 7ff183fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1841002c0 0x7ff184198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53400/0 (socket says 172.21.15.2:53400) 2023-11-16T16:48:52.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.813+0000 7ff183fff700 1 -- 172.21.15.2:0/3034522638 learned_addr learned my addr 172.21.15.2:0/3034522638 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:52.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.813+0000 7ff1837fe700 1 --2- 172.21.15.2:0/3034522638 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1841014b0 0x7ff1841987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.813+0000 7ff18899c700 1 --2- 172.21.15.2:0/3034522638 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff184101e40 0x7ff18419f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.814+0000 7ff183fff700 1 -- 172.21.15.2:0/3034522638 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff184101e40 msgr2=0x7ff18419f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.814+0000 7ff183fff700 1 --2- 172.21.15.2:0/3034522638 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff184101e40 0x7ff18419f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.814+0000 7ff183fff700 1 -- 172.21.15.2:0/3034522638 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1841014b0 msgr2=0x7ff1841987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:52.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.814+0000 7ff183fff700 1 --2- 172.21.15.2:0/3034522638 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1841014b0 0x7ff1841987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:52.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.814+0000 7ff183fff700 1 -- 172.21.15.2:0/3034522638 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1740095f0 con 0x7ff1841002c0 2023-11-16T16:48:52.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.814+0000 7ff1837fe700 1 --2- 172.21.15.2:0/3034522638 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1841014b0 0x7ff1841987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:48:52.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.814+0000 7ff183fff700 1 --2- 172.21.15.2:0/3034522638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1841002c0 0x7ff184198280 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff174004760 tx=0x7ff174009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:52.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.814+0000 7ff1817fa700 1 -- 172.21.15.2:0/3034522638 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17401d070 con 0x7ff1841002c0 2023-11-16T16:48:52.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.814+0000 7ff1817fa700 1 -- 172.21.15.2:0/3034522638 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff174004120 con 0x7ff1841002c0 2023-11-16T16:48:52.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.814+0000 7ff1817fa700 1 -- 172.21.15.2:0/3034522638 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff174005290 con 0x7ff1841002c0 2023-11-16T16:48:52.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.814+0000 7ff18a3ff700 1 -- 172.21.15.2:0/3034522638 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff18410bc90 con 0x7ff1841002c0 2023-11-16T16:48:52.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.814+0000 7ff18a3ff700 1 -- 172.21.15.2:0/3034522638 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18410bb00 con 0x7ff1841002c0 2023-11-16T16:48:52.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.815+0000 7ff18a3ff700 1 -- 172.21.15.2:0/3034522638 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1841929f0 con 0x7ff1841002c0 2023-11-16T16:48:52.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.817+0000 7ff1817fa700 1 -- 172.21.15.2:0/3034522638 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7ff174004810 con 0x7ff1841002c0 2023-11-16T16:48:52.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.819+0000 7ff1817fa700 1 --2- 172.21.15.2:0/3034522638 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff16c0a1260 0x7ff16c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:52.823 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.819+0000 7ff1817fa700 1 -- 172.21.15.2:0/3034522638 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7ff1740cde40 con 0x7ff1841002c0 2023-11-16T16:48:52.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.819+0000 7ff1837fe700 1 --2- 172.21.15.2:0/3034522638 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff16c0a1260 0x7ff16c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:52.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.820+0000 7ff1837fe700 1 --2- 172.21.15.2:0/3034522638 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff16c0a1260 0x7ff16c0a3710 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff178005fd0 tx=0x7ff178005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:52.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:52.822+0000 7ff1817fa700 1 -- 172.21.15.2:0/3034522638 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff174097100 con 0x7ff1841002c0 2023-11-16T16:48:53.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.034+0000 7ff18a3ff700 1 -- 172.21.15.2:0/3034522638 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff184061700 con 0x7ff1841002c0 2023-11-16T16:48:53.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.034+0000 7ff1817fa700 1 -- 172.21.15.2:0/3034522638 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff17409a720 con 0x7ff1841002c0 2023-11-16T16:48:53.036 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T16:48:53.038 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.036+0000 7ff18a3ff700 1 -- 172.21.15.2:0/3034522638 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff16c0a1260 msgr2=0x7ff16c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:53.038 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.036+0000 7ff18a3ff700 1 --2- 172.21.15.2:0/3034522638 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff16c0a1260 0x7ff16c0a3710 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff178005fd0 tx=0x7ff178005f60 comp rx=0 tx=0).stop 2023-11-16T16:48:53.038 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.036+0000 7ff18a3ff700 1 -- 172.21.15.2:0/3034522638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1841002c0 msgr2=0x7ff184198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:53.038 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.036+0000 7ff18a3ff700 1 --2- 172.21.15.2:0/3034522638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1841002c0 0x7ff184198280 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff174004760 tx=0x7ff174009580 comp rx=0 tx=0).stop 2023-11-16T16:48:53.039 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.036+0000 7ff18a3ff700 1 -- 172.21.15.2:0/3034522638 shutdown_connections 2023-11-16T16:48:53.039 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.036+0000 7ff18a3ff700 1 --2- 172.21.15.2:0/3034522638 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff16c0a1260 0x7ff16c0a3710 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.039 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.036+0000 7ff18a3ff700 1 --2- 172.21.15.2:0/3034522638 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff184101e40 0x7ff18419f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.039 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.036+0000 7ff18a3ff700 1 --2- 172.21.15.2:0/3034522638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1841002c0 0x7ff184198280 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.039 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.036+0000 7ff18a3ff700 1 --2- 172.21.15.2:0/3034522638 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff1841014b0 0x7ff1841987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.039 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.036+0000 7ff18a3ff700 1 -- 172.21.15.2:0/3034522638 >> 172.21.15.2:0/3034522638 conn(0x7ff1840fb8a0 msgr2=0x7ff184106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:53.039 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.036+0000 7ff18a3ff700 1 -- 172.21.15.2:0/3034522638 shutdown_connections 2023-11-16T16:48:53.039 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.037+0000 7ff18a3ff700 1 -- 172.21.15.2:0/3034522638 wait complete. 2023-11-16T16:48:53.057 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T16:48:53.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:52 smithi002 ceph-mon[128712]: pgmap v327: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 8.1 KiB/s, 2 keys/s, 1 objects/s recovering 2023-11-16T16:48:53.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:52 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1590335660' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:53.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:52 smithi141 ceph-mon[131184]: pgmap v327: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 8.1 KiB/s, 2 keys/s, 1 objects/s recovering 2023-11-16T16:48:53.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:52 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1590335660' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:53.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.264+0000 7f5cc29ff700 1 -- 172.21.15.2:0/1851428738 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc1014b0 msgr2=0x7f5cbc101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:53.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.264+0000 7f5cc29ff700 1 --2- 172.21.15.2:0/1851428738 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc1014b0 0x7f5cbc101900 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5cb0009990 tx=0x7f5cb0009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.264+0000 7f5cc29ff700 1 -- 172.21.15.2:0/1851428738 shutdown_connections 2023-11-16T16:48:53.266 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.264+0000 7f5cc29ff700 1 --2- 172.21.15.2:0/1851428738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5cbc1002c0 0x7f5cbc1006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.264+0000 7f5cc29ff700 1 --2- 172.21.15.2:0/1851428738 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5cbc101e40 0x7f5cbc108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.264+0000 7f5cc29ff700 1 --2- 172.21.15.2:0/1851428738 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc1014b0 0x7f5cbc101900 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.264+0000 7f5cc29ff700 1 -- 172.21.15.2:0/1851428738 >> 172.21.15.2:0/1851428738 conn(0x7f5cbc0fb8a0 msgr2=0x7f5cbc0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:53.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.264+0000 7f5cc29ff700 1 -- 172.21.15.2:0/1851428738 shutdown_connections 2023-11-16T16:48:53.267 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.264+0000 7f5cc29ff700 1 -- 172.21.15.2:0/1851428738 wait complete. 2023-11-16T16:48:53.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.265+0000 7f5cc29ff700 1 Processor -- start 2023-11-16T16:48:53.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.266+0000 7f5cc29ff700 1 -- start start 2023-11-16T16:48:53.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.266+0000 7f5cc29ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc1002c0 0x7f5cbc1960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:53.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.266+0000 7f5cc29ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5cbc1014b0 0x7f5cbc1965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:53.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.266+0000 7f5cc29ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5cbc101e40 0x7f5cbc19d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:53.269 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.266+0000 7f5cc29ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cbc19dc30 con 0x7f5cbc1002c0 2023-11-16T16:48:53.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.266+0000 7f5cc29ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cbc19dd70 con 0x7f5cbc1014b0 2023-11-16T16:48:53.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.266+0000 7f5cc29ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cbc19df00 con 0x7f5cbc101e40 2023-11-16T16:48:53.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.266+0000 7f5cbbfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc1002c0 0x7f5cbc1960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:53.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.266+0000 7f5cbbfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc1002c0 0x7f5cbc1960b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53418/0 (socket says 172.21.15.2:53418) 2023-11-16T16:48:53.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.266+0000 7f5cbbfff700 1 -- 172.21.15.2:0/3787203225 learned_addr learned my addr 172.21.15.2:0/3787203225 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:53.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.266+0000 7f5cc0f9c700 1 --2- 172.21.15.2:0/3787203225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5cbc101e40 0x7f5cbc19d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:53.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.266+0000 7f5cbb7fe700 1 --2- 172.21.15.2:0/3787203225 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5cbc1014b0 0x7f5cbc1965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:53.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.267+0000 7f5cbbfff700 1 -- 172.21.15.2:0/3787203225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5cbc101e40 msgr2=0x7f5cbc19d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:53.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.267+0000 7f5cbbfff700 1 --2- 172.21.15.2:0/3787203225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5cbc101e40 0x7f5cbc19d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.270 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.267+0000 7f5cbbfff700 1 -- 172.21.15.2:0/3787203225 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5cbc1014b0 msgr2=0x7f5cbc1965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:53.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.267+0000 7f5cbbfff700 1 --2- 172.21.15.2:0/3787203225 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5cbc1014b0 0x7f5cbc1965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.267+0000 7f5cbbfff700 1 -- 172.21.15.2:0/3787203225 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb00095f0 con 0x7f5cbc1002c0 2023-11-16T16:48:53.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.267+0000 7f5cbbfff700 1 --2- 172.21.15.2:0/3787203225 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc1002c0 0x7f5cbc1960b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5cac00e970 tx=0x7f5cac00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:53.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.267+0000 7f5cb97fa700 1 -- 172.21.15.2:0/3787203225 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cac009790 con 0x7f5cbc1002c0 2023-11-16T16:48:53.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.267+0000 7f5cc29ff700 1 -- 172.21.15.2:0/3787203225 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cbc10bc70 con 0x7f5cbc1002c0 2023-11-16T16:48:53.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.267+0000 7f5cb97fa700 1 -- 172.21.15.2:0/3787203225 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5cac00c7e0 con 0x7f5cbc1002c0 2023-11-16T16:48:53.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.267+0000 7f5cc29ff700 1 -- 172.21.15.2:0/3787203225 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cbc10bae0 con 0x7f5cbc1002c0 2023-11-16T16:48:53.271 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.267+0000 7f5cb97fa700 1 -- 172.21.15.2:0/3787203225 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cac003c10 con 0x7f5cbc1002c0 2023-11-16T16:48:53.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.269+0000 7f5cc29ff700 1 -- 172.21.15.2:0/3787203225 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cbc107880 con 0x7f5cbc1002c0 2023-11-16T16:48:53.272 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.270+0000 7f5cb97fa700 1 -- 172.21.15.2:0/3787203225 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f5cac00cc50 con 0x7f5cbc1002c0 2023-11-16T16:48:53.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.272+0000 7f5cb97fa700 1 --2- 172.21.15.2:0/3787203225 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5ca40a11e0 0x7f5ca40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:53.273 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.272+0000 7f5cb97fa700 1 -- 172.21.15.2:0/3787203225 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f5cac014070 con 0x7f5cbc1002c0 2023-11-16T16:48:53.274 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.272+0000 7f5cbb7fe700 1 --2- 172.21.15.2:0/3787203225 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5ca40a11e0 0x7f5ca40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:53.274 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.273+0000 7f5cbb7fe700 1 --2- 172.21.15.2:0/3787203225 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5ca40a11e0 0x7f5ca40a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5cb000b2e0 tx=0x7f5cb0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:53.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.276+0000 7f5cb97fa700 1 -- 172.21.15.2:0/3787203225 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5cac0961d0 con 0x7f5cbc1002c0 2023-11-16T16:48:53.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:52 smithi155 ceph-mon[131607]: pgmap v327: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 8.1 KiB/s, 2 keys/s, 1 objects/s recovering 2023-11-16T16:48:53.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:52 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1590335660' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:53.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.485+0000 7f5cc29ff700 1 -- 172.21.15.2:0/3787203225 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5cbc061700 con 0x7f5cbc1002c0 2023-11-16T16:48:53.487 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.486+0000 7f5cb97fa700 1 -- 172.21.15.2:0/3787203225 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5cac0997f0 con 0x7f5cbc1002c0 2023-11-16T16:48:53.488 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T16:48:53.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.488+0000 7f5cc29ff700 1 -- 172.21.15.2:0/3787203225 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5ca40a11e0 msgr2=0x7f5ca40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:53.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.488+0000 7f5cc29ff700 1 --2- 172.21.15.2:0/3787203225 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5ca40a11e0 0x7f5ca40a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5cb000b2e0 tx=0x7f5cb0005fb0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.488+0000 7f5cc29ff700 1 -- 172.21.15.2:0/3787203225 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc1002c0 msgr2=0x7f5cbc1960b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:53.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.488+0000 7f5cc29ff700 1 --2- 172.21.15.2:0/3787203225 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc1002c0 0x7f5cbc1960b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5cac00e970 tx=0x7f5cac00ec80 comp rx=0 tx=0).stop 2023-11-16T16:48:53.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.488+0000 7f5cc29ff700 1 -- 172.21.15.2:0/3787203225 shutdown_connections 2023-11-16T16:48:53.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.488+0000 7f5cc29ff700 1 --2- 172.21.15.2:0/3787203225 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5ca40a11e0 0x7f5ca40a3690 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.488+0000 7f5cc29ff700 1 --2- 172.21.15.2:0/3787203225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5cbc101e40 0x7f5cbc19d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.488+0000 7f5cc29ff700 1 --2- 172.21.15.2:0/3787203225 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5cbc1002c0 0x7f5cbc1960b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.488+0000 7f5cc29ff700 1 --2- 172.21.15.2:0/3787203225 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5cbc1014b0 0x7f5cbc1965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.488+0000 7f5cc29ff700 1 -- 172.21.15.2:0/3787203225 >> 172.21.15.2:0/3787203225 conn(0x7f5cbc0fb8a0 msgr2=0x7f5cbc1049c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:53.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.488+0000 7f5cc29ff700 1 -- 172.21.15.2:0/3787203225 shutdown_connections 2023-11-16T16:48:53.491 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.488+0000 7f5cc29ff700 1 -- 172.21.15.2:0/3787203225 wait complete. 2023-11-16T16:48:53.509 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T16:48:53.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.715+0000 7f08db5ff700 1 -- 172.21.15.2:0/1944079225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d41002c0 msgr2=0x7f08d41006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:53.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.715+0000 7f08db5ff700 1 --2- 172.21.15.2:0/1944079225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d41002c0 0x7f08d41006d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f08d00098f0 tx=0x7f08d0009c00 comp rx=0 tx=0).stop 2023-11-16T16:48:53.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.715+0000 7f08db5ff700 1 -- 172.21.15.2:0/1944079225 shutdown_connections 2023-11-16T16:48:53.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.715+0000 7f08db5ff700 1 --2- 172.21.15.2:0/1944079225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d41002c0 0x7f08d41006d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.717 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.715+0000 7f08db5ff700 1 --2- 172.21.15.2:0/1944079225 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08d4101e40 0x7f08d4108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.718 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.715+0000 7f08db5ff700 1 --2- 172.21.15.2:0/1944079225 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08d41014b0 0x7f08d4101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.718 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.715+0000 7f08db5ff700 1 -- 172.21.15.2:0/1944079225 >> 172.21.15.2:0/1944079225 conn(0x7f08d40fb8a0 msgr2=0x7f08d40fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:53.718 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.715+0000 7f08db5ff700 1 -- 172.21.15.2:0/1944079225 shutdown_connections 2023-11-16T16:48:53.718 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.715+0000 7f08db5ff700 1 -- 172.21.15.2:0/1944079225 wait complete. 2023-11-16T16:48:53.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.716+0000 7f08db5ff700 1 Processor -- start 2023-11-16T16:48:53.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.716+0000 7f08db5ff700 1 -- start start 2023-11-16T16:48:53.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.717+0000 7f08db5ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08d41002c0 0x7f08d40748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:53.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.717+0000 7f08db5ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d41014b0 0x7f08d4072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:53.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.717+0000 7f08db5ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08d4101e40 0x7f08d4073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:53.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.717+0000 7f08db5ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d4073bc0 con 0x7f08d4101e40 2023-11-16T16:48:53.720 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.717+0000 7f08db5ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d4073d30 con 0x7f08d41002c0 2023-11-16T16:48:53.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.717+0000 7f08db5ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d41a0160 con 0x7f08d41014b0 2023-11-16T16:48:53.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.717+0000 7f08d8b9a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d41014b0 0x7f08d4072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:53.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.717+0000 7f08d8b9a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d41014b0 0x7f08d4072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:33056/0 (socket says 172.21.15.2:33056) 2023-11-16T16:48:53.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.717+0000 7f08d8b9a700 1 -- 172.21.15.2:0/1337328862 learned_addr learned my addr 172.21.15.2:0/1337328862 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:53.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.717+0000 7f08d939b700 1 --2- 172.21.15.2:0/1337328862 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08d41002c0 0x7f08d40748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:53.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.717+0000 7f08d9b9c700 1 --2- 172.21.15.2:0/1337328862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08d4101e40 0x7f08d4073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:53.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.717+0000 7f08d8b9a700 1 -- 172.21.15.2:0/1337328862 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08d41002c0 msgr2=0x7f08d40748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:53.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.717+0000 7f08d8b9a700 1 --2- 172.21.15.2:0/1337328862 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08d41002c0 0x7f08d40748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.717+0000 7f08d8b9a700 1 -- 172.21.15.2:0/1337328862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08d4101e40 msgr2=0x7f08d4073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:53.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.717+0000 7f08d8b9a700 1 --2- 172.21.15.2:0/1337328862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08d4101e40 0x7f08d4073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.717+0000 7f08d8b9a700 1 -- 172.21.15.2:0/1337328862 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d0009520 con 0x7f08d41014b0 2023-11-16T16:48:53.721 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.717+0000 7f08d939b700 1 --2- 172.21.15.2:0/1337328862 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08d41002c0 0x7f08d40748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:53.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.718+0000 7f08d8b9a700 1 --2- 172.21.15.2:0/1337328862 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d41014b0 0x7f08d4072ee0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f08c400e9a0 tx=0x7f08c400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:53.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.718+0000 7f08ca7fc700 1 -- 172.21.15.2:0/1337328862 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c4009790 con 0x7f08d41014b0 2023-11-16T16:48:53.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.718+0000 7f08ca7fc700 1 -- 172.21.15.2:0/1337328862 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08c4010430 con 0x7f08d41014b0 2023-11-16T16:48:53.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.718+0000 7f08db5ff700 1 -- 172.21.15.2:0/1337328862 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d410bc10 con 0x7f08d41014b0 2023-11-16T16:48:53.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.718+0000 7f08ca7fc700 1 -- 172.21.15.2:0/1337328862 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c4018680 con 0x7f08d41014b0 2023-11-16T16:48:53.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.718+0000 7f08db5ff700 1 -- 172.21.15.2:0/1337328862 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d410ba80 con 0x7f08d41014b0 2023-11-16T16:48:53.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.719+0000 7f08db5ff700 1 -- 172.21.15.2:0/1337328862 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d4107920 con 0x7f08d41014b0 2023-11-16T16:48:53.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.721+0000 7f08ca7fc700 1 -- 172.21.15.2:0/1337328862 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f08c40105a0 con 0x7f08d41014b0 2023-11-16T16:48:53.724 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.723+0000 7f08ca7fc700 1 --2- 172.21.15.2:0/1337328862 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08c00a11e0 0x7f08c00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:53.724 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.723+0000 7f08ca7fc700 1 -- 172.21.15.2:0/1337328862 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f08c4014070 con 0x7f08d41014b0 2023-11-16T16:48:53.724 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.723+0000 7f08d939b700 1 --2- 172.21.15.2:0/1337328862 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08c00a11e0 0x7f08c00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:53.725 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.723+0000 7f08d939b700 1 --2- 172.21.15.2:0/1337328862 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08c00a11e0 0x7f08c00a3690 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f08d0009fd0 tx=0x7f08d0019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:53.726 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.725+0000 7f08ca7fc700 1 -- 172.21.15.2:0/1337328862 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f08c4096440 con 0x7f08d41014b0 2023-11-16T16:48:53.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.938+0000 7f08db5ff700 1 -- 172.21.15.2:0/1337328862 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f08d4061700 con 0x7f08d41014b0 2023-11-16T16:48:53.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.939+0000 7f08ca7fc700 1 -- 172.21.15.2:0/1337328862 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f08c40260a0 con 0x7f08d41014b0 2023-11-16T16:48:53.941 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2023-11-16T16:48:53.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.941+0000 7f08db5ff700 1 -- 172.21.15.2:0/1337328862 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08c00a11e0 msgr2=0x7f08c00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:53.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.941+0000 7f08db5ff700 1 --2- 172.21.15.2:0/1337328862 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08c00a11e0 0x7f08c00a3690 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f08d0009fd0 tx=0x7f08d0019040 comp rx=0 tx=0).stop 2023-11-16T16:48:53.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.941+0000 7f08db5ff700 1 -- 172.21.15.2:0/1337328862 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d41014b0 msgr2=0x7f08d4072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:53.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.941+0000 7f08db5ff700 1 --2- 172.21.15.2:0/1337328862 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d41014b0 0x7f08d4072ee0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f08c400e9a0 tx=0x7f08c400ecb0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.941+0000 7f08db5ff700 1 -- 172.21.15.2:0/1337328862 shutdown_connections 2023-11-16T16:48:53.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.941+0000 7f08db5ff700 1 --2- 172.21.15.2:0/1337328862 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f08c00a11e0 0x7f08c00a3690 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.941+0000 7f08db5ff700 1 --2- 172.21.15.2:0/1337328862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f08d4101e40 0x7f08d4073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.941+0000 7f08db5ff700 1 --2- 172.21.15.2:0/1337328862 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f08d41002c0 0x7f08d40748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.941+0000 7f08db5ff700 1 --2- 172.21.15.2:0/1337328862 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d41014b0 0x7f08d4072ee0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:53.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.941+0000 7f08db5ff700 1 -- 172.21.15.2:0/1337328862 >> 172.21.15.2:0/1337328862 conn(0x7f08d40fb8a0 msgr2=0x7f08d4104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:53.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.942+0000 7f08db5ff700 1 -- 172.21.15.2:0/1337328862 shutdown_connections 2023-11-16T16:48:53.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:53.942+0000 7f08db5ff700 1 -- 172.21.15.2:0/1337328862 wait complete. 2023-11-16T16:48:53.963 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T16:48:54.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:53 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3034522638' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:48:54.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:53 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3787203225' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:48:54.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.168+0000 7f98df1ff700 1 -- 172.21.15.2:0/2523808499 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98d8073550 msgr2=0x7f98d80739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:54.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.168+0000 7f98df1ff700 1 --2- 172.21.15.2:0/2523808499 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98d8073550 0x7f98d80739c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f98c4009990 tx=0x7f98c4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.168+0000 7f98df1ff700 1 -- 172.21.15.2:0/2523808499 shutdown_connections 2023-11-16T16:48:54.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.168+0000 7f98df1ff700 1 --2- 172.21.15.2:0/2523808499 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98d8074b30 0x7f98d8072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.168+0000 7f98df1ff700 1 --2- 172.21.15.2:0/2523808499 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98d810c890 0x7f98d810ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.170 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.168+0000 7f98df1ff700 1 --2- 172.21.15.2:0/2523808499 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98d8073550 0x7f98d80739c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.168+0000 7f98df1ff700 1 -- 172.21.15.2:0/2523808499 >> 172.21.15.2:0/2523808499 conn(0x7f98d80fb8a0 msgr2=0x7f98d80fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:54.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.168+0000 7f98df1ff700 1 -- 172.21.15.2:0/2523808499 shutdown_connections 2023-11-16T16:48:54.171 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.168+0000 7f98df1ff700 1 -- 172.21.15.2:0/2523808499 wait complete. 2023-11-16T16:48:54.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.169+0000 7f98df1ff700 1 Processor -- start 2023-11-16T16:48:54.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.169+0000 7f98df1ff700 1 -- start start 2023-11-16T16:48:54.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.170+0000 7f98df1ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98d8073550 0x7f98d819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:54.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.170+0000 7f98df1ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98d8074b30 0x7f98d819cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:54.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.170+0000 7f98df1ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98d810c890 0x7f98d81a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:54.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.170+0000 7f98df1ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d81a4220 con 0x7f98d8074b30 2023-11-16T16:48:54.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.170+0000 7f98df1ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d81a4360 con 0x7f98d810c890 2023-11-16T16:48:54.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.170+0000 7f98df1ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d81a44f0 con 0x7f98d8073550 2023-11-16T16:48:54.173 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.170+0000 7f98d7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98d8074b30 0x7f98d819cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:54.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.170+0000 7f98d7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98d8074b30 0x7f98d819cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53456/0 (socket says 172.21.15.2:53456) 2023-11-16T16:48:54.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.170+0000 7f98d7fff700 1 -- 172.21.15.2:0/1527673404 learned_addr learned my addr 172.21.15.2:0/1527673404 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:54.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.170+0000 7f98dcf9b700 1 --2- 172.21.15.2:0/1527673404 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98d8073550 0x7f98d819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:54.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.170+0000 7f98dd79c700 1 --2- 172.21.15.2:0/1527673404 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98d810c890 0x7f98d81a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:54.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.170+0000 7f98d7fff700 1 -- 172.21.15.2:0/1527673404 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98d8073550 msgr2=0x7f98d819c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:54.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.170+0000 7f98d7fff700 1 --2- 172.21.15.2:0/1527673404 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98d8073550 0x7f98d819c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.170+0000 7f98d7fff700 1 -- 172.21.15.2:0/1527673404 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98d810c890 msgr2=0x7f98d81a3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:54.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.170+0000 7f98d7fff700 1 --2- 172.21.15.2:0/1527673404 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98d810c890 0x7f98d81a3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.174 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.170+0000 7f98d7fff700 1 -- 172.21.15.2:0/1527673404 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c40095f0 con 0x7f98d8074b30 2023-11-16T16:48:54.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.170+0000 7f98dd79c700 1 --2- 172.21.15.2:0/1527673404 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98d810c890 0x7f98d81a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:48:54.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.170+0000 7f98d7fff700 1 --2- 172.21.15.2:0/1527673404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98d8074b30 0x7f98d819cbe0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f98c40098e0 tx=0x7f98c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:54.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.171+0000 7f98d5ffb700 1 -- 172.21.15.2:0/1527673404 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c401d070 con 0x7f98d8074b30 2023-11-16T16:48:54.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.171+0000 7f98df1ff700 1 -- 172.21.15.2:0/1527673404 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d81122a0 con 0x7f98d8074b30 2023-11-16T16:48:54.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.171+0000 7f98d5ffb700 1 -- 172.21.15.2:0/1527673404 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98c40054f0 con 0x7f98d8074b30 2023-11-16T16:48:54.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.171+0000 7f98d5ffb700 1 -- 172.21.15.2:0/1527673404 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4004ca0 con 0x7f98d8074b30 2023-11-16T16:48:54.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.171+0000 7f98df1ff700 1 -- 172.21.15.2:0/1527673404 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d8112110 con 0x7f98d8074b30 2023-11-16T16:48:54.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.172+0000 7f98df1ff700 1 -- 172.21.15.2:0/1527673404 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d81023e0 con 0x7f98d8074b30 2023-11-16T16:48:54.176 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.174+0000 7f98d5ffb700 1 -- 172.21.15.2:0/1527673404 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f98c400fc70 con 0x7f98d8074b30 2023-11-16T16:48:54.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.176+0000 7f98d5ffb700 1 --2- 172.21.15.2:0/1527673404 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f98c80a11e0 0x7f98c80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:54.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.176+0000 7f98d5ffb700 1 -- 172.21.15.2:0/1527673404 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f98c40ce140 con 0x7f98d8074b30 2023-11-16T16:48:54.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.176+0000 7f98dcf9b700 1 --2- 172.21.15.2:0/1527673404 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f98c80a11e0 0x7f98c80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:54.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.177+0000 7f98dcf9b700 1 --2- 172.21.15.2:0/1527673404 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f98c80a11e0 0x7f98c80a3690 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f98d8074f00 tx=0x7f98cc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:54.181 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.180+0000 7f98d5ffb700 1 -- 172.21.15.2:0/1527673404 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f98c405a730 con 0x7f98d8074b30 2023-11-16T16:48:54.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:53 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3034522638' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:48:54.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:53 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3787203225' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:48:54.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:53 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3034522638' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:48:54.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:53 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3787203225' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:48:54.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.394+0000 7f98df1ff700 1 -- 172.21.15.2:0/1527673404 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f98d8067b00 con 0x7f98d8074b30 2023-11-16T16:48:54.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.395+0000 7f98d5ffb700 1 -- 172.21.15.2:0/1527673404 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f98c4097440 con 0x7f98d8074b30 2023-11-16T16:48:54.397 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T16:48:54.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.397+0000 7f98df1ff700 1 -- 172.21.15.2:0/1527673404 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f98c80a11e0 msgr2=0x7f98c80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:54.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.397+0000 7f98df1ff700 1 --2- 172.21.15.2:0/1527673404 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f98c80a11e0 0x7f98c80a3690 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f98d8074f00 tx=0x7f98cc009260 comp rx=0 tx=0).stop 2023-11-16T16:48:54.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.397+0000 7f98df1ff700 1 -- 172.21.15.2:0/1527673404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98d8074b30 msgr2=0x7f98d819cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:54.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.397+0000 7f98df1ff700 1 --2- 172.21.15.2:0/1527673404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98d8074b30 0x7f98d819cbe0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f98c40098e0 tx=0x7f98c4004730 comp rx=0 tx=0).stop 2023-11-16T16:48:54.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.398+0000 7f98df1ff700 1 -- 172.21.15.2:0/1527673404 shutdown_connections 2023-11-16T16:48:54.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.398+0000 7f98df1ff700 1 --2- 172.21.15.2:0/1527673404 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f98c80a11e0 0x7f98c80a3690 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.398+0000 7f98df1ff700 1 --2- 172.21.15.2:0/1527673404 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98d8073550 0x7f98d819c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.398+0000 7f98df1ff700 1 --2- 172.21.15.2:0/1527673404 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f98d810c890 0x7f98d81a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.398+0000 7f98df1ff700 1 --2- 172.21.15.2:0/1527673404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98d8074b30 0x7f98d819cbe0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.398+0000 7f98df1ff700 1 -- 172.21.15.2:0/1527673404 >> 172.21.15.2:0/1527673404 conn(0x7f98d80fb8a0 msgr2=0x7f98d80fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:54.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.398+0000 7f98df1ff700 1 -- 172.21.15.2:0/1527673404 shutdown_connections 2023-11-16T16:48:54.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.398+0000 7f98df1ff700 1 -- 172.21.15.2:0/1527673404 wait complete. 2023-11-16T16:48:54.419 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:48:54.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.622+0000 7fc7091ff700 1 -- 172.21.15.2:0/168323058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7040ff1a0 msgr2=0x7fc7040ff5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:54.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.622+0000 7fc7091ff700 1 --2- 172.21.15.2:0/168323058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7040ff1a0 0x7fc7040ff5b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4009990 tx=0x7fc6f4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.622+0000 7fc7091ff700 1 -- 172.21.15.2:0/168323058 shutdown_connections 2023-11-16T16:48:54.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.622+0000 7fc7091ff700 1 --2- 172.21.15.2:0/168323058 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc704100d20 0x7fc7041075e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.622+0000 7fc7091ff700 1 --2- 172.21.15.2:0/168323058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7040ff1a0 0x7fc7040ff5b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.622+0000 7fc7091ff700 1 --2- 172.21.15.2:0/168323058 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc704100390 0x7fc7041007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.622+0000 7fc7091ff700 1 -- 172.21.15.2:0/168323058 >> 172.21.15.2:0/168323058 conn(0x7fc7040fa7a0 msgr2=0x7fc7040fcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:54.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.622+0000 7fc7091ff700 1 -- 172.21.15.2:0/168323058 shutdown_connections 2023-11-16T16:48:54.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.622+0000 7fc7091ff700 1 -- 172.21.15.2:0/168323058 wait complete. 2023-11-16T16:48:54.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.624+0000 7fc7091ff700 1 Processor -- start 2023-11-16T16:48:54.626 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.624+0000 7fc7091ff700 1 -- start start 2023-11-16T16:48:54.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.624+0000 7fc7091ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7040ff1a0 0x7fc704103130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:54.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.624+0000 7fc7091ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc704100390 0x7fc704101770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:54.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.624+0000 7fc7091ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc704100d20 0x7fc704101cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:54.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.624+0000 7fc7091ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc704102450 con 0x7fc704100390 2023-11-16T16:48:54.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.624+0000 7fc7091ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7041025c0 con 0x7fc7040ff1a0 2023-11-16T16:48:54.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.624+0000 7fc7091ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7041a0110 con 0x7fc704100d20 2023-11-16T16:48:54.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.625+0000 7fc702d9d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7040ff1a0 0x7fc704103130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:54.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.625+0000 7fc702d9d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7040ff1a0 0x7fc704103130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:35684/0 (socket says 172.21.15.2:35684) 2023-11-16T16:48:54.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.625+0000 7fc702d9d700 1 -- 172.21.15.2:0/1540471505 learned_addr learned my addr 172.21.15.2:0/1540471505 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:54.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.625+0000 7fc6fbfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc704100390 0x7fc704101770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:54.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.625+0000 7fc70359e700 1 --2- 172.21.15.2:0/1540471505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc704100d20 0x7fc704101cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:54.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.625+0000 7fc702d9d700 1 -- 172.21.15.2:0/1540471505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc704100d20 msgr2=0x7fc704101cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:54.628 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.625+0000 7fc702d9d700 1 --2- 172.21.15.2:0/1540471505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc704100d20 0x7fc704101cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.625+0000 7fc702d9d700 1 -- 172.21.15.2:0/1540471505 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc704100390 msgr2=0x7fc704101770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:54.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.625+0000 7fc702d9d700 1 --2- 172.21.15.2:0/1540471505 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc704100390 0x7fc704101770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.625+0000 7fc702d9d700 1 -- 172.21.15.2:0/1540471505 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f40095f0 con 0x7fc7040ff1a0 2023-11-16T16:48:54.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.625+0000 7fc6fbfff700 1 --2- 172.21.15.2:0/1540471505 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc704100390 0x7fc704101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:54.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.626+0000 7fc702d9d700 1 --2- 172.21.15.2:0/1540471505 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7040ff1a0 0x7fc704103130 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4000c00 tx=0x7fc6f4005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:54.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.626+0000 7fc700d99700 1 -- 172.21.15.2:0/1540471505 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f401d070 con 0x7fc7040ff1a0 2023-11-16T16:48:54.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.626+0000 7fc700d99700 1 -- 172.21.15.2:0/1540471505 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6f4021410 con 0x7fc7040ff1a0 2023-11-16T16:48:54.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.626+0000 7fc7091ff700 1 -- 172.21.15.2:0/1540471505 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc70410ab60 con 0x7fc7040ff1a0 2023-11-16T16:48:54.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.626+0000 7fc700d99700 1 -- 172.21.15.2:0/1540471505 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4017650 con 0x7fc7040ff1a0 2023-11-16T16:48:54.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.626+0000 7fc7091ff700 1 -- 172.21.15.2:0/1540471505 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc70410a9d0 con 0x7fc7040ff1a0 2023-11-16T16:48:54.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.627+0000 7fc7091ff700 1 -- 172.21.15.2:0/1540471505 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7041a06b0 con 0x7fc7040ff1a0 2023-11-16T16:48:54.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.629+0000 7fc700d99700 1 -- 172.21.15.2:0/1540471505 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fc6f4021580 con 0x7fc7040ff1a0 2023-11-16T16:48:54.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.630+0000 7fc700d99700 1 --2- 172.21.15.2:0/1540471505 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc6f00a11e0 0x7fc6f00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:54.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.630+0000 7fc700d99700 1 -- 172.21.15.2:0/1540471505 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fc6f40ce0a0 con 0x7fc7040ff1a0 2023-11-16T16:48:54.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.630+0000 7fc6fbfff700 1 --2- 172.21.15.2:0/1540471505 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc6f00a11e0 0x7fc6f00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:54.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.631+0000 7fc6fbfff700 1 --2- 172.21.15.2:0/1540471505 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc6f00a11e0 0x7fc6f00a3690 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec011490 tx=0x7fc6ec009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:54.636 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.635+0000 7fc700d99700 1 -- 172.21.15.2:0/1540471505 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc6f4097360 con 0x7fc7040ff1a0 2023-11-16T16:48:54.894 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.892+0000 7fc7091ff700 1 -- 172.21.15.2:0/1540471505 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7041a0c60 con 0x7fc7040ff1a0 2023-11-16T16:48:54.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.894+0000 7fc700d99700 1 -- 172.21.15.2:0/1540471505 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10811 (secure 0 0 0) 0x7fc6f409a980 con 0x7fc7040ff1a0 2023-11-16T16:48:54.895 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:48:54.896 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/1610211225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":1610211225},{"type":"v1","addr":"172.21.15.2:6837","nonce":1610211225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14538,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2583261818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2583261818},{"type":"v1","addr":"172.21.15.2:6839","nonce":2583261818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14550,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3170991923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3170991923},{"type":"v1","addr":"172.21.15.2:6841","nonce":3170991923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:26.349176+0000","modified":"2023-11-16T16:48:05.774072+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24544,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.141:6835/3516869188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3516869188},{"type":"v1","addr":"172.21.15.141:6835","nonce":3516869188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/2527313191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":2527313191},{"type":"v1","addr":"172.21.15.155:6835","nonce":2527313191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24523":{"gid":24523,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1607756751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1607756751},{"type":"v1","addr":"172.21.15.155:6837","nonce":1607756751}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.141:6837/2934891475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6836","nonce":2934891475},{"type":"v1","addr":"172.21.15.141:6837","nonce":2934891475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24544":{"gid":24544,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1158457009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1158457009},{"type":"v1","addr":"172.21.15.155:6839","nonce":1158457009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6839/3916358272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6838","nonce":3916358272},{"type":"v1","addr":"172.21.15.141:6839","nonce":3916358272}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:48:54.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.896+0000 7fc7091ff700 1 -- 172.21.15.2:0/1540471505 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc6f00a11e0 msgr2=0x7fc6f00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:54.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.896+0000 7fc7091ff700 1 --2- 172.21.15.2:0/1540471505 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc6f00a11e0 0x7fc6f00a3690 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec011490 tx=0x7fc6ec009260 comp rx=0 tx=0).stop 2023-11-16T16:48:54.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.896+0000 7fc7091ff700 1 -- 172.21.15.2:0/1540471505 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7040ff1a0 msgr2=0x7fc704103130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:54.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.896+0000 7fc7091ff700 1 --2- 172.21.15.2:0/1540471505 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7040ff1a0 0x7fc704103130 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4000c00 tx=0x7fc6f4005740 comp rx=0 tx=0).stop 2023-11-16T16:48:54.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.896+0000 7fc7091ff700 1 -- 172.21.15.2:0/1540471505 shutdown_connections 2023-11-16T16:48:54.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.896+0000 7fc7091ff700 1 --2- 172.21.15.2:0/1540471505 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc6f00a11e0 0x7fc6f00a3690 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.896+0000 7fc7091ff700 1 --2- 172.21.15.2:0/1540471505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc704100d20 0x7fc704101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.896+0000 7fc7091ff700 1 --2- 172.21.15.2:0/1540471505 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc7040ff1a0 0x7fc704103130 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.896+0000 7fc7091ff700 1 --2- 172.21.15.2:0/1540471505 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc704100390 0x7fc704101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:54.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.896+0000 7fc7091ff700 1 -- 172.21.15.2:0/1540471505 >> 172.21.15.2:0/1540471505 conn(0x7fc7040fa7a0 msgr2=0x7fc704069070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:54.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.896+0000 7fc7091ff700 1 -- 172.21.15.2:0/1540471505 shutdown_connections 2023-11-16T16:48:54.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:54.896+0000 7fc7091ff700 1 -- 172.21.15.2:0/1540471505 wait complete. 2023-11-16T16:48:54.900 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 26 2023-11-16T16:48:54.918 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:48:55.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.120+0000 7fb234fff700 1 -- 172.21.15.2:0/218066034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb230101e10 msgr2=0x7fb2301086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:55.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.120+0000 7fb234fff700 1 --2- 172.21.15.2:0/218066034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb230101e10 0x7fb2301086d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb22800b0a0 tx=0x7fb22800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.120+0000 7fb234fff700 1 -- 172.21.15.2:0/218066034 shutdown_connections 2023-11-16T16:48:55.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.120+0000 7fb234fff700 1 --2- 172.21.15.2:0/218066034 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb230100290 0x7fb2301006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.120+0000 7fb234fff700 1 --2- 172.21.15.2:0/218066034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb230101e10 0x7fb2301086d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.120+0000 7fb234fff700 1 --2- 172.21.15.2:0/218066034 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb230101480 0x7fb2301018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.120+0000 7fb234fff700 1 -- 172.21.15.2:0/218066034 >> 172.21.15.2:0/218066034 conn(0x7fb2300fb870 msgr2=0x7fb2300fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:55.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.120+0000 7fb234fff700 1 -- 172.21.15.2:0/218066034 shutdown_connections 2023-11-16T16:48:55.123 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.120+0000 7fb234fff700 1 -- 172.21.15.2:0/218066034 wait complete. 2023-11-16T16:48:55.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.121+0000 7fb234fff700 1 Processor -- start 2023-11-16T16:48:55.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.122+0000 7fb234fff700 1 -- start start 2023-11-16T16:48:55.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.122+0000 7fb234fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb230100290 0x7fb230196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:55.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.122+0000 7fb234fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb230101480 0x7fb2301965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:55.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.122+0000 7fb234fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb230101e10 0x7fb23019d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:55.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.122+0000 7fb234fff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23019dc10 con 0x7fb230101e10 2023-11-16T16:48:55.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.122+0000 7fb234fff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23019dd50 con 0x7fb230100290 2023-11-16T16:48:55.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.122+0000 7fb234fff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23019dee0 con 0x7fb230101480 2023-11-16T16:48:55.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.122+0000 7fb22dd9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb230101480 0x7fb2301965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:55.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.122+0000 7fb22ed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb230101e10 0x7fb23019d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:55.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.122+0000 7fb22dd9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb230101480 0x7fb2301965d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:58538/0 (socket says 172.21.15.2:58538) 2023-11-16T16:48:55.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.122+0000 7fb22dd9b700 1 -- 172.21.15.2:0/2795784398 learned_addr learned my addr 172.21.15.2:0/2795784398 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:55.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.122+0000 7fb22e59c700 1 --2- 172.21.15.2:0/2795784398 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb230100290 0x7fb230196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:55.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.123+0000 7fb22dd9b700 1 -- 172.21.15.2:0/2795784398 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb230100290 msgr2=0x7fb230196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:55.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.123+0000 7fb22dd9b700 1 --2- 172.21.15.2:0/2795784398 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb230100290 0x7fb230196090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.123+0000 7fb22dd9b700 1 -- 172.21.15.2:0/2795784398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb230101e10 msgr2=0x7fb23019d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:55.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.123+0000 7fb22dd9b700 1 --2- 172.21.15.2:0/2795784398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb230101e10 0x7fb23019d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.123+0000 7fb22dd9b700 1 -- 172.21.15.2:0/2795784398 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb228009d00 con 0x7fb230101480 2023-11-16T16:48:55.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.123+0000 7fb22ed9d700 1 --2- 172.21.15.2:0/2795784398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb230101e10 0x7fb23019d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:55.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.123+0000 7fb22dd9b700 1 --2- 172.21.15.2:0/2795784398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb230101480 0x7fb2301965d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb22400e9a0 tx=0x7fb22400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:55.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.123+0000 7fb21f7fe700 1 -- 172.21.15.2:0/2795784398 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb224009790 con 0x7fb230101480 2023-11-16T16:48:55.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.123+0000 7fb234fff700 1 -- 172.21.15.2:0/2795784398 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb23010bc70 con 0x7fb230101480 2023-11-16T16:48:55.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.123+0000 7fb234fff700 1 -- 172.21.15.2:0/2795784398 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb23010bae0 con 0x7fb230101480 2023-11-16T16:48:55.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.124+0000 7fb21f7fe700 1 -- 172.21.15.2:0/2795784398 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb224010430 con 0x7fb230101480 2023-11-16T16:48:55.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.124+0000 7fb21f7fe700 1 -- 172.21.15.2:0/2795784398 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb224018680 con 0x7fb230101480 2023-11-16T16:48:55.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.124+0000 7fb234fff700 1 -- 172.21.15.2:0/2795784398 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb210005310 con 0x7fb230101480 2023-11-16T16:48:55.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.127+0000 7fb21f7fe700 1 -- 172.21.15.2:0/2795784398 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fb22400ca10 con 0x7fb230101480 2023-11-16T16:48:55.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.128+0000 7fb21f7fe700 1 --2- 172.21.15.2:0/2795784398 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb2180a11e0 0x7fb2180a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:55.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.128+0000 7fb21f7fe700 1 -- 172.21.15.2:0/2795784398 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fb224014070 con 0x7fb230101480 2023-11-16T16:48:55.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.128+0000 7fb22e59c700 1 --2- 172.21.15.2:0/2795784398 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb2180a11e0 0x7fb2180a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:55.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.129+0000 7fb22e59c700 1 --2- 172.21.15.2:0/2795784398 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb2180a11e0 0x7fb2180a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb2301012e0 tx=0x7fb220009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:55.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.130+0000 7fb21f7fe700 1 -- 172.21.15.2:0/2795784398 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb224051730 con 0x7fb230101480 2023-11-16T16:48:55.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:54 smithi002 ceph-mon[128712]: pgmap v328: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s; 7.0 KiB/s, 1 keys/s, 1 objects/s recovering 2023-11-16T16:48:55.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:54 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1337328862' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:48:55.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:54 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1527673404' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:48:55.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:54 smithi141 ceph-mon[131184]: pgmap v328: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s; 7.0 KiB/s, 1 keys/s, 1 objects/s recovering 2023-11-16T16:48:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:54 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1337328862' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:48:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:54 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1527673404' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:48:55.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.342+0000 7fb234fff700 1 -- 172.21.15.2:0/2795784398 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb210005180 con 0x7fb230101480 2023-11-16T16:48:55.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.345+0000 7fb21f7fe700 1 -- 172.21.15.2:0/2795784398 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24567 (secure 0 0 0) 0x7fb224099ae0 con 0x7fb230101480 2023-11-16T16:48:55.347 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:48:55.347 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":93,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:48:45.491739+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:24.458452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:25.786005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:29.313181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:48:55.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.347+0000 7fb234fff700 1 -- 172.21.15.2:0/2795784398 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb2180a11e0 msgr2=0x7fb2180a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:55.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.347+0000 7fb234fff700 1 --2- 172.21.15.2:0/2795784398 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb2180a11e0 0x7fb2180a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb2301012e0 tx=0x7fb220009260 comp rx=0 tx=0).stop 2023-11-16T16:48:55.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.347+0000 7fb234fff700 1 -- 172.21.15.2:0/2795784398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb230101480 msgr2=0x7fb2301965d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:55.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.347+0000 7fb234fff700 1 --2- 172.21.15.2:0/2795784398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb230101480 0x7fb2301965d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb22400e9a0 tx=0x7fb22400ecb0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.347+0000 7fb234fff700 1 -- 172.21.15.2:0/2795784398 shutdown_connections 2023-11-16T16:48:55.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.347+0000 7fb234fff700 1 --2- 172.21.15.2:0/2795784398 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb2180a11e0 0x7fb2180a3690 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.347+0000 7fb234fff700 1 --2- 172.21.15.2:0/2795784398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb230101e10 0x7fb23019d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.347+0000 7fb234fff700 1 --2- 172.21.15.2:0/2795784398 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb230100290 0x7fb230196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.347+0000 7fb234fff700 1 --2- 172.21.15.2:0/2795784398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb230101480 0x7fb2301965d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.347+0000 7fb234fff700 1 -- 172.21.15.2:0/2795784398 >> 172.21.15.2:0/2795784398 conn(0x7fb2300fb870 msgr2=0x7fb2300fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:55.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.347+0000 7fb234fff700 1 -- 172.21.15.2:0/2795784398 shutdown_connections 2023-11-16T16:48:55.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.347+0000 7fb234fff700 1 -- 172.21.15.2:0/2795784398 wait complete. 2023-11-16T16:48:55.368 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:04.306584+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T16:47:24.458452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T16:47:25.786005+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T16:47:29.313181+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T16:48:55.368 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:48:55.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:54 smithi155 ceph-mon[131607]: pgmap v328: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s; 7.0 KiB/s, 1 keys/s, 1 objects/s recovering 2023-11-16T16:48:55.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:54 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1337328862' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:48:55.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:54 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1527673404' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:48:55.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.574+0000 7fca1e7ff700 1 -- 172.21.15.2:0/3616503147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca181002c0 msgr2=0x7fca181006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:55.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.574+0000 7fca1e7ff700 1 --2- 172.21.15.2:0/3616503147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca181002c0 0x7fca181006d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fca08009990 tx=0x7fca08009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.574+0000 7fca1e7ff700 1 -- 172.21.15.2:0/3616503147 shutdown_connections 2023-11-16T16:48:55.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.574+0000 7fca1e7ff700 1 --2- 172.21.15.2:0/3616503147 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca18101e40 0x7fca18108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.576 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.574+0000 7fca1e7ff700 1 --2- 172.21.15.2:0/3616503147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca181002c0 0x7fca181006d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.574+0000 7fca1e7ff700 1 --2- 172.21.15.2:0/3616503147 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca181014b0 0x7fca18101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.574+0000 7fca1e7ff700 1 -- 172.21.15.2:0/3616503147 >> 172.21.15.2:0/3616503147 conn(0x7fca180fb8a0 msgr2=0x7fca180fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:55.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.574+0000 7fca1e7ff700 1 -- 172.21.15.2:0/3616503147 shutdown_connections 2023-11-16T16:48:55.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.574+0000 7fca1e7ff700 1 -- 172.21.15.2:0/3616503147 wait complete. 2023-11-16T16:48:55.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.575+0000 7fca1e7ff700 1 Processor -- start 2023-11-16T16:48:55.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.576+0000 7fca1e7ff700 1 -- start start 2023-11-16T16:48:55.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.576+0000 7fca1e7ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca181002c0 0x7fca18198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:55.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.576+0000 7fca1e7ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca181014b0 0x7fca181987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:55.579 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.576+0000 7fca1e7ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca18101e40 0x7fca1819f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:55.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.576+0000 7fca1e7ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1819fe00 con 0x7fca181002c0 2023-11-16T16:48:55.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.576+0000 7fca1e7ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1819ff40 con 0x7fca181014b0 2023-11-16T16:48:55.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.576+0000 7fca1e7ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca181a00d0 con 0x7fca18101e40 2023-11-16T16:48:55.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.576+0000 7fca17fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca181002c0 0x7fca18198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:55.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.576+0000 7fca17fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca181002c0 0x7fca18198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44194/0 (socket says 172.21.15.2:44194) 2023-11-16T16:48:55.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.576+0000 7fca17fff700 1 -- 172.21.15.2:0/1258136219 learned_addr learned my addr 172.21.15.2:0/1258136219 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:55.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.576+0000 7fca1cd9c700 1 --2- 172.21.15.2:0/1258136219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca18101e40 0x7fca1819f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:55.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.576+0000 7fca177fe700 1 --2- 172.21.15.2:0/1258136219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca181014b0 0x7fca181987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:55.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.577+0000 7fca17fff700 1 -- 172.21.15.2:0/1258136219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca18101e40 msgr2=0x7fca1819f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:55.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.577+0000 7fca17fff700 1 --2- 172.21.15.2:0/1258136219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca18101e40 0x7fca1819f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.577+0000 7fca17fff700 1 -- 172.21.15.2:0/1258136219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca181014b0 msgr2=0x7fca181987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:55.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.577+0000 7fca17fff700 1 --2- 172.21.15.2:0/1258136219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca181014b0 0x7fca181987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.580 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.577+0000 7fca17fff700 1 -- 172.21.15.2:0/1258136219 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c009520 con 0x7fca181002c0 2023-11-16T16:48:55.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.577+0000 7fca17fff700 1 --2- 172.21.15.2:0/1258136219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca181002c0 0x7fca18198280 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fca08004730 tx=0x7fca08004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:55.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.577+0000 7fca157fa700 1 -- 172.21.15.2:0/1258136219 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0801d070 con 0x7fca181002c0 2023-11-16T16:48:55.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.577+0000 7fca157fa700 1 -- 172.21.15.2:0/1258136219 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca08021410 con 0x7fca181002c0 2023-11-16T16:48:55.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.577+0000 7fca1e7ff700 1 -- 172.21.15.2:0/1258136219 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca080095f0 con 0x7fca181002c0 2023-11-16T16:48:55.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.577+0000 7fca157fa700 1 -- 172.21.15.2:0/1258136219 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca08017650 con 0x7fca181002c0 2023-11-16T16:48:55.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.577+0000 7fca1e7ff700 1 -- 172.21.15.2:0/1258136219 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca1810bc90 con 0x7fca181002c0 2023-11-16T16:48:55.581 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.578+0000 7fca1e7ff700 1 -- 172.21.15.2:0/1258136219 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca181929f0 con 0x7fca181002c0 2023-11-16T16:48:55.582 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.580+0000 7fca157fa700 1 -- 172.21.15.2:0/1258136219 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fca08021580 con 0x7fca181002c0 2023-11-16T16:48:55.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.582+0000 7fca157fa700 1 --2- 172.21.15.2:0/1258136219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fca000a1260 0x7fca000a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:55.583 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.582+0000 7fca157fa700 1 -- 172.21.15.2:0/1258136219 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fca080ce1c0 con 0x7fca181002c0 2023-11-16T16:48:55.584 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.582+0000 7fca177fe700 1 --2- 172.21.15.2:0/1258136219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fca000a1260 0x7fca000a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:55.584 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.583+0000 7fca177fe700 1 --2- 172.21.15.2:0/1258136219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fca000a1260 0x7fca000a3710 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009df0 tx=0x7fca0c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:55.587 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.586+0000 7fca157fa700 1 -- 172.21.15.2:0/1258136219 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fca08097480 con 0x7fca181002c0 2023-11-16T16:48:55.794 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.793+0000 7fca1e7ff700 1 -- 172.21.15.2:0/1258136219 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fca18067b00 con 0x7fca181002c0 2023-11-16T16:48:55.795 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.793+0000 7fca157fa700 1 -- 172.21.15.2:0/1258136219 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fca0809aaa0 con 0x7fca181002c0 2023-11-16T16:48:55.795 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T16:48:55.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.796+0000 7fca1e7ff700 1 -- 172.21.15.2:0/1258136219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fca000a1260 msgr2=0x7fca000a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:55.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.796+0000 7fca1e7ff700 1 --2- 172.21.15.2:0/1258136219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fca000a1260 0x7fca000a3710 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009df0 tx=0x7fca0c009260 comp rx=0 tx=0).stop 2023-11-16T16:48:55.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.796+0000 7fca1e7ff700 1 -- 172.21.15.2:0/1258136219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca181002c0 msgr2=0x7fca18198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:55.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.796+0000 7fca1e7ff700 1 --2- 172.21.15.2:0/1258136219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca181002c0 0x7fca18198280 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fca08004730 tx=0x7fca08004760 comp rx=0 tx=0).stop 2023-11-16T16:48:55.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.796+0000 7fca1e7ff700 1 -- 172.21.15.2:0/1258136219 shutdown_connections 2023-11-16T16:48:55.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.796+0000 7fca1e7ff700 1 --2- 172.21.15.2:0/1258136219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fca000a1260 0x7fca000a3710 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.796+0000 7fca1e7ff700 1 --2- 172.21.15.2:0/1258136219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca18101e40 0x7fca1819f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.796+0000 7fca1e7ff700 1 --2- 172.21.15.2:0/1258136219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca181002c0 0x7fca18198280 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.796+0000 7fca1e7ff700 1 --2- 172.21.15.2:0/1258136219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca181014b0 0x7fca181987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:55.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.796+0000 7fca1e7ff700 1 -- 172.21.15.2:0/1258136219 >> 172.21.15.2:0/1258136219 conn(0x7fca180fb8a0 msgr2=0x7fca18106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:55.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.796+0000 7fca1e7ff700 1 -- 172.21.15.2:0/1258136219 shutdown_connections 2023-11-16T16:48:55.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:55.796+0000 7fca1e7ff700 1 -- 172.21.15.2:0/1258136219 wait complete. 2023-11-16T16:48:55.817 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T16:48:56.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.023+0000 7f5b685ff700 1 -- 172.21.15.2:0/360042559 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b6010c890 msgr2=0x7f5b6010ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:56.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.023+0000 7f5b685ff700 1 --2- 172.21.15.2:0/360042559 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b6010c890 0x7f5b6010ec70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5b5800b0a0 tx=0x7f5b5800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.023+0000 7f5b685ff700 1 -- 172.21.15.2:0/360042559 shutdown_connections 2023-11-16T16:48:56.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.023+0000 7f5b685ff700 1 --2- 172.21.15.2:0/360042559 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b6010c890 0x7f5b6010ec70 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.023+0000 7f5b685ff700 1 --2- 172.21.15.2:0/360042559 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5b60074b30 0x7f5b60072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.023+0000 7f5b685ff700 1 --2- 172.21.15.2:0/360042559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b60073550 0x7f5b600739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.023+0000 7f5b685ff700 1 -- 172.21.15.2:0/360042559 >> 172.21.15.2:0/360042559 conn(0x7f5b600fb8a0 msgr2=0x7f5b600fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:56.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.023+0000 7f5b685ff700 1 -- 172.21.15.2:0/360042559 shutdown_connections 2023-11-16T16:48:56.026 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.023+0000 7f5b685ff700 1 -- 172.21.15.2:0/360042559 wait complete. 2023-11-16T16:48:56.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.024+0000 7f5b685ff700 1 Processor -- start 2023-11-16T16:48:56.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.025+0000 7f5b685ff700 1 -- start start 2023-11-16T16:48:56.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.025+0000 7f5b685ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b60073550 0x7f5b6019c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:56.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.025+0000 7f5b685ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5b60074b30 0x7f5b6019ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:56.028 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.025+0000 7f5b685ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b6010c890 0x7f5b601a3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:56.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.025+0000 7f5b685ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b601a42f0 con 0x7f5b6010c890 2023-11-16T16:48:56.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.025+0000 7f5b685ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b601a4430 con 0x7f5b60074b30 2023-11-16T16:48:56.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.025+0000 7f5b685ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b601a45c0 con 0x7f5b60073550 2023-11-16T16:48:56.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.025+0000 7f5b6639b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b60073550 0x7f5b6019c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:56.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.025+0000 7f5b66b9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b6010c890 0x7f5b601a3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:56.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.025+0000 7f5b6639b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b60073550 0x7f5b6019c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:58578/0 (socket says 172.21.15.2:58578) 2023-11-16T16:48:56.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.025+0000 7f5b6639b700 1 -- 172.21.15.2:0/613984082 learned_addr learned my addr 172.21.15.2:0/613984082 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:56.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.025+0000 7f5b65b9a700 1 --2- 172.21.15.2:0/613984082 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5b60074b30 0x7f5b6019ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:56.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.026+0000 7f5b6639b700 1 -- 172.21.15.2:0/613984082 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5b60074b30 msgr2=0x7f5b6019ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:56.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.026+0000 7f5b6639b700 1 --2- 172.21.15.2:0/613984082 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5b60074b30 0x7f5b6019ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.026+0000 7f5b6639b700 1 -- 172.21.15.2:0/613984082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b6010c890 msgr2=0x7f5b601a3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:56.029 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.026+0000 7f5b6639b700 1 --2- 172.21.15.2:0/613984082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b6010c890 0x7f5b601a3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.026+0000 7f5b6639b700 1 -- 172.21.15.2:0/613984082 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b58009d00 con 0x7f5b60073550 2023-11-16T16:48:56.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.026+0000 7f5b66b9c700 1 --2- 172.21.15.2:0/613984082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b6010c890 0x7f5b601a3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:56.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.026+0000 7f5b6639b700 1 --2- 172.21.15.2:0/613984082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b60073550 0x7f5b6019c770 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c00e970 tx=0x7f5b5c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:56.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.026+0000 7f5b577fe700 1 -- 172.21.15.2:0/613984082 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c009790 con 0x7f5b60073550 2023-11-16T16:48:56.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.026+0000 7f5b685ff700 1 -- 172.21.15.2:0/613984082 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b601122a0 con 0x7f5b60073550 2023-11-16T16:48:56.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.026+0000 7f5b577fe700 1 -- 172.21.15.2:0/613984082 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b5c00c7e0 con 0x7f5b60073550 2023-11-16T16:48:56.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.026+0000 7f5b577fe700 1 -- 172.21.15.2:0/613984082 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c0054e0 con 0x7f5b60073550 2023-11-16T16:48:56.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.027+0000 7f5b685ff700 1 -- 172.21.15.2:0/613984082 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b60112110 con 0x7f5b60073550 2023-11-16T16:48:56.030 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.028+0000 7f5b685ff700 1 -- 172.21.15.2:0/613984082 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b601023e0 con 0x7f5b60073550 2023-11-16T16:48:56.031 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.029+0000 7f5b577fe700 1 -- 172.21.15.2:0/613984082 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f5b5c00c950 con 0x7f5b60073550 2023-11-16T16:48:56.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.031+0000 7f5b577fe700 1 --2- 172.21.15.2:0/613984082 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5b4c0a11e0 0x7f5b4c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:56.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.031+0000 7f5b577fe700 1 -- 172.21.15.2:0/613984082 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f5b5c014070 con 0x7f5b60073550 2023-11-16T16:48:56.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.031+0000 7f5b65b9a700 1 --2- 172.21.15.2:0/613984082 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5b4c0a11e0 0x7f5b4c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:56.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.032+0000 7f5b65b9a700 1 --2- 172.21.15.2:0/613984082 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5b4c0a11e0 0x7f5b4c0a3690 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5b60074850 tx=0x7f5b50009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:56.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.033+0000 7f5b577fe700 1 -- 172.21.15.2:0/613984082 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5b5c096990 con 0x7f5b60073550 2023-11-16T16:48:56.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:55 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1540471505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:48:56.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:55 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2795784398' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:56.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:55 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1258136219' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:48:56.244 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.243+0000 7f5b685ff700 1 -- 172.21.15.2:0/613984082 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5b60061700 con 0x7f5b60073550 2023-11-16T16:48:56.245 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.243+0000 7f5b577fe700 1 -- 172.21.15.2:0/613984082 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5b5c099fb0 con 0x7f5b60073550 2023-11-16T16:48:56.245 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T16:48:56.247 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.246+0000 7f5b685ff700 1 -- 172.21.15.2:0/613984082 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5b4c0a11e0 msgr2=0x7f5b4c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:56.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.246+0000 7f5b685ff700 1 --2- 172.21.15.2:0/613984082 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5b4c0a11e0 0x7f5b4c0a3690 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5b60074850 tx=0x7f5b50009260 comp rx=0 tx=0).stop 2023-11-16T16:48:56.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.246+0000 7f5b685ff700 1 -- 172.21.15.2:0/613984082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b60073550 msgr2=0x7f5b6019c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:56.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.246+0000 7f5b685ff700 1 --2- 172.21.15.2:0/613984082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b60073550 0x7f5b6019c770 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c00e970 tx=0x7f5b5c00ec80 comp rx=0 tx=0).stop 2023-11-16T16:48:56.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.246+0000 7f5b685ff700 1 -- 172.21.15.2:0/613984082 shutdown_connections 2023-11-16T16:48:56.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.246+0000 7f5b685ff700 1 --2- 172.21.15.2:0/613984082 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5b4c0a11e0 0x7f5b4c0a3690 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.246+0000 7f5b685ff700 1 --2- 172.21.15.2:0/613984082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b60073550 0x7f5b6019c770 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.246+0000 7f5b685ff700 1 --2- 172.21.15.2:0/613984082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b6010c890 0x7f5b601a3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.246+0000 7f5b685ff700 1 --2- 172.21.15.2:0/613984082 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5b60074b30 0x7f5b6019ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.246+0000 7f5b685ff700 1 -- 172.21.15.2:0/613984082 >> 172.21.15.2:0/613984082 conn(0x7f5b600fb8a0 msgr2=0x7f5b600fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:56.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.246+0000 7f5b685ff700 1 -- 172.21.15.2:0/613984082 shutdown_connections 2023-11-16T16:48:56.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.246+0000 7f5b685ff700 1 -- 172.21.15.2:0/613984082 wait complete. 2023-11-16T16:48:56.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:55 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1540471505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:48:56.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:55 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2795784398' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:56.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:55 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1258136219' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:48:56.266 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T16:48:56.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:55 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1540471505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:48:56.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:55 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2795784398' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:56.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:55 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1258136219' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:48:56.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.473+0000 7f39753ff700 1 -- 172.21.15.2:0/2660279605 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3970101e40 msgr2=0x7f3970108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:56.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.473+0000 7f39753ff700 1 --2- 172.21.15.2:0/2660279605 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3970101e40 0x7f3970108700 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f396800b0a0 tx=0x7f396800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.473+0000 7f39753ff700 1 -- 172.21.15.2:0/2660279605 shutdown_connections 2023-11-16T16:48:56.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.473+0000 7f39753ff700 1 --2- 172.21.15.2:0/2660279605 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3970101e40 0x7f3970108700 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.473+0000 7f39753ff700 1 --2- 172.21.15.2:0/2660279605 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f39701002c0 0x7f39701006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.473+0000 7f39753ff700 1 --2- 172.21.15.2:0/2660279605 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39701014b0 0x7f3970101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.473+0000 7f39753ff700 1 -- 172.21.15.2:0/2660279605 >> 172.21.15.2:0/2660279605 conn(0x7f39700fb8a0 msgr2=0x7f39700fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:56.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.473+0000 7f39753ff700 1 -- 172.21.15.2:0/2660279605 shutdown_connections 2023-11-16T16:48:56.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.473+0000 7f39753ff700 1 -- 172.21.15.2:0/2660279605 wait complete. 2023-11-16T16:48:56.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.474+0000 7f39753ff700 1 Processor -- start 2023-11-16T16:48:56.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.474+0000 7f39753ff700 1 -- start start 2023-11-16T16:48:56.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.474+0000 7f39753ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39701002c0 0x7f3970198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:56.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.474+0000 7f39753ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39701014b0 0x7f3970198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:56.478 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.474+0000 7f39753ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3970101e40 0x7f397019f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:56.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.474+0000 7f39753ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397019fec0 con 0x7f39701014b0 2023-11-16T16:48:56.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.474+0000 7f39753ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39701a0000 con 0x7f3970101e40 2023-11-16T16:48:56.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.474+0000 7f39753ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39701a0190 con 0x7f39701002c0 2023-11-16T16:48:56.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.475+0000 7f3967fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39701014b0 0x7f3970198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:56.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.475+0000 7f3967fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39701014b0 0x7f3970198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44234/0 (socket says 172.21.15.2:44234) 2023-11-16T16:48:56.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.475+0000 7f3967fff700 1 -- 172.21.15.2:0/91223908 learned_addr learned my addr 172.21.15.2:0/91223908 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:56.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.475+0000 7f396effd700 1 --2- 172.21.15.2:0/91223908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39701002c0 0x7f3970198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:56.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.475+0000 7f396f7fe700 1 --2- 172.21.15.2:0/91223908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3970101e40 0x7f397019f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:56.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.475+0000 7f3967fff700 1 -- 172.21.15.2:0/91223908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39701002c0 msgr2=0x7f3970198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:56.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.475+0000 7f3967fff700 1 --2- 172.21.15.2:0/91223908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39701002c0 0x7f3970198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.475+0000 7f3967fff700 1 -- 172.21.15.2:0/91223908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3970101e40 msgr2=0x7f397019f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:56.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.475+0000 7f3967fff700 1 --2- 172.21.15.2:0/91223908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3970101e40 0x7f397019f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.475+0000 7f3967fff700 1 -- 172.21.15.2:0/91223908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3968009d00 con 0x7f39701014b0 2023-11-16T16:48:56.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.475+0000 7f3967fff700 1 --2- 172.21.15.2:0/91223908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39701014b0 0x7f3970198880 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f395800e9a0 tx=0x7f395800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:56.480 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.476+0000 7f396cff9700 1 -- 172.21.15.2:0/91223908 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958009790 con 0x7f39701014b0 2023-11-16T16:48:56.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.476+0000 7f396cff9700 1 -- 172.21.15.2:0/91223908 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3958010430 con 0x7f39701014b0 2023-11-16T16:48:56.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.476+0000 7f39753ff700 1 -- 172.21.15.2:0/91223908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397010bc90 con 0x7f39701014b0 2023-11-16T16:48:56.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.476+0000 7f396cff9700 1 -- 172.21.15.2:0/91223908 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958018680 con 0x7f39701014b0 2023-11-16T16:48:56.481 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.476+0000 7f39753ff700 1 -- 172.21.15.2:0/91223908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397010bb00 con 0x7f39701014b0 2023-11-16T16:48:56.482 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.477+0000 7f39753ff700 1 -- 172.21.15.2:0/91223908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39701a0880 con 0x7f39701014b0 2023-11-16T16:48:56.482 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.479+0000 7f396cff9700 1 -- 172.21.15.2:0/91223908 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f39580105a0 con 0x7f39701014b0 2023-11-16T16:48:56.482 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.480+0000 7f396cff9700 1 --2- 172.21.15.2:0/91223908 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f395c0a11e0 0x7f395c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:56.482 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.480+0000 7f396cff9700 1 -- 172.21.15.2:0/91223908 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f3958014070 con 0x7f39701014b0 2023-11-16T16:48:56.482 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.480+0000 7f396effd700 1 --2- 172.21.15.2:0/91223908 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f395c0a11e0 0x7f395c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:56.482 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.481+0000 7f396effd700 1 --2- 172.21.15.2:0/91223908 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f395c0a11e0 0x7f395c0a3690 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3970101310 tx=0x7f3960009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:56.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.484+0000 7f396cff9700 1 -- 172.21.15.2:0/91223908 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3958051730 con 0x7f39701014b0 2023-11-16T16:48:56.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.693+0000 7f39753ff700 1 -- 172.21.15.2:0/91223908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f39701a0e30 con 0x7f39701014b0 2023-11-16T16:48:56.695 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.694+0000 7f396cff9700 1 -- 172.21.15.2:0/91223908 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3958096250 con 0x7f39701014b0 2023-11-16T16:48:56.695 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2023-11-16T16:48:56.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.696+0000 7f39753ff700 1 -- 172.21.15.2:0/91223908 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f395c0a11e0 msgr2=0x7f395c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:56.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.696+0000 7f39753ff700 1 --2- 172.21.15.2:0/91223908 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f395c0a11e0 0x7f395c0a3690 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3970101310 tx=0x7f3960009260 comp rx=0 tx=0).stop 2023-11-16T16:48:56.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.696+0000 7f39753ff700 1 -- 172.21.15.2:0/91223908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39701014b0 msgr2=0x7f3970198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:56.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.696+0000 7f39753ff700 1 --2- 172.21.15.2:0/91223908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39701014b0 0x7f3970198880 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f395800e9a0 tx=0x7f395800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.696+0000 7f39753ff700 1 -- 172.21.15.2:0/91223908 shutdown_connections 2023-11-16T16:48:56.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.696+0000 7f39753ff700 1 --2- 172.21.15.2:0/91223908 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f395c0a11e0 0x7f395c0a3690 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.696+0000 7f39753ff700 1 --2- 172.21.15.2:0/91223908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f39701002c0 0x7f3970198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.696+0000 7f39753ff700 1 --2- 172.21.15.2:0/91223908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3970101e40 0x7f397019f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.698 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.696+0000 7f39753ff700 1 --2- 172.21.15.2:0/91223908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39701014b0 0x7f3970198880 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.696+0000 7f39753ff700 1 -- 172.21.15.2:0/91223908 >> 172.21.15.2:0/91223908 conn(0x7f39700fb8a0 msgr2=0x7f3970106c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:56.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.696+0000 7f39753ff700 1 -- 172.21.15.2:0/91223908 shutdown_connections 2023-11-16T16:48:56.699 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.696+0000 7f39753ff700 1 -- 172.21.15.2:0/91223908 wait complete. 2023-11-16T16:48:56.717 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T16:48:56.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.922+0000 7fca8cbff700 1 -- 172.21.15.2:0/3002984258 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca881002d0 msgr2=0x7fca881006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:56.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.922+0000 7fca8cbff700 1 --2- 172.21.15.2:0/3002984258 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca881002d0 0x7fca881006e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fca78009990 tx=0x7fca78009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.922+0000 7fca8cbff700 1 -- 172.21.15.2:0/3002984258 shutdown_connections 2023-11-16T16:48:56.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.922+0000 7fca8cbff700 1 --2- 172.21.15.2:0/3002984258 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca881002d0 0x7fca881006e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.922+0000 7fca8cbff700 1 --2- 172.21.15.2:0/3002984258 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca88101e50 0x7fca88108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.922+0000 7fca8cbff700 1 --2- 172.21.15.2:0/3002984258 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca881014c0 0x7fca88101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.922+0000 7fca8cbff700 1 -- 172.21.15.2:0/3002984258 >> 172.21.15.2:0/3002984258 conn(0x7fca880fb8b0 msgr2=0x7fca880fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:56.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.922+0000 7fca8cbff700 1 -- 172.21.15.2:0/3002984258 shutdown_connections 2023-11-16T16:48:56.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.922+0000 7fca8cbff700 1 -- 172.21.15.2:0/3002984258 wait complete. 2023-11-16T16:48:56.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.923+0000 7fca8cbff700 1 Processor -- start 2023-11-16T16:48:56.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.923+0000 7fca8cbff700 1 -- start start 2023-11-16T16:48:56.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.924+0000 7fca8cbff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca881002d0 0x7fca880748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:56.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.924+0000 7fca8cbff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca881014c0 0x7fca88072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:56.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.924+0000 7fca8cbff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca88101e50 0x7fca88073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:56.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.924+0000 7fca8cbff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca88073bc0 con 0x7fca88101e50 2023-11-16T16:48:56.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.924+0000 7fca8cbff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca88073d30 con 0x7fca881002d0 2023-11-16T16:48:56.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.924+0000 7fca8cbff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca881a0070 con 0x7fca881014c0 2023-11-16T16:48:56.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.924+0000 7fca86d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca88101e50 0x7fca88073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:56.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.924+0000 7fca86d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca88101e50 0x7fca88073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44248/0 (socket says 172.21.15.2:44248) 2023-11-16T16:48:56.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.924+0000 7fca86d9d700 1 -- 172.21.15.2:0/1027834976 learned_addr learned my addr 172.21.15.2:0/1027834976 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:56.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.924+0000 7fca8659c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca881002d0 0x7fca880748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:56.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.924+0000 7fca85d9b700 1 --2- 172.21.15.2:0/1027834976 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca881014c0 0x7fca88072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:56.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.924+0000 7fca86d9d700 1 -- 172.21.15.2:0/1027834976 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca881014c0 msgr2=0x7fca88072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:56.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.924+0000 7fca86d9d700 1 --2- 172.21.15.2:0/1027834976 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca881014c0 0x7fca88072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.924+0000 7fca86d9d700 1 -- 172.21.15.2:0/1027834976 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca881002d0 msgr2=0x7fca880748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:56.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.924+0000 7fca86d9d700 1 --2- 172.21.15.2:0/1027834976 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca881002d0 0x7fca880748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:56.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.924+0000 7fca86d9d700 1 -- 172.21.15.2:0/1027834976 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca780095f0 con 0x7fca88101e50 2023-11-16T16:48:56.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.925+0000 7fca86d9d700 1 --2- 172.21.15.2:0/1027834976 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca88101e50 0x7fca88073420 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fca8000ef30 tx=0x7fca8000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:56.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.925+0000 7fca777fe700 1 -- 172.21.15.2:0/1027834976 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca80018070 con 0x7fca88101e50 2023-11-16T16:48:56.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.925+0000 7fca8cbff700 1 -- 172.21.15.2:0/1027834976 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca8810bbd0 con 0x7fca88101e50 2023-11-16T16:48:56.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.925+0000 7fca777fe700 1 -- 172.21.15.2:0/1027834976 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca8000f040 con 0x7fca88101e50 2023-11-16T16:48:56.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.925+0000 7fca777fe700 1 -- 172.21.15.2:0/1027834976 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8000cca0 con 0x7fca88101e50 2023-11-16T16:48:56.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.925+0000 7fca8cbff700 1 -- 172.21.15.2:0/1027834976 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8810ba40 con 0x7fca88101e50 2023-11-16T16:48:56.929 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.926+0000 7fca8cbff700 1 -- 172.21.15.2:0/1027834976 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca881078c0 con 0x7fca88101e50 2023-11-16T16:48:56.930 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.928+0000 7fca777fe700 1 -- 172.21.15.2:0/1027834976 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fca80003a00 con 0x7fca88101e50 2023-11-16T16:48:56.931 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.930+0000 7fca777fe700 1 --2- 172.21.15.2:0/1027834976 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fca700a11e0 0x7fca700a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:56.931 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.930+0000 7fca777fe700 1 -- 172.21.15.2:0/1027834976 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fca800cc910 con 0x7fca88101e50 2023-11-16T16:48:56.931 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.930+0000 7fca8659c700 1 --2- 172.21.15.2:0/1027834976 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fca700a11e0 0x7fca700a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:56.932 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.930+0000 7fca8659c700 1 --2- 172.21.15.2:0/1027834976 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fca700a11e0 0x7fca700a3690 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fca7800b2e0 tx=0x7fca78005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:56.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:56.933+0000 7fca777fe700 1 -- 172.21.15.2:0/1027834976 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fca80095bd0 con 0x7fca88101e50 2023-11-16T16:48:57.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.144+0000 7fca8cbff700 1 -- 172.21.15.2:0/1027834976 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fca88061700 con 0x7fca88101e50 2023-11-16T16:48:57.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.144+0000 7fca777fe700 1 -- 172.21.15.2:0/1027834976 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7fca800991f0 con 0x7fca88101e50 2023-11-16T16:48:57.146 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T16:48:57.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.147+0000 7fca8cbff700 1 -- 172.21.15.2:0/1027834976 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fca700a11e0 msgr2=0x7fca700a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:57.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.147+0000 7fca8cbff700 1 --2- 172.21.15.2:0/1027834976 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fca700a11e0 0x7fca700a3690 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fca7800b2e0 tx=0x7fca78005960 comp rx=0 tx=0).stop 2023-11-16T16:48:57.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.147+0000 7fca8cbff700 1 -- 172.21.15.2:0/1027834976 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca88101e50 msgr2=0x7fca88073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:57.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.147+0000 7fca8cbff700 1 --2- 172.21.15.2:0/1027834976 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca88101e50 0x7fca88073420 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fca8000ef30 tx=0x7fca8000c3a0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.147+0000 7fca8cbff700 1 -- 172.21.15.2:0/1027834976 shutdown_connections 2023-11-16T16:48:57.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.147+0000 7fca8cbff700 1 --2- 172.21.15.2:0/1027834976 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fca700a11e0 0x7fca700a3690 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.147+0000 7fca8cbff700 1 --2- 172.21.15.2:0/1027834976 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca88101e50 0x7fca88073420 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.147+0000 7fca8cbff700 1 --2- 172.21.15.2:0/1027834976 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fca881002d0 0x7fca880748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.147+0000 7fca8cbff700 1 --2- 172.21.15.2:0/1027834976 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca881014c0 0x7fca88072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.147+0000 7fca8cbff700 1 -- 172.21.15.2:0/1027834976 >> 172.21.15.2:0/1027834976 conn(0x7fca880fb8b0 msgr2=0x7fca88104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:57.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.147+0000 7fca8cbff700 1 -- 172.21.15.2:0/1027834976 shutdown_connections 2023-11-16T16:48:57.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.147+0000 7fca8cbff700 1 -- 172.21.15.2:0/1027834976 wait complete. 2023-11-16T16:48:57.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:56 smithi002 ceph-mon[128712]: pgmap v329: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s; 6.3 KiB/s, 1 keys/s, 1 objects/s recovering 2023-11-16T16:48:57.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:56 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/613984082' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:48:57.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:56 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/91223908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:48:57.168 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:48:57.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:56 smithi141 ceph-mon[131184]: pgmap v329: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s; 6.3 KiB/s, 1 keys/s, 1 objects/s recovering 2023-11-16T16:48:57.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:56 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/613984082' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:48:57.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:56 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/91223908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:48:57.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:56 smithi155 ceph-mon[131607]: pgmap v329: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s; 6.3 KiB/s, 1 keys/s, 1 objects/s recovering 2023-11-16T16:48:57.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:56 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/613984082' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:48:57.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:56 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/91223908' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:48:57.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.370+0000 7ff0d15ff700 1 -- 172.21.15.2:0/3017274119 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0cc100290 msgr2=0x7ff0cc1006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:57.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.370+0000 7ff0d15ff700 1 --2- 172.21.15.2:0/3017274119 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0cc100290 0x7ff0cc1006a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc009990 tx=0x7ff0bc009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.370+0000 7ff0d15ff700 1 -- 172.21.15.2:0/3017274119 shutdown_connections 2023-11-16T16:48:57.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.370+0000 7ff0d15ff700 1 --2- 172.21.15.2:0/3017274119 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0cc101e10 0x7ff0cc1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.370+0000 7ff0d15ff700 1 --2- 172.21.15.2:0/3017274119 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0cc100290 0x7ff0cc1006a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.370+0000 7ff0d15ff700 1 --2- 172.21.15.2:0/3017274119 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0cc101480 0x7ff0cc1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.370+0000 7ff0d15ff700 1 -- 172.21.15.2:0/3017274119 >> 172.21.15.2:0/3017274119 conn(0x7ff0cc0fb870 msgr2=0x7ff0cc0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:57.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.370+0000 7ff0d15ff700 1 -- 172.21.15.2:0/3017274119 shutdown_connections 2023-11-16T16:48:57.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.370+0000 7ff0d15ff700 1 -- 172.21.15.2:0/3017274119 wait complete. 2023-11-16T16:48:57.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.371+0000 7ff0d15ff700 1 Processor -- start 2023-11-16T16:48:57.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.371+0000 7ff0d15ff700 1 -- start start 2023-11-16T16:48:57.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.371+0000 7ff0d15ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0cc100290 0x7ff0cc198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:57.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.371+0000 7ff0d15ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0cc101480 0x7ff0cc1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:57.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.371+0000 7ff0d15ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0cc101e10 0x7ff0cc19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:57.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.371+0000 7ff0d15ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc19fd00 con 0x7ff0cc101e10 2023-11-16T16:48:57.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.371+0000 7ff0d15ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc19fe40 con 0x7ff0cc101480 2023-11-16T16:48:57.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.371+0000 7ff0d15ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc19ffd0 con 0x7ff0cc100290 2023-11-16T16:48:57.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.372+0000 7ff0caffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0cc100290 0x7ff0cc198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:57.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.372+0000 7ff0cb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0cc101e10 0x7ff0cc19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:57.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.372+0000 7ff0caffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0cc100290 0x7ff0cc198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:58622/0 (socket says 172.21.15.2:58622) 2023-11-16T16:48:57.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.372+0000 7ff0caffd700 1 -- 172.21.15.2:0/1702712837 learned_addr learned my addr 172.21.15.2:0/1702712837 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:57.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.372+0000 7ff0c3fff700 1 --2- 172.21.15.2:0/1702712837 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0cc101480 0x7ff0cc1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:57.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.372+0000 7ff0caffd700 1 -- 172.21.15.2:0/1702712837 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0cc101480 msgr2=0x7ff0cc1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:57.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.372+0000 7ff0caffd700 1 --2- 172.21.15.2:0/1702712837 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0cc101480 0x7ff0cc1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.372+0000 7ff0caffd700 1 -- 172.21.15.2:0/1702712837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0cc101e10 msgr2=0x7ff0cc19f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:57.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.372+0000 7ff0caffd700 1 --2- 172.21.15.2:0/1702712837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0cc101e10 0x7ff0cc19f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.372+0000 7ff0caffd700 1 -- 172.21.15.2:0/1702712837 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c4009c40 con 0x7ff0cc100290 2023-11-16T16:48:57.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.372+0000 7ff0cb7fe700 1 --2- 172.21.15.2:0/1702712837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0cc101e10 0x7ff0cc19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:57.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.372+0000 7ff0caffd700 1 --2- 172.21.15.2:0/1702712837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0cc100290 0x7ff0cc198180 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc000c00 tx=0x7ff0bc005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:57.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.373+0000 7ff0c8ff9700 1 -- 172.21.15.2:0/1702712837 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc01d070 con 0x7ff0cc100290 2023-11-16T16:48:57.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.373+0000 7ff0d15ff700 1 -- 172.21.15.2:0/1702712837 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0bc0095f0 con 0x7ff0cc100290 2023-11-16T16:48:57.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.373+0000 7ff0c8ff9700 1 -- 172.21.15.2:0/1702712837 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff0bc022470 con 0x7ff0cc100290 2023-11-16T16:48:57.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.373+0000 7ff0c8ff9700 1 -- 172.21.15.2:0/1702712837 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc017650 con 0x7ff0cc100290 2023-11-16T16:48:57.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.373+0000 7ff0d15ff700 1 -- 172.21.15.2:0/1702712837 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0cc10bbf0 con 0x7ff0cc100290 2023-11-16T16:48:57.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.374+0000 7ff0d15ff700 1 -- 172.21.15.2:0/1702712837 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0cc10bd80 con 0x7ff0cc100290 2023-11-16T16:48:57.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.376+0000 7ff0c8ff9700 1 -- 172.21.15.2:0/1702712837 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7ff0bc0225e0 con 0x7ff0cc100290 2023-11-16T16:48:57.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.377+0000 7ff0c8ff9700 1 --2- 172.21.15.2:0/1702712837 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff0b80a11e0 0x7ff0b80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:57.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.377+0000 7ff0c8ff9700 1 -- 172.21.15.2:0/1702712837 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7ff0bc0ce500 con 0x7ff0cc100290 2023-11-16T16:48:57.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.377+0000 7ff0c3fff700 1 --2- 172.21.15.2:0/1702712837 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff0b80a11e0 0x7ff0b80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:57.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.378+0000 7ff0c3fff700 1 --2- 172.21.15.2:0/1702712837 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff0b80a11e0 0x7ff0b80a3690 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4009c60 tx=0x7ff0b4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:57.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.379+0000 7ff0c8ff9700 1 -- 172.21.15.2:0/1702712837 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff0bc0977c0 con 0x7ff0cc100290 2023-11-16T16:48:57.637 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.635+0000 7ff0d15ff700 1 -- 172.21.15.2:0/1702712837 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0cc192880 con 0x7ff0cc100290 2023-11-16T16:48:57.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.636+0000 7ff0c8ff9700 1 -- 172.21.15.2:0/1702712837 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10811 (secure 0 0 0) 0x7ff0bc09ade0 con 0x7ff0cc100290 2023-11-16T16:48:57.638 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:48:57.638 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/1610211225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":1610211225},{"type":"v1","addr":"172.21.15.2:6837","nonce":1610211225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14538,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2583261818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2583261818},{"type":"v1","addr":"172.21.15.2:6839","nonce":2583261818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14550,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3170991923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3170991923},{"type":"v1","addr":"172.21.15.2:6841","nonce":3170991923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:26.349176+0000","modified":"2023-11-16T16:48:05.774072+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24544,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.141:6835/3516869188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3516869188},{"type":"v1","addr":"172.21.15.141:6835","nonce":3516869188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/2527313191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":2527313191},{"type":"v1","addr":"172.21.15.155:6835","nonce":2527313191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24523":{"gid":24523,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1607756751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1607756751},{"type":"v1","addr":"172.21.15.155:6837","nonce":1607756751}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.141:6837/2934891475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6836","nonce":2934891475},{"type":"v1","addr":"172.21.15.141:6837","nonce":2934891475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24544":{"gid":24544,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1158457009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1158457009},{"type":"v1","addr":"172.21.15.155:6839","nonce":1158457009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6839/3916358272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6838","nonce":3916358272},{"type":"v1","addr":"172.21.15.141:6839","nonce":3916358272}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:48:57.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.639+0000 7ff0d15ff700 1 -- 172.21.15.2:0/1702712837 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff0b80a11e0 msgr2=0x7ff0b80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:57.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.639+0000 7ff0d15ff700 1 --2- 172.21.15.2:0/1702712837 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff0b80a11e0 0x7ff0b80a3690 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4009c60 tx=0x7ff0b4009260 comp rx=0 tx=0).stop 2023-11-16T16:48:57.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.639+0000 7ff0d15ff700 1 -- 172.21.15.2:0/1702712837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0cc100290 msgr2=0x7ff0cc198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:57.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.639+0000 7ff0d15ff700 1 --2- 172.21.15.2:0/1702712837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0cc100290 0x7ff0cc198180 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc000c00 tx=0x7ff0bc005740 comp rx=0 tx=0).stop 2023-11-16T16:48:57.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.639+0000 7ff0d15ff700 1 -- 172.21.15.2:0/1702712837 shutdown_connections 2023-11-16T16:48:57.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.639+0000 7ff0d15ff700 1 --2- 172.21.15.2:0/1702712837 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff0b80a11e0 0x7ff0b80a3690 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.639+0000 7ff0d15ff700 1 --2- 172.21.15.2:0/1702712837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0cc100290 0x7ff0cc198180 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.639+0000 7ff0d15ff700 1 --2- 172.21.15.2:0/1702712837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0cc101e10 0x7ff0cc19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.639+0000 7ff0d15ff700 1 --2- 172.21.15.2:0/1702712837 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff0cc101480 0x7ff0cc1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.639+0000 7ff0d15ff700 1 -- 172.21.15.2:0/1702712837 >> 172.21.15.2:0/1702712837 conn(0x7ff0cc0fb870 msgr2=0x7ff0cc106db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.639+0000 7ff0d15ff700 1 -- 172.21.15.2:0/1702712837 shutdown_connections 2023-11-16T16:48:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.639+0000 7ff0d15ff700 1 -- 172.21.15.2:0/1702712837 wait complete. 2023-11-16T16:48:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 26 2023-11-16T16:48:57.661 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:48:57.865 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.863+0000 7f1366fff700 1 -- 172.21.15.2:0/705049088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1360101e10 msgr2=0x7f13601086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:57.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.863+0000 7f1366fff700 1 --2- 172.21.15.2:0/705049088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1360101e10 0x7f13601086d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f135800b0a0 tx=0x7f135800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.864+0000 7f1366fff700 1 -- 172.21.15.2:0/705049088 shutdown_connections 2023-11-16T16:48:57.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.864+0000 7f1366fff700 1 --2- 172.21.15.2:0/705049088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1360100290 0x7f13601006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.864+0000 7f1366fff700 1 --2- 172.21.15.2:0/705049088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1360101e10 0x7f13601086d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.864+0000 7f1366fff700 1 --2- 172.21.15.2:0/705049088 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1360101480 0x7f13601018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.864+0000 7f1366fff700 1 -- 172.21.15.2:0/705049088 >> 172.21.15.2:0/705049088 conn(0x7f13600fb870 msgr2=0x7f13600fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:57.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.864+0000 7f1366fff700 1 -- 172.21.15.2:0/705049088 shutdown_connections 2023-11-16T16:48:57.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.864+0000 7f1366fff700 1 -- 172.21.15.2:0/705049088 wait complete. 2023-11-16T16:48:57.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.865+0000 7f1366fff700 1 Processor -- start 2023-11-16T16:48:57.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.865+0000 7f1366fff700 1 -- start start 2023-11-16T16:48:57.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.865+0000 7f1366fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1360100290 0x7f1360196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:57.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.865+0000 7f1366fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1360101480 0x7f13601965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:57.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.865+0000 7f1366fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1360101e10 0x7f136019d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:57.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.865+0000 7f1366fff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136019dc10 con 0x7f1360101e10 2023-11-16T16:48:57.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.865+0000 7f1366fff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136019dd50 con 0x7f1360100290 2023-11-16T16:48:57.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.865+0000 7f1366fff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136019dee0 con 0x7f1360101480 2023-11-16T16:48:57.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.866+0000 7f136559c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1360101e10 0x7f136019d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:57.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.866+0000 7f136559c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1360101e10 0x7f136019d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44270/0 (socket says 172.21.15.2:44270) 2023-11-16T16:48:57.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.866+0000 7f135ffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1360101480 0x7f13601965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:57.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.866+0000 7f136559c700 1 -- 172.21.15.2:0/3319900235 learned_addr learned my addr 172.21.15.2:0/3319900235 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:57.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.866+0000 7f1364d9b700 1 --2- 172.21.15.2:0/3319900235 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1360100290 0x7f1360196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:57.868 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.866+0000 7f136559c700 1 -- 172.21.15.2:0/3319900235 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1360101480 msgr2=0x7f13601965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:57.869 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.866+0000 7f136559c700 1 --2- 172.21.15.2:0/3319900235 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1360101480 0x7f13601965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.869 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.866+0000 7f136559c700 1 -- 172.21.15.2:0/3319900235 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1360100290 msgr2=0x7f1360196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:57.869 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.866+0000 7f136559c700 1 --2- 172.21.15.2:0/3319900235 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1360100290 0x7f1360196090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:57.869 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.866+0000 7f136559c700 1 -- 172.21.15.2:0/3319900235 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1358009d00 con 0x7f1360101e10 2023-11-16T16:48:57.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.866+0000 7f136559c700 1 --2- 172.21.15.2:0/3319900235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1360101e10 0x7f136019d640 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f135800d010 tx=0x7f135800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:57.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.867+0000 7f135dffb700 1 -- 172.21.15.2:0/3319900235 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135800e040 con 0x7f1360101e10 2023-11-16T16:48:57.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.867+0000 7f135dffb700 1 -- 172.21.15.2:0/3319900235 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1358003e10 con 0x7f1360101e10 2023-11-16T16:48:57.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.867+0000 7f135dffb700 1 -- 172.21.15.2:0/3319900235 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13580052a0 con 0x7f1360101e10 2023-11-16T16:48:57.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.867+0000 7f1366fff700 1 -- 172.21.15.2:0/3319900235 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f136010bc70 con 0x7f1360101e10 2023-11-16T16:48:57.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.867+0000 7f1366fff700 1 -- 172.21.15.2:0/3319900235 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136010bae0 con 0x7f1360101e10 2023-11-16T16:48:57.870 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.868+0000 7f1366fff700 1 -- 172.21.15.2:0/3319900235 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f136019e430 con 0x7f1360101e10 2023-11-16T16:48:57.871 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.870+0000 7f135dffb700 1 -- 172.21.15.2:0/3319900235 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f135801d480 con 0x7f1360101e10 2023-11-16T16:48:57.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.871+0000 7f135dffb700 1 --2- 172.21.15.2:0/3319900235 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f13500a1260 0x7f13500a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:57.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.871+0000 7f135dffb700 1 -- 172.21.15.2:0/3319900235 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f13580ce130 con 0x7f1360101e10 2023-11-16T16:48:57.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.871+0000 7f1364d9b700 1 --2- 172.21.15.2:0/3319900235 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f13500a1260 0x7f13500a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:57.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.872+0000 7f1364d9b700 1 --2- 172.21.15.2:0/3319900235 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f13500a1260 0x7f13500a3710 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f13601012e0 tx=0x7f1354009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:57.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:57.875+0000 7f135dffb700 1 -- 172.21.15.2:0/3319900235 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f13580973f0 con 0x7f1360101e10 2023-11-16T16:48:58.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.088+0000 7f1366fff700 1 -- 172.21.15.2:0/3319900235 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1360107100 con 0x7f1360101e10 2023-11-16T16:48:58.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.090+0000 7f135dffb700 1 -- 172.21.15.2:0/3319900235 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24567 (secure 0 0 0) 0x7f135809aa10 con 0x7f1360101e10 2023-11-16T16:48:58.092 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:48:58.092 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":93,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:48:45.491739+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:24.458452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:25.786005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:29.313181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:48:58.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.092+0000 7f1366fff700 1 -- 172.21.15.2:0/3319900235 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f13500a1260 msgr2=0x7f13500a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.092+0000 7f1366fff700 1 --2- 172.21.15.2:0/3319900235 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f13500a1260 0x7f13500a3710 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f13601012e0 tx=0x7f1354009260 comp rx=0 tx=0).stop 2023-11-16T16:48:58.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.092+0000 7f1366fff700 1 -- 172.21.15.2:0/3319900235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1360101e10 msgr2=0x7f136019d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.092+0000 7f1366fff700 1 --2- 172.21.15.2:0/3319900235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1360101e10 0x7f136019d640 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f135800d010 tx=0x7f135800bc80 comp rx=0 tx=0).stop 2023-11-16T16:48:58.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.092+0000 7f1366fff700 1 -- 172.21.15.2:0/3319900235 shutdown_connections 2023-11-16T16:48:58.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.092+0000 7f1366fff700 1 --2- 172.21.15.2:0/3319900235 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f13500a1260 0x7f13500a3710 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.092+0000 7f1366fff700 1 --2- 172.21.15.2:0/3319900235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1360101e10 0x7f136019d640 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.092+0000 7f1366fff700 1 --2- 172.21.15.2:0/3319900235 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1360100290 0x7f1360196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.092+0000 7f1366fff700 1 --2- 172.21.15.2:0/3319900235 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1360101480 0x7f13601965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.092+0000 7f1366fff700 1 -- 172.21.15.2:0/3319900235 >> 172.21.15.2:0/3319900235 conn(0x7f13600fb870 msgr2=0x7f13600fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:58.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.092+0000 7f1366fff700 1 -- 172.21.15.2:0/3319900235 shutdown_connections 2023-11-16T16:48:58.095 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.092+0000 7f1366fff700 1 -- 172.21.15.2:0/3319900235 wait complete. 2023-11-16T16:48:58.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:57 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1027834976' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:48:58.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:57 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1702712837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:48:58.113 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:48:58.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:57 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1027834976' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:48:58.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:57 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1702712837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:48:58.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.315+0000 7ff79e9ff700 1 -- 172.21.15.2:0/910860874 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff798100290 msgr2=0x7ff7981006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.315+0000 7ff79e9ff700 1 --2- 172.21.15.2:0/910860874 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff798100290 0x7ff7981006a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff788009990 tx=0x7ff788009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.315+0000 7ff79e9ff700 1 -- 172.21.15.2:0/910860874 shutdown_connections 2023-11-16T16:48:58.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.315+0000 7ff79e9ff700 1 --2- 172.21.15.2:0/910860874 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff798101e10 0x7ff7981086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.315+0000 7ff79e9ff700 1 --2- 172.21.15.2:0/910860874 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff798100290 0x7ff7981006a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.315+0000 7ff79e9ff700 1 --2- 172.21.15.2:0/910860874 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff798101480 0x7ff7981018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.315+0000 7ff79e9ff700 1 -- 172.21.15.2:0/910860874 >> 172.21.15.2:0/910860874 conn(0x7ff7980fb870 msgr2=0x7ff7980fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:58.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.315+0000 7ff79e9ff700 1 -- 172.21.15.2:0/910860874 shutdown_connections 2023-11-16T16:48:58.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.315+0000 7ff79e9ff700 1 -- 172.21.15.2:0/910860874 wait complete. 2023-11-16T16:48:58.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.316+0000 7ff79e9ff700 1 Processor -- start 2023-11-16T16:48:58.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.317+0000 7ff79e9ff700 1 -- start start 2023-11-16T16:48:58.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.317+0000 7ff79e9ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff798100290 0x7ff7981981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.317+0000 7ff79e9ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff798101480 0x7ff798198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.317+0000 7ff79e9ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff798101e10 0x7ff79819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.317+0000 7ff79e9ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79819fd50 con 0x7ff798100290 2023-11-16T16:48:58.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.317+0000 7ff79e9ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79819fe90 con 0x7ff798101480 2023-11-16T16:48:58.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.317+0000 7ff79e9ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7981a0020 con 0x7ff798101e10 2023-11-16T16:48:58.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.317+0000 7ff797fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff798100290 0x7ff7981981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.317+0000 7ff797fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff798100290 0x7ff7981981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44294/0 (socket says 172.21.15.2:44294) 2023-11-16T16:48:58.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.317+0000 7ff797fff700 1 -- 172.21.15.2:0/3473697270 learned_addr learned my addr 172.21.15.2:0/3473697270 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:58.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.317+0000 7ff79cf9c700 1 --2- 172.21.15.2:0/3473697270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff798101e10 0x7ff79819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.317+0000 7ff7977fe700 1 --2- 172.21.15.2:0/3473697270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff798101480 0x7ff798198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.318+0000 7ff797fff700 1 -- 172.21.15.2:0/3473697270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff798101e10 msgr2=0x7ff79819f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.318+0000 7ff797fff700 1 --2- 172.21.15.2:0/3473697270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff798101e10 0x7ff79819f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.318+0000 7ff797fff700 1 -- 172.21.15.2:0/3473697270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff798101480 msgr2=0x7ff798198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.318+0000 7ff797fff700 1 --2- 172.21.15.2:0/3473697270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff798101480 0x7ff798198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.318+0000 7ff797fff700 1 -- 172.21.15.2:0/3473697270 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7880095f0 con 0x7ff798100290 2023-11-16T16:48:58.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.318+0000 7ff797fff700 1 --2- 172.21.15.2:0/3473697270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff798100290 0x7ff7981981d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff7880098e0 tx=0x7ff788004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:58.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.318+0000 7ff7957fa700 1 -- 172.21.15.2:0/3473697270 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78801d070 con 0x7ff798100290 2023-11-16T16:48:58.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.318+0000 7ff7957fa700 1 -- 172.21.15.2:0/3473697270 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff788005030 con 0x7ff798100290 2023-11-16T16:48:58.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.318+0000 7ff79e9ff700 1 -- 172.21.15.2:0/3473697270 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff79810bc40 con 0x7ff798100290 2023-11-16T16:48:58.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.318+0000 7ff7957fa700 1 -- 172.21.15.2:0/3473697270 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff788004b50 con 0x7ff798100290 2023-11-16T16:48:58.322 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.318+0000 7ff79e9ff700 1 -- 172.21.15.2:0/3473697270 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff79810bab0 con 0x7ff798100290 2023-11-16T16:48:58.322 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.319+0000 7ff79e9ff700 1 -- 172.21.15.2:0/3473697270 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7981928d0 con 0x7ff798100290 2023-11-16T16:48:58.323 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.321+0000 7ff7957fa700 1 -- 172.21.15.2:0/3473697270 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7ff788004d30 con 0x7ff798100290 2023-11-16T16:48:58.324 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.323+0000 7ff7957fa700 1 --2- 172.21.15.2:0/3473697270 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff7800a1260 0x7ff7800a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.324 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.323+0000 7ff7957fa700 1 -- 172.21.15.2:0/3473697270 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7ff7880ce080 con 0x7ff798100290 2023-11-16T16:48:58.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.323+0000 7ff7977fe700 1 --2- 172.21.15.2:0/3473697270 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff7800a1260 0x7ff7800a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.324+0000 7ff7977fe700 1 --2- 172.21.15.2:0/3473697270 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff7800a1260 0x7ff7800a3710 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009c60 tx=0x7ff78c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:58.328 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.327+0000 7ff7957fa700 1 -- 172.21.15.2:0/3473697270 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff78805a730 con 0x7ff798100290 2023-11-16T16:48:58.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:57 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1027834976' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:48:58.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:57 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1702712837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:48:58.584 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.582+0000 7ff79e9ff700 1 -- 172.21.15.2:0/3473697270 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff798061700 con 0x7ff798100290 2023-11-16T16:48:58.585 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.583+0000 7ff7957fa700 1 -- 172.21.15.2:0/3473697270 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10811 (secure 0 0 0) 0x7ff78809add0 con 0x7ff798100290 2023-11-16T16:48:58.585 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:48:58.585 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/1610211225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":1610211225},{"type":"v1","addr":"172.21.15.2:6837","nonce":1610211225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14538,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2583261818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2583261818},{"type":"v1","addr":"172.21.15.2:6839","nonce":2583261818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14550,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3170991923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3170991923},{"type":"v1","addr":"172.21.15.2:6841","nonce":3170991923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:26.349176+0000","modified":"2023-11-16T16:48:05.774072+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24544,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.141:6835/3516869188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3516869188},{"type":"v1","addr":"172.21.15.141:6835","nonce":3516869188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/2527313191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":2527313191},{"type":"v1","addr":"172.21.15.155:6835","nonce":2527313191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24523":{"gid":24523,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1607756751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1607756751},{"type":"v1","addr":"172.21.15.155:6837","nonce":1607756751}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.141:6837/2934891475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6836","nonce":2934891475},{"type":"v1","addr":"172.21.15.141:6837","nonce":2934891475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24544":{"gid":24544,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1158457009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1158457009},{"type":"v1","addr":"172.21.15.155:6839","nonce":1158457009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6839/3916358272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6838","nonce":3916358272},{"type":"v1","addr":"172.21.15.141:6839","nonce":3916358272}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:48:58.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.586+0000 7ff79e9ff700 1 -- 172.21.15.2:0/3473697270 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff7800a1260 msgr2=0x7ff7800a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.586+0000 7ff79e9ff700 1 --2- 172.21.15.2:0/3473697270 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff7800a1260 0x7ff7800a3710 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009c60 tx=0x7ff78c009260 comp rx=0 tx=0).stop 2023-11-16T16:48:58.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.586+0000 7ff79e9ff700 1 -- 172.21.15.2:0/3473697270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff798100290 msgr2=0x7ff7981981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.586+0000 7ff79e9ff700 1 --2- 172.21.15.2:0/3473697270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff798100290 0x7ff7981981d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff7880098e0 tx=0x7ff788004730 comp rx=0 tx=0).stop 2023-11-16T16:48:58.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.586+0000 7ff79e9ff700 1 -- 172.21.15.2:0/3473697270 shutdown_connections 2023-11-16T16:48:58.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.586+0000 7ff79e9ff700 1 --2- 172.21.15.2:0/3473697270 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff7800a1260 0x7ff7800a3710 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.588 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.586+0000 7ff79e9ff700 1 --2- 172.21.15.2:0/3473697270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff798101e10 0x7ff79819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.586+0000 7ff79e9ff700 1 --2- 172.21.15.2:0/3473697270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff798100290 0x7ff7981981d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.586+0000 7ff79e9ff700 1 --2- 172.21.15.2:0/3473697270 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff798101480 0x7ff798198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.586+0000 7ff79e9ff700 1 -- 172.21.15.2:0/3473697270 >> 172.21.15.2:0/3473697270 conn(0x7ff7980fb870 msgr2=0x7ff798106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:58.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.586+0000 7ff79e9ff700 1 -- 172.21.15.2:0/3473697270 shutdown_connections 2023-11-16T16:48:58.589 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.586+0000 7ff79e9ff700 1 -- 172.21.15.2:0/3473697270 wait complete. 2023-11-16T16:48:58.589 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 26 2023-11-16T16:48:58.607 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-11-16T16:47:26.349176+0000', 'modified': '2023-11-16T16:48:05.774072+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24590, 'mds_1': 24544, 'mds_2': 24542}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24506': {'gid': 24506, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.141:6835/3516869188', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6834', 'nonce': 3516869188}, {'type': 'v1', 'addr': '172.21.15.141:6835', 'nonce': 3516869188}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24511': {'gid': 24511, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.155:6835/2527313191', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6834', 'nonce': 2527313191}, {'type': 'v1', 'addr': '172.21.15.155:6835', 'nonce': 2527313191}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24523': {'gid': 24523, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.155:6837/1607756751', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6836', 'nonce': 1607756751}, {'type': 'v1', 'addr': '172.21.15.155:6837', 'nonce': 1607756751}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24542': {'gid': 24542, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.141:6837/2934891475', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6836', 'nonce': 2934891475}, {'type': 'v1', 'addr': '172.21.15.141:6837', 'nonce': 2934891475}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24544': {'gid': 24544, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.155:6839/1158457009', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6838', 'nonce': 1158457009}, {'type': 'v1', 'addr': '172.21.15.155:6839', 'nonce': 1158457009}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24590': {'gid': 24590, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.141:6839/3916358272', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6838', 'nonce': 3916358272}, {'type': 'v1', 'addr': '172.21.15.141:6839', 'nonce': 3916358272}]}, 'join_fscid': -1, 'export_targets': [1, 2], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-16T16:48:58.607 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-11-16T16:48:58.607 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-11-16T16:48:58.813 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.811+0000 7fe8991ff700 1 -- 172.21.15.2:0/292475558 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe894101db0 msgr2=0x7fe894108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.813 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.811+0000 7fe8991ff700 1 --2- 172.21.15.2:0/292475558 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe894101db0 0x7fe894108670 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe88c00b0a0 tx=0x7fe88c00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.811+0000 7fe8991ff700 1 -- 172.21.15.2:0/292475558 shutdown_connections 2023-11-16T16:48:58.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.811+0000 7fe8991ff700 1 --2- 172.21.15.2:0/292475558 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe894101db0 0x7fe894108670 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.811+0000 7fe8991ff700 1 --2- 172.21.15.2:0/292475558 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe894100230 0x7fe894100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.811+0000 7fe8991ff700 1 --2- 172.21.15.2:0/292475558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe894101420 0x7fe894101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.811+0000 7fe8991ff700 1 -- 172.21.15.2:0/292475558 >> 172.21.15.2:0/292475558 conn(0x7fe8940fb810 msgr2=0x7fe8940fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:58.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.811+0000 7fe8991ff700 1 -- 172.21.15.2:0/292475558 shutdown_connections 2023-11-16T16:48:58.814 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.811+0000 7fe8991ff700 1 -- 172.21.15.2:0/292475558 wait complete. 2023-11-16T16:48:58.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.813+0000 7fe8991ff700 1 Processor -- start 2023-11-16T16:48:58.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.813+0000 7fe8991ff700 1 -- start start 2023-11-16T16:48:58.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.813+0000 7fe8991ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe894100230 0x7fe894195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.814+0000 7fe8991ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe894101420 0x7fe8941963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.814+0000 7fe8991ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe894101db0 0x7fe89419d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.814+0000 7fe8991ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89419d9f0 con 0x7fe894100230 2023-11-16T16:48:58.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.814+0000 7fe8991ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89419db30 con 0x7fe894101420 2023-11-16T16:48:58.817 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.814+0000 7fe8991ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89419dcc0 con 0x7fe894101db0 2023-11-16T16:48:58.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.814+0000 7fe892d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe894100230 0x7fe894195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.814+0000 7fe892d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe894100230 0x7fe894195e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44310/0 (socket says 172.21.15.2:44310) 2023-11-16T16:48:58.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.814+0000 7fe892d9d700 1 -- 172.21.15.2:0/1413661895 learned_addr learned my addr 172.21.15.2:0/1413661895 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:58.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.814+0000 7fe89259c700 1 --2- 172.21.15.2:0/1413661895 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe894101420 0x7fe8941963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.814+0000 7fe89359e700 1 --2- 172.21.15.2:0/1413661895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe894101db0 0x7fe89419d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.814+0000 7fe892d9d700 1 -- 172.21.15.2:0/1413661895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe894101db0 msgr2=0x7fe89419d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.814+0000 7fe892d9d700 1 --2- 172.21.15.2:0/1413661895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe894101db0 0x7fe89419d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.814+0000 7fe892d9d700 1 -- 172.21.15.2:0/1413661895 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe894101420 msgr2=0x7fe8941963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.814+0000 7fe892d9d700 1 --2- 172.21.15.2:0/1413661895 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe894101420 0x7fe8941963b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.818 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.814+0000 7fe892d9d700 1 -- 172.21.15.2:0/1413661895 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe888009520 con 0x7fe894100230 2023-11-16T16:48:58.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.814+0000 7fe89259c700 1 --2- 172.21.15.2:0/1413661895 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe894101420 0x7fe8941963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:58.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.815+0000 7fe892d9d700 1 --2- 172.21.15.2:0/1413661895 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe894100230 0x7fe894195e70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe88400e970 tx=0x7fe88400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:58.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.815+0000 7fe87bfff700 1 -- 172.21.15.2:0/1413661895 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884009790 con 0x7fe894100230 2023-11-16T16:48:58.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.815+0000 7fe87bfff700 1 -- 172.21.15.2:0/1413661895 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe88400c7e0 con 0x7fe894100230 2023-11-16T16:48:58.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.815+0000 7fe8991ff700 1 -- 172.21.15.2:0/1413661895 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe88c009d00 con 0x7fe894100230 2023-11-16T16:48:58.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.815+0000 7fe87bfff700 1 -- 172.21.15.2:0/1413661895 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884003c70 con 0x7fe894100230 2023-11-16T16:48:58.819 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.815+0000 7fe8991ff700 1 -- 172.21.15.2:0/1413661895 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89410bb40 con 0x7fe894100230 2023-11-16T16:48:58.820 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.818+0000 7fe87bfff700 1 -- 172.21.15.2:0/1413661895 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7fe88400cc50 con 0x7fe894100230 2023-11-16T16:48:58.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.819+0000 7fe87bfff700 1 --2- 172.21.15.2:0/1413661895 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe87c0a1260 0x7fe87c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.819+0000 7fe8acb1ab80 1 Processor -- start 2023-11-16T16:48:58.821 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.819+0000 7fe87bfff700 1 -- 172.21.15.2:0/1413661895 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fe884014070 con 0x7fe894100230 2023-11-16T16:48:58.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.820+0000 7fe89259c700 1 --2- 172.21.15.2:0/1413661895 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe87c0a1260 0x7fe87c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.822 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.820+0000 7fe8acb1ab80 1 -- start start 2023-11-16T16:48:58.824 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.820+0000 7fe8acb1ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 0x558f39b3cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.820+0000 7fe8acb1ab80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3d500 0x558f39b3dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.820+0000 7fe8acb1ab80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3e0c0 0x558f39b3e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.820+0000 7fe8acb1ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f39a42010 con 0x558f39b3cbb0 2023-11-16T16:48:58.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.820+0000 7fe8acb1ab80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f39a94850 con 0x558f39b3d500 2023-11-16T16:48:58.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.820+0000 7fe8acb1ab80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f39a8dce0 con 0x558f39b3e0c0 2023-11-16T16:48:58.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.820+0000 7fe89259c700 1 --2- 172.21.15.2:0/1413661895 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe87c0a1260 0x7fe87c0a3710 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe888009fd0 tx=0x7fe888009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:58.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.820+0000 7fe892d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 0x558f39b3cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.825 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.820+0000 7fe892d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 0x558f39b3cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44312/0 (socket says 172.21.15.2:44312) 2023-11-16T16:48:58.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.820+0000 7fe892d9d700 1 -- 172.21.15.2:0/697143690 learned_addr learned my addr 172.21.15.2:0/697143690 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:58.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.821+0000 7fe89259c700 1 --2- 172.21.15.2:0/697143690 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3d500 0x558f39b3dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.821+0000 7fe89359e700 1 --2- 172.21.15.2:0/697143690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3e0c0 0x558f39b3e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.821+0000 7fe892d9d700 1 -- 172.21.15.2:0/697143690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3e0c0 msgr2=0x558f39b3e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:58.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.821+0000 7fe892d9d700 1 --2- 172.21.15.2:0/697143690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3e0c0 0x558f39b3e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.821+0000 7fe892d9d700 1 -- 172.21.15.2:0/697143690 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3d500 msgr2=0x558f39b3dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:58.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.821+0000 7fe892d9d700 1 --2- 172.21.15.2:0/697143690 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3d500 0x558f39b3dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.821+0000 7fe892d9d700 1 -- 172.21.15.2:0/697143690 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f3990f5e0 con 0x558f39b3cbb0 2023-11-16T16:48:58.826 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.821+0000 7fe89259c700 1 --2- 172.21.15.2:0/697143690 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3d500 0x558f39b3dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:58.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.821+0000 7fe892d9d700 1 --2- 172.21.15.2:0/697143690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 0x558f39b3cfc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe884096390 tx=0x7fe884099d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e6f2e5390d975b0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:58.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.821+0000 7fe8797fa700 1 -- 172.21.15.2:0/697143690 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88409dec0 con 0x558f39b3cbb0 2023-11-16T16:48:58.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.821+0000 7fe8797fa700 1 -- 172.21.15.2:0/697143690 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe8840a22e0 con 0x558f39b3cbb0 2023-11-16T16:48:58.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.821+0000 7fe8797fa700 1 -- 172.21.15.2:0/697143690 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88409a2f0 con 0x558f39b3cbb0 2023-11-16T16:48:58.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.822+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/697143690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 msgr2=0x558f39b3cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.822+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/697143690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 0x558f39b3cfc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe884096390 tx=0x7fe884099d90 comp rx=0 tx=0).stop 2023-11-16T16:48:58.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.822+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/697143690 shutdown_connections 2023-11-16T16:48:58.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.822+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/697143690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3e0c0 0x558f39b3e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.822+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/697143690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 0x558f39b3cfc0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.822+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/697143690 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3d500 0x558f39b3dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.822+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/697143690 >> 172.21.15.2:0/697143690 conn(0x558f39b32310 msgr2=0x558f39b37620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:58.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.822+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/697143690 shutdown_connections 2023-11-16T16:48:58.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.822+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/697143690 wait complete. 2023-11-16T16:48:58.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.823+0000 7fe8acb1ab80 1 -- start start 2023-11-16T16:48:58.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.823+0000 7fe8acb1ab80 1 Processor -- start 2023-11-16T16:48:58.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.824+0000 7fe8acb1ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 0x558f39b46590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.824+0000 7fe8acb1ab80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3d500 0x558f39b46ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.824+0000 7fe8acb1ab80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3e0c0 0x558f39b4db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.824+0000 7fe8acb1ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f39a2f890 con 0x558f39b3cbb0 2023-11-16T16:48:58.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.824+0000 7fe8acb1ab80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f39a2f530 con 0x558f39b3e0c0 2023-11-16T16:48:58.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.824+0000 7fe8acb1ab80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f399118d0 con 0x558f39b3d500 2023-11-16T16:48:58.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.824+0000 7fe8acb1ab80 10 client.0 _ll_register_callbacks cb 0x558f39a4d730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:48:58.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.824+0000 7fe892d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 0x558f39b46590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.824+0000 7fe892d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 0x558f39b46590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44326/0 (socket says 172.21.15.2:44326) 2023-11-16T16:48:58.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.824+0000 7fe892d9d700 1 -- 172.21.15.2:0/1973535639 learned_addr learned my addr 172.21.15.2:0/1973535639 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:58.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.824+0000 7fe89359e700 1 --2- 172.21.15.2:0/1973535639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3e0c0 0x558f39b4db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.824+0000 7fe89259c700 1 --2- 172.21.15.2:0/1973535639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3d500 0x558f39b46ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.824+0000 7fe892d9d700 1 -- 172.21.15.2:0/1973535639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3d500 msgr2=0x558f39b46ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:58.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.824+0000 7fe892d9d700 1 --2- 172.21.15.2:0/1973535639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3d500 0x558f39b46ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.824+0000 7fe892d9d700 1 -- 172.21.15.2:0/1973535639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3e0c0 msgr2=0x558f39b4db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:58.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.824+0000 7fe892d9d700 1 --2- 172.21.15.2:0/1973535639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3e0c0 0x558f39b4db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.824+0000 7fe892d9d700 1 -- 172.21.15.2:0/1973535639 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89410bb40 con 0x558f39b3cbb0 2023-11-16T16:48:58.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.825+0000 7fe892d9d700 1 --2- 172.21.15.2:0/1973535639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 0x558f39b46590 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe884099b90 tx=0x7fe88409a010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=357e873335c3298f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:58.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.825+0000 7fe876ff5700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:48:58.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.825+0000 7fe876ff5700 1 -- 172.21.15.2:0/1973535639 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8840c8070 con 0x558f39b3cbb0 2023-11-16T16:48:58.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.825+0000 7fe876ff5700 1 -- 172.21.15.2:0/1973535639 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fe88409dec0 con 0x558f39b3cbb0 2023-11-16T16:48:58.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.825+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/1973535639 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558f39b2fad0 con 0x558f39b3cbb0 2023-11-16T16:48:58.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.825+0000 7fe876ff5700 1 -- 172.21.15.2:0/1973535639 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8840ca2b0 con 0x558f39b3cbb0 2023-11-16T16:48:58.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.825+0000 7fe8acb1ab80 10 client.15126 fetch_fsmap learned FSMap version 26 2023-11-16T16:48:58.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.825+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/1973535639 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x558f39a54140 con 0x558f39b3cbb0 2023-11-16T16:48:58.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.825+0000 7fe876ff5700 1 -- 172.21.15.2:0/1973535639 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558f39b2fad0 con 0x558f39b3cbb0 2023-11-16T16:48:58.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.826+0000 7fe876ff5700 1 -- 172.21.15.2:0/1973535639 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fe8840caba0 con 0x558f39b3cbb0 2023-11-16T16:48:58.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.826+0000 7fe8acb1ab80 10 client.15126 fetch_fsmap finished waiting for FSMap version 26 2023-11-16T16:48:58.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.826+0000 7fe8acb1ab80 10 client.15126 resolve_mds: resolved name 'h' to daemon mds.h 2023-11-16T16:48:58.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.826+0000 7fe8acb1ab80 20 client.15126 populate_metadata read hostname 'smithi002' 2023-11-16T16:48:58.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.826+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/1973535639 >> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] conn(0x558f39b513f0 0x558f39b50a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.826+0000 7fe8acb1ab80 4 client.15126 mds_command: new command op to 24542 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:48:58.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.826+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/1973535639 --> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558f39a50050 con 0x558f39b513f0 2023-11-16T16:48:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.826+0000 7fe89259c700 1 --2- 172.21.15.2:0/1973535639 >> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] conn(0x558f39b513f0 0x558f39b50a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.827+0000 7fe876ff5700 10 client.15126 ms_handle_connect on v2:172.21.15.141:6836/2934891475 2023-11-16T16:48:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.827+0000 7fe89259c700 1 --2- 172.21.15.2:0/1973535639 >> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] conn(0x558f39b513f0 0x558f39b50a40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=987e5a113c9f6586 server_cookie=cbc93d70e29ff730 in_seq=0 out_seq=0 2023-11-16T16:48:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.828+0000 7fe876ff5700 1 -- 172.21.15.2:0/1973535639 <== mds.2 v2:172.21.15.141:6836/2934891475 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x558f39a50050 con 0x558f39b513f0 2023-11-16T16:48:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.828+0000 7fe876ff5700 10 client.15126 handle_command_reply: tid=0 2023-11-16T16:48:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.828+0000 7fe8acb1ab80 1 client.15126 shutdown 2023-11-16T16:48:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.828+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/1973535639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 msgr2=0x558f39b46590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.828+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/1973535639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 0x558f39b46590 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe884099b90 tx=0x7fe88409a010 comp rx=0 tx=0).stop 2023-11-16T16:48:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.828+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/1973535639 shutdown_connections 2023-11-16T16:48:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.828+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/1973535639 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3e0c0 0x558f39b4db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.828+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/1973535639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 0x558f39b46590 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.828+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/1973535639 >> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] conn(0x558f39b513f0 0x558f39b50a40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.828+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/1973535639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3d500 0x558f39b46ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.828+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/1973535639 >> 172.21.15.2:0/1973535639 conn(0x558f39b32310 msgr2=0x558f39b36d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:58.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.828+0000 7fe876ff5700 0 client.15126 ms_handle_reset on v2:172.21.15.141:6836/2934891475 2023-11-16T16:48:58.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.828+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/1973535639 shutdown_connections 2023-11-16T16:48:58.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.829+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/1973535639 wait complete. 2023-11-16T16:48:58.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.829+0000 7fe8acb1ab80 20 client.15126 trim_cache size 0 max 16384 2023-11-16T16:48:58.851 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.849+0000 7fe8acb1ab80 1 Processor -- start 2023-11-16T16:48:58.851 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.849+0000 7fe8acb1ab80 1 -- start start 2023-11-16T16:48:58.852 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.850+0000 7fe8acb1ab80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3cbb0 0x558f39b50a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.850+0000 7fe8acb1ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3d500 0x558f39b4dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.850+0000 7fe8acb1ab80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3e0c0 0x558f39b452a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.850+0000 7fe8acb1ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f39b31f00 con 0x558f39b3d500 2023-11-16T16:48:58.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.850+0000 7fe8acb1ab80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f39a50050 con 0x558f39b3e0c0 2023-11-16T16:48:58.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.850+0000 7fe8acb1ab80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f39b47460 con 0x558f39b3cbb0 2023-11-16T16:48:58.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.850+0000 7fe89259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3d500 0x558f39b4dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.850+0000 7fe89259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3d500 0x558f39b4dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44340/0 (socket says 172.21.15.2:44340) 2023-11-16T16:48:58.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.850+0000 7fe89259c700 1 -- 172.21.15.2:0/2320868316 learned_addr learned my addr 172.21.15.2:0/2320868316 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:58.853 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.850+0000 7fe892d9d700 1 --2- 172.21.15.2:0/2320868316 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3cbb0 0x558f39b50a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.850+0000 7fe89359e700 1 --2- 172.21.15.2:0/2320868316 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3e0c0 0x558f39b452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.851+0000 7fe89259c700 1 -- 172.21.15.2:0/2320868316 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3cbb0 msgr2=0x558f39b50a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:58.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.851+0000 7fe89259c700 1 --2- 172.21.15.2:0/2320868316 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3cbb0 0x558f39b50a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.851+0000 7fe89259c700 1 -- 172.21.15.2:0/2320868316 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3e0c0 msgr2=0x558f39b452a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:58.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.851+0000 7fe89259c700 1 --2- 172.21.15.2:0/2320868316 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3e0c0 0x558f39b452a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.851+0000 7fe89259c700 1 -- 172.21.15.2:0/2320868316 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f39a53e90 con 0x558f39b3d500 2023-11-16T16:48:58.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.851+0000 7fe89259c700 1 --2- 172.21.15.2:0/2320868316 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3d500 0x558f39b4dad0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe888005950 tx=0x7fe888014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=854e10a9fb1a236c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:58.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.851+0000 7fe879ffb700 1 -- 172.21.15.2:0/2320868316 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe888021040 con 0x558f39b3d500 2023-11-16T16:48:58.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.851+0000 7fe879ffb700 1 -- 172.21.15.2:0/2320868316 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe888022040 con 0x558f39b3d500 2023-11-16T16:48:58.854 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.851+0000 7fe879ffb700 1 -- 172.21.15.2:0/2320868316 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88801cd90 con 0x558f39b3d500 2023-11-16T16:48:58.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.852+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/2320868316 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3d500 msgr2=0x558f39b4dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.852+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/2320868316 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3d500 0x558f39b4dad0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe888005950 tx=0x7fe888014000 comp rx=0 tx=0).stop 2023-11-16T16:48:58.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.852+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/2320868316 shutdown_connections 2023-11-16T16:48:58.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.852+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/2320868316 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3cbb0 0x558f39b50a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.852+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/2320868316 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3e0c0 0x558f39b452a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.852+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/2320868316 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3d500 0x558f39b4dad0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.852+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/2320868316 >> 172.21.15.2:0/2320868316 conn(0x558f39b32310 msgr2=0x558f39b5a180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:58.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.852+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/2320868316 shutdown_connections 2023-11-16T16:48:58.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.852+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/2320868316 wait complete. 2023-11-16T16:48:58.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.852+0000 7fe8acb1ab80 1 -- start start 2023-11-16T16:48:58.859 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.853+0000 7fe8acb1ab80 1 Processor -- start 2023-11-16T16:48:58.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.853+0000 7fe8acb1ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 0x558f39b4a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.853+0000 7fe8acb1ab80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3d500 0x558f39b4ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.853+0000 7fe8acb1ab80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3e0c0 0x558f39ba52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.853+0000 7fe8acb1ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f39b4b530 con 0x558f39b3cbb0 2023-11-16T16:48:58.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.853+0000 7fe8acb1ab80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f39ba57f0 con 0x558f39b3d500 2023-11-16T16:48:58.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.853+0000 7fe8acb1ab80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f39ba5930 con 0x558f39b3e0c0 2023-11-16T16:48:58.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.853+0000 7fe8acb1ab80 10 client.0 _ll_register_callbacks cb 0x7fe88800e3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:48:58.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.853+0000 7fe892d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 0x558f39b4a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.853+0000 7fe892d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 0x558f39b4a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44352/0 (socket says 172.21.15.2:44352) 2023-11-16T16:48:58.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.853+0000 7fe892d9d700 1 -- 172.21.15.2:0/369825787 learned_addr learned my addr 172.21.15.2:0/369825787 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:58.860 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.853+0000 7fe89259c700 1 --2- 172.21.15.2:0/369825787 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3d500 0x558f39b4ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.854+0000 7fe89359e700 1 --2- 172.21.15.2:0/369825787 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3e0c0 0x558f39ba52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.854+0000 7fe892d9d700 1 -- 172.21.15.2:0/369825787 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3e0c0 msgr2=0x558f39ba52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:58.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.854+0000 7fe892d9d700 1 --2- 172.21.15.2:0/369825787 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3e0c0 0x558f39ba52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.854+0000 7fe892d9d700 1 -- 172.21.15.2:0/369825787 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3d500 msgr2=0x558f39b4ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:58.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.854+0000 7fe892d9d700 1 --2- 172.21.15.2:0/369825787 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3d500 0x558f39b4ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.854+0000 7fe892d9d700 1 -- 172.21.15.2:0/369825787 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe88800e5b0 con 0x558f39b3cbb0 2023-11-16T16:48:58.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.854+0000 7fe892d9d700 1 --2- 172.21.15.2:0/369825787 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 0x558f39b4a910 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe8840caf30 tx=0x7fe8840047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22ee54720e1b6615 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:58.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.854+0000 7fe876ff5700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:48:58.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.854+0000 7fe876ff5700 1 -- 172.21.15.2:0/369825787 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884016330 con 0x558f39b3cbb0 2023-11-16T16:48:58.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.854+0000 7fe876ff5700 1 -- 172.21.15.2:0/369825787 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fe88409dec0 con 0x558f39b3cbb0 2023-11-16T16:48:58.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.854+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/369825787 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe8840a22e0 con 0x558f39b3cbb0 2023-11-16T16:48:58.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.854+0000 7fe876ff5700 1 -- 172.21.15.2:0/369825787 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8840ca2b0 con 0x558f39b3cbb0 2023-11-16T16:48:58.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.854+0000 7fe8acb1ab80 10 client.15132 fetch_fsmap learned FSMap version 26 2023-11-16T16:48:58.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.854+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/369825787 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x558f398a4c10 con 0x558f39b3cbb0 2023-11-16T16:48:58.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.855+0000 7fe876ff5700 1 -- 172.21.15.2:0/369825787 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe8840bd020 con 0x558f39b3cbb0 2023-11-16T16:48:58.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.855+0000 7fe876ff5700 1 -- 172.21.15.2:0/369825787 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fe8840b2af0 con 0x558f39b3cbb0 2023-11-16T16:48:58.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.855+0000 7fe8acb1ab80 10 client.15132 fetch_fsmap finished waiting for FSMap version 26 2023-11-16T16:48:58.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.855+0000 7fe8acb1ab80 10 client.15132 resolve_mds: resolved name 'h' to daemon mds.h 2023-11-16T16:48:58.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.855+0000 7fe8acb1ab80 20 client.15132 populate_metadata read hostname 'smithi002' 2023-11-16T16:48:58.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.855+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/369825787 >> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] conn(0x558f39baa1b0 0x558f39ba6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:58.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.855+0000 7fe8acb1ab80 4 client.15132 mds_command: new command op to 24542 tid=0[{"prefix": "status"}] 2023-11-16T16:48:58.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.855+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/369825787 --> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558f39a5c610 con 0x558f39baa1b0 2023-11-16T16:48:58.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.855+0000 7fe89259c700 1 --2- 172.21.15.2:0/369825787 >> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] conn(0x558f39baa1b0 0x558f39ba6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:58.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.856+0000 7fe876ff5700 10 client.15132 ms_handle_connect on v2:172.21.15.141:6836/2934891475 2023-11-16T16:48:58.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.856+0000 7fe89259c700 1 --2- 172.21.15.2:0/369825787 >> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] conn(0x558f39baa1b0 0x558f39ba6b80 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5a71fdb179c68d2e server_cookie=3c35bc03e6bf2637 in_seq=0 out_seq=0 2023-11-16T16:48:58.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.856+0000 7fe876ff5700 1 -- 172.21.15.2:0/369825787 <== mds.2 v2:172.21.15.141:6836/2934891475 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x558f39a5c610 con 0x558f39baa1b0 2023-11-16T16:48:58.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.856+0000 7fe876ff5700 10 client.15132 handle_command_reply: tid=0 2023-11-16T16:48:58.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.856+0000 7fe8acb1ab80 1 client.15132 shutdown 2023-11-16T16:48:58.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.856+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/369825787 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 msgr2=0x558f39b4a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.856+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/369825787 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 0x558f39b4a910 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe8840caf30 tx=0x7fe8840047c0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.856+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/369825787 shutdown_connections 2023-11-16T16:48:58.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.856+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/369825787 >> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] conn(0x558f39baa1b0 0x558f39ba6b80 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.863 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.856+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/369825787 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558f39b3e0c0 0x558f39ba52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.856+0000 7fe876ff5700 0 client.15132 ms_handle_reset on v2:172.21.15.141:6836/2934891475 2023-11-16T16:48:58.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.856+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/369825787 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558f39b3cbb0 0x558f39b4a910 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.856+0000 7fe8acb1ab80 1 --2- 172.21.15.2:0/369825787 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x558f39b3d500 0x558f39b4ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.856+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/369825787 >> 172.21.15.2:0/369825787 conn(0x558f39b32310 msgr2=0x558f39b50a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:58.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.856+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/369825787 shutdown_connections 2023-11-16T16:48:58.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.857+0000 7fe8acb1ab80 1 -- 172.21.15.2:0/369825787 wait complete. 2023-11-16T16:48:58.864 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.857+0000 7fe8acb1ab80 20 client.15132 trim_cache size 0 max 16384 2023-11-16T16:48:58.864 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T16:48:58.865 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "875fcd24-849e-11ee-8dbe-212e2dc638e7", 2023-11-16T16:48:58.865 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 2, 2023-11-16T16:48:58.865 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24542, 2023-11-16T16:48:58.865 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2023-11-16T16:48:58.865 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2023-11-16T16:48:58.865 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2023-11-16T16:48:58.865 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 66.960312768999998, 2023-11-16T16:48:58.865 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 26, 2023-11-16T16:48:58.865 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 93, 2023-11-16T16:48:58.865 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 91, 2023-11-16T16:48:58.866 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 109.42099492299999 2023-11-16T16:48:58.866 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T16:48:58.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.857+0000 7fe8991ff700 1 -- 172.21.15.2:0/1413661895 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe87c0a1260 msgr2=0x7fe87c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.866 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.857+0000 7fe8991ff700 1 --2- 172.21.15.2:0/1413661895 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe87c0a1260 0x7fe87c0a3710 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe888009fd0 tx=0x7fe888009260 comp rx=0 tx=0).stop 2023-11-16T16:48:58.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.857+0000 7fe8991ff700 1 -- 172.21.15.2:0/1413661895 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe894100230 msgr2=0x7fe894195e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:58.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.857+0000 7fe8991ff700 1 --2- 172.21.15.2:0/1413661895 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe894100230 0x7fe894195e70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe88400e970 tx=0x7fe88400ec80 comp rx=0 tx=0).stop 2023-11-16T16:48:58.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.857+0000 7fe8991ff700 1 -- 172.21.15.2:0/1413661895 shutdown_connections 2023-11-16T16:48:58.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.857+0000 7fe8991ff700 1 --2- 172.21.15.2:0/1413661895 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fe87c0a1260 0x7fe87c0a3710 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.857+0000 7fe8991ff700 1 --2- 172.21.15.2:0/1413661895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe894101db0 0x7fe89419d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.857+0000 7fe8991ff700 1 --2- 172.21.15.2:0/1413661895 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe894100230 0x7fe894195e70 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.857+0000 7fe8991ff700 1 --2- 172.21.15.2:0/1413661895 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe894101420 0x7fe8941963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:58.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.857+0000 7fe8991ff700 1 -- 172.21.15.2:0/1413661895 >> 172.21.15.2:0/1413661895 conn(0x7fe8940fb810 msgr2=0x7fe8940fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:58.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.857+0000 7fe8991ff700 1 -- 172.21.15.2:0/1413661895 shutdown_connections 2023-11-16T16:48:58.867 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:58.858+0000 7fe8991ff700 1 -- 172.21.15.2:0/1413661895 wait complete. 2023-11-16T16:48:58.878 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-11-16T16:48:59.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.082+0000 7f9a945ff700 1 -- 172.21.15.2:0/356064821 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a8c073550 msgr2=0x7f9a8c0739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:59.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.082+0000 7f9a945ff700 1 --2- 172.21.15.2:0/356064821 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a8c073550 0x7f9a8c0739c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c009990 tx=0x7f9a7c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.082+0000 7f9a945ff700 1 -- 172.21.15.2:0/356064821 shutdown_connections 2023-11-16T16:48:59.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.082+0000 7f9a945ff700 1 --2- 172.21.15.2:0/356064821 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a8c108100 0x7f9a8c10a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.082+0000 7f9a945ff700 1 --2- 172.21.15.2:0/356064821 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a8c074b30 0x7f9a8c072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.082+0000 7f9a945ff700 1 --2- 172.21.15.2:0/356064821 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a8c073550 0x7f9a8c0739c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.082+0000 7f9a945ff700 1 -- 172.21.15.2:0/356064821 >> 172.21.15.2:0/356064821 conn(0x7f9a8c0fb820 msgr2=0x7f9a8c0fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:59.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.082+0000 7f9a945ff700 1 -- 172.21.15.2:0/356064821 shutdown_connections 2023-11-16T16:48:59.084 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.082+0000 7f9a945ff700 1 -- 172.21.15.2:0/356064821 wait complete. 2023-11-16T16:48:59.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.083+0000 7f9a945ff700 1 Processor -- start 2023-11-16T16:48:59.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.083+0000 7f9a945ff700 1 -- start start 2023-11-16T16:48:59.086 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.083+0000 7f9a945ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a8c073550 0x7f9a8c1981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.083+0000 7f9a945ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a8c074b30 0x7f9a8c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.083+0000 7f9a945ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a8c108100 0x7f9a8c19f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.083+0000 7f9a945ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c19fd60 con 0x7f9a8c074b30 2023-11-16T16:48:59.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.083+0000 7f9a945ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c19fea0 con 0x7f9a8c073550 2023-11-16T16:48:59.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.083+0000 7f9a945ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c1a0030 con 0x7f9a8c108100 2023-11-16T16:48:59.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.084+0000 7f9a91b9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a8c074b30 0x7f9a8c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.084+0000 7f9a9239b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a8c073550 0x7f9a8c1981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.084+0000 7f9a9239b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a8c073550 0x7f9a8c1981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:35880/0 (socket says 172.21.15.2:35880) 2023-11-16T16:48:59.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.084+0000 7f9a9239b700 1 -- 172.21.15.2:0/34349480 learned_addr learned my addr 172.21.15.2:0/34349480 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:59.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.084+0000 7f9a92b9c700 1 --2- 172.21.15.2:0/34349480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a8c108100 0x7f9a8c19f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.084+0000 7f9a9239b700 1 -- 172.21.15.2:0/34349480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a8c108100 msgr2=0x7f9a8c19f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:59.087 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.084+0000 7f9a9239b700 1 --2- 172.21.15.2:0/34349480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a8c108100 0x7f9a8c19f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.084+0000 7f9a9239b700 1 -- 172.21.15.2:0/34349480 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a8c074b30 msgr2=0x7f9a8c198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:59.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.084+0000 7f9a9239b700 1 --2- 172.21.15.2:0/34349480 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a8c074b30 0x7f9a8c198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.084+0000 7f9a9239b700 1 -- 172.21.15.2:0/34349480 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c0095f0 con 0x7f9a8c073550 2023-11-16T16:48:59.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.084+0000 7f9a9239b700 1 --2- 172.21.15.2:0/34349480 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a8c073550 0x7f9a8c1981e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800e970 tx=0x7f9a8800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:59.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.085+0000 7f9a837fe700 1 -- 172.21.15.2:0/34349480 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a88009790 con 0x7f9a8c073550 2023-11-16T16:48:59.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.085+0000 7f9a945ff700 1 -- 172.21.15.2:0/34349480 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a8c10de20 con 0x7f9a8c073550 2023-11-16T16:48:59.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.085+0000 7f9a837fe700 1 -- 172.21.15.2:0/34349480 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a8800eec0 con 0x7f9a8c073550 2023-11-16T16:48:59.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.085+0000 7f9a837fe700 1 -- 172.21.15.2:0/34349480 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a88005590 con 0x7f9a8c073550 2023-11-16T16:48:59.088 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.085+0000 7f9a945ff700 1 -- 172.21.15.2:0/34349480 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a8c10dc90 con 0x7f9a8c073550 2023-11-16T16:48:59.089 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.088+0000 7f9a837fe700 1 -- 172.21.15.2:0/34349480 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f9a8800cbd0 con 0x7f9a8c073550 2023-11-16T16:48:59.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.089+0000 7f9aa7fc7b80 1 Processor -- start 2023-11-16T16:48:59.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.089+0000 7f9aa7fc7b80 1 -- start start 2023-11-16T16:48:59.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.089+0000 7f9aa7fc7b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f3bb0 0x5559303f3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.089+0000 7f9aa7fc7b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f4500 0x5559303f4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.089+0000 7f9aa7fc7b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f50c0 0x5559303f5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.089+0000 7f9aa7fc7b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559302f9010 con 0x5559303f4500 2023-11-16T16:48:59.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.089+0000 7f9aa7fc7b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55593034b850 con 0x5559303f50c0 2023-11-16T16:48:59.091 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.089+0000 7f9aa7fc7b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x555930344ce0 con 0x5559303f3bb0 2023-11-16T16:48:59.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.089+0000 7f9a837fe700 1 --2- 172.21.15.2:0/34349480 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9a780a11e0 0x7f9a780a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.090+0000 7f9a9239b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f4500 0x5559303f4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.090+0000 7f9a92b9c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f3bb0 0x5559303f3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.090+0000 7f9a92b9c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f3bb0 0x5559303f3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:58756/0 (socket says 172.21.15.2:58756) 2023-11-16T16:48:59.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.090+0000 7f9a92b9c700 1 -- 172.21.15.2:0/3230886155 learned_addr learned my addr 172.21.15.2:0/3230886155 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:59.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.090+0000 7f9a91b9a700 1 --2- 172.21.15.2:0/3230886155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f50c0 0x5559303f5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.090+0000 7f9a92b9c700 1 --2- 172.21.15.2:0/34349480 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9a780a11e0 0x7f9a780a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.090+0000 7f9a837fe700 1 -- 172.21.15.2:0/34349480 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f9a88014070 con 0x7f9a8c073550 2023-11-16T16:48:59.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.090+0000 7f9a91b9a700 1 -- 172.21.15.2:0/3230886155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f3bb0 msgr2=0x5559303f3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:59.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.090+0000 7f9a91b9a700 1 --2- 172.21.15.2:0/3230886155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f3bb0 0x5559303f3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.090+0000 7f9a91b9a700 1 -- 172.21.15.2:0/3230886155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f4500 msgr2=0x5559303f4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:59.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.090+0000 7f9a91b9a700 1 --2- 172.21.15.2:0/3230886155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f4500 0x5559303f4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.090+0000 7f9a91b9a700 1 -- 172.21.15.2:0/3230886155 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559301c65e0 con 0x5559303f50c0 2023-11-16T16:48:59.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.091+0000 7f9a92b9c700 1 --2- 172.21.15.2:0/34349480 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9a780a11e0 0x7f9a780a3690 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9a84012f00 tx=0x7f9a8400b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:59.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.091+0000 7f9a91b9a700 1 --2- 172.21.15.2:0/3230886155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f50c0 0x5559303f5530 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c00b680 tx=0x7f9a7c005ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19d9b599e0df7ae server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:59.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.091+0000 7f9a80bfe700 1 -- 172.21.15.2:0/3230886155 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c01d070 con 0x5559303f50c0 2023-11-16T16:48:59.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.091+0000 7f9a80bfe700 1 -- 172.21.15.2:0/3230886155 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a7c01e040 con 0x5559303f50c0 2023-11-16T16:48:59.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.091+0000 7f9a80bfe700 1 -- 172.21.15.2:0/3230886155 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c004900 con 0x5559303f50c0 2023-11-16T16:48:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.091+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/3230886155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f50c0 msgr2=0x5559303f5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.091+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/3230886155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f50c0 0x5559303f5530 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c00b680 tx=0x7f9a7c005ea0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.092+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/3230886155 shutdown_connections 2023-11-16T16:48:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.092+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/3230886155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f3bb0 0x5559303f3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.092+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/3230886155 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f50c0 0x5559303f5530 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.092+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/3230886155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f4500 0x5559303f4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.092+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/3230886155 >> 172.21.15.2:0/3230886155 conn(0x5559303e9310 msgr2=0x5559303ee620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.092+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/3230886155 shutdown_connections 2023-11-16T16:48:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.092+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/3230886155 wait complete. 2023-11-16T16:48:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.092+0000 7f9aa7fc7b80 1 -- start start 2023-11-16T16:48:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.092+0000 7f9aa7fc7b80 1 Processor -- start 2023-11-16T16:48:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.093+0000 7f9aa7fc7b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f3bb0 0x5559303fd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.093+0000 7f9aa7fc7b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f4500 0x5559303fd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.093+0000 7f9aa7fc7b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f50c0 0x555930404a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.093+0000 7f9aa7fc7b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559302e6890 con 0x5559303f4500 2023-11-16T16:48:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.093+0000 7f9aa7fc7b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559302e6530 con 0x5559303f50c0 2023-11-16T16:48:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.093+0000 7f9aa7fc7b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559301c88d0 con 0x5559303f3bb0 2023-11-16T16:48:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.093+0000 7f9aa7fc7b80 10 client.0 _ll_register_callbacks cb 0x555930304730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:48:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.093+0000 7f9a91b9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f4500 0x5559303fd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.093+0000 7f9a92b9c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f50c0 0x555930404a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.093+0000 7f9a9239b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f3bb0 0x5559303fd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.093+0000 7f9a92b9c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f50c0 0x555930404a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:35892/0 (socket says 172.21.15.2:35892) 2023-11-16T16:48:59.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.093+0000 7f9a92b9c700 1 -- 172.21.15.2:0/3988802914 learned_addr learned my addr 172.21.15.2:0/3988802914 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:59.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.094+0000 7f9a92b9c700 1 -- 172.21.15.2:0/3988802914 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f3bb0 msgr2=0x5559303fd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:59.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.094+0000 7f9a92b9c700 1 --2- 172.21.15.2:0/3988802914 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f3bb0 0x5559303fd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.094+0000 7f9a92b9c700 1 -- 172.21.15.2:0/3988802914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f4500 msgr2=0x5559303fd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:59.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.094+0000 7f9a92b9c700 1 --2- 172.21.15.2:0/3988802914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f4500 0x5559303fd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.094+0000 7f9a92b9c700 1 -- 172.21.15.2:0/3988802914 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a8c10b1a0 con 0x5559303f50c0 2023-11-16T16:48:59.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.094+0000 7f9a92b9c700 1 --2- 172.21.15.2:0/3988802914 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f50c0 0x555930404a20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9a84012790 tx=0x7f9a84004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e499cc64c8103f2c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:59.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.094+0000 7f9a75ffb700 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2023-11-16T16:48:59.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.094+0000 7f9a75ffb700 1 -- 172.21.15.2:0/3988802914 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a84028070 con 0x5559303f50c0 2023-11-16T16:48:59.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.094+0000 7f9a75ffb700 1 -- 172.21.15.2:0/3988802914 <== mon.2 v2:172.21.15.141:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f9a84007e00 con 0x5559303f50c0 2023-11-16T16:48:59.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.094+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/3988802914 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5559303e6ad0 con 0x5559303f50c0 2023-11-16T16:48:59.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.095+0000 7f9a75ffb700 1 -- 172.21.15.2:0/3988802914 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a8400e040 con 0x5559303f50c0 2023-11-16T16:48:59.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.095+0000 7f9aa7fc7b80 10 client.25175 fetch_fsmap learned FSMap version 26 2023-11-16T16:48:59.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.095+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/3988802914 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55593030b140 con 0x5559303f50c0 2023-11-16T16:48:59.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.095+0000 7f9a75ffb700 1 -- 172.21.15.2:0/3988802914 <== mon.2 v2:172.21.15.141:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5559303e6ad0 con 0x5559303f50c0 2023-11-16T16:48:59.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.095+0000 7f9a75ffb700 1 -- 172.21.15.2:0/3988802914 <== mon.2 v2:172.21.15.141:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f9a8402b030 con 0x5559303f50c0 2023-11-16T16:48:59.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.095+0000 7f9aa7fc7b80 10 client.25175 fetch_fsmap finished waiting for FSMap version 26 2023-11-16T16:48:59.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.095+0000 7f9aa7fc7b80 10 client.25175 resolve_mds: resolved name 'i' to daemon mds.i 2023-11-16T16:48:59.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.095+0000 7f9aa7fc7b80 20 client.25175 populate_metadata read hostname 'smithi002' 2023-11-16T16:48:59.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.095+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/3988802914 >> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] conn(0x555930408760 0x555930407b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.100 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.095+0000 7f9aa7fc7b80 4 client.25175 mds_command: new command op to 24544 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:48:59.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.095+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/3988802914 --> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555930307050 con 0x555930408760 2023-11-16T16:48:59.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.096+0000 7f9a9239b700 1 --2- 172.21.15.2:0/3988802914 >> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] conn(0x555930408760 0x555930407b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.096+0000 7f9a9239b700 1 --2- 172.21.15.2:0/3988802914 >> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] conn(0x555930408760 0x555930407b20 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9a9d2d781fb63879 server_cookie=6eec98b122887030 in_seq=0 out_seq=0 2023-11-16T16:48:59.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.096+0000 7f9a75ffb700 10 client.25175 ms_handle_connect on v2:172.21.15.155:6838/1158457009 2023-11-16T16:48:59.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.097+0000 7f9a75ffb700 1 -- 172.21.15.2:0/3988802914 <== mds.1 v2:172.21.15.155:6838/1158457009 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x555930307050 con 0x555930408760 2023-11-16T16:48:59.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.097+0000 7f9a75ffb700 10 client.25175 handle_command_reply: tid=0 2023-11-16T16:48:59.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.097+0000 7f9aa7fc7b80 1 client.25175 shutdown 2023-11-16T16:48:59.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.097+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/3988802914 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f50c0 msgr2=0x555930404a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:59.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.097+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/3988802914 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f50c0 0x555930404a20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9a84012790 tx=0x7f9a84004ad0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.098+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/3988802914 shutdown_connections 2023-11-16T16:48:59.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.098+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/3988802914 >> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] conn(0x555930408760 0x555930407b20 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.098+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/3988802914 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f3bb0 0x5559303fd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.098+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/3988802914 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f50c0 0x555930404a20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.098+0000 7f9a75ffb700 0 client.25175 ms_handle_reset on v2:172.21.15.155:6838/1158457009 2023-11-16T16:48:59.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.098+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/3988802914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f4500 0x5559303fd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.098+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/3988802914 >> 172.21.15.2:0/3988802914 conn(0x5559303e9310 msgr2=0x5559303edd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:59.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.098+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/3988802914 shutdown_connections 2023-11-16T16:48:59.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.098+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/3988802914 wait complete. 2023-11-16T16:48:59.103 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.098+0000 7f9aa7fc7b80 20 client.25175 trim_cache size 0 max 16384 2023-11-16T16:48:59.119 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.116+0000 7f9aa7fc7b80 1 Processor -- start 2023-11-16T16:48:59.119 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.116+0000 7f9aa7fc7b80 1 -- start start 2023-11-16T16:48:59.119 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.117+0000 7f9aa7fc7b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f3bb0 0x555930411550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.119 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.117+0000 7f9aa7fc7b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f4500 0x5559304049e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.119 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.117+0000 7f9aa7fc7b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f50c0 0x555930407b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.119 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.117+0000 7f9aa7fc7b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559301caa30 con 0x5559303f4500 2023-11-16T16:48:59.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.117+0000 7f9aa7fc7b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559303f4f40 con 0x5559303f3bb0 2023-11-16T16:48:59.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.117+0000 7f9aa7fc7b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559303fd010 con 0x5559303f50c0 2023-11-16T16:48:59.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.117+0000 7f9a91b9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f4500 0x5559304049e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.117+0000 7f9a91b9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f4500 0x5559304049e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44410/0 (socket says 172.21.15.2:44410) 2023-11-16T16:48:59.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.117+0000 7f9a91b9a700 1 -- 172.21.15.2:0/2062904653 learned_addr learned my addr 172.21.15.2:0/2062904653 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:59.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.117+0000 7f9a9239b700 1 --2- 172.21.15.2:0/2062904653 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f3bb0 0x555930411550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.117+0000 7f9a92b9c700 1 --2- 172.21.15.2:0/2062904653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f50c0 0x555930407b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.117+0000 7f9a91b9a700 1 -- 172.21.15.2:0/2062904653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f50c0 msgr2=0x555930407b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:59.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.117+0000 7f9a91b9a700 1 --2- 172.21.15.2:0/2062904653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f50c0 0x555930407b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.117+0000 7f9a91b9a700 1 -- 172.21.15.2:0/2062904653 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f3bb0 msgr2=0x555930411550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:59.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.117+0000 7f9a91b9a700 1 --2- 172.21.15.2:0/2062904653 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f3bb0 0x555930411550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.117+0000 7f9a91b9a700 1 -- 172.21.15.2:0/2062904653 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55593030ae90 con 0x5559303f4500 2023-11-16T16:48:59.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.118+0000 7f9a91b9a700 1 --2- 172.21.15.2:0/2062904653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f4500 0x5559304049e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c000c00 tx=0x7f9a7c0058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e2c4be963a7d531 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:59.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.118+0000 7f9a813ff700 1 -- 172.21.15.2:0/2062904653 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c01d040 con 0x5559303f4500 2023-11-16T16:48:59.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.118+0000 7f9a813ff700 1 -- 172.21.15.2:0/2062904653 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a7c005530 con 0x5559303f4500 2023-11-16T16:48:59.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.118+0000 7f9a813ff700 1 -- 172.21.15.2:0/2062904653 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c004ce0 con 0x5559303f4500 2023-11-16T16:48:59.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.118+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/2062904653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f4500 msgr2=0x5559304049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:59.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.118+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/2062904653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f4500 0x5559304049e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c000c00 tx=0x7f9a7c0058e0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.118+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/2062904653 shutdown_connections 2023-11-16T16:48:59.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.118+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/2062904653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f50c0 0x555930407b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.118+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/2062904653 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f3bb0 0x555930411550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.124 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.118+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/2062904653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f4500 0x5559304049e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.118+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/2062904653 >> 172.21.15.2:0/2062904653 conn(0x5559303e9310 msgr2=0x5559303ee5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:59.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.119+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/2062904653 shutdown_connections 2023-11-16T16:48:59.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.119+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/2062904653 wait complete. 2023-11-16T16:48:59.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.119+0000 7f9aa7fc7b80 1 -- start start 2023-11-16T16:48:59.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.119+0000 7f9aa7fc7b80 1 Processor -- start 2023-11-16T16:48:59.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.120+0000 7f9aa7fc7b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f3bb0 0x5559303ff220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.120+0000 7f9aa7fc7b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f4500 0x555930404790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.120+0000 7f9aa7fc7b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f50c0 0x5559303ff760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.120+0000 7f9aa7fc7b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559303ffce0 con 0x5559303f50c0 2023-11-16T16:48:59.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.120+0000 7f9aa7fc7b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559303ffe50 con 0x5559303f3bb0 2023-11-16T16:48:59.125 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.120+0000 7f9aa7fc7b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55593045c6b0 con 0x5559303f4500 2023-11-16T16:48:59.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.120+0000 7f9aa7fc7b80 10 client.0 _ll_register_callbacks cb 0x7f9a840093e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:48:59.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.120+0000 7f9a92b9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f50c0 0x5559303ff760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.120+0000 7f9a91b9a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f4500 0x555930404790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.120+0000 7f9a9239b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f3bb0 0x5559303ff220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.120+0000 7f9a91b9a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f4500 0x555930404790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:58778/0 (socket says 172.21.15.2:58778) 2023-11-16T16:48:59.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.120+0000 7f9a91b9a700 1 -- 172.21.15.2:0/1548082857 learned_addr learned my addr 172.21.15.2:0/1548082857 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:59.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.120+0000 7f9a9239b700 1 --2- 172.21.15.2:0/1548082857 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f3bb0 0x5559303ff220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:35904/0 (socket says 172.21.15.2:35904) 2023-11-16T16:48:59.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.120+0000 7f9a9239b700 1 -- 172.21.15.2:0/1548082857 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f4500 msgr2=0x555930404790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:59.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.120+0000 7f9a9239b700 1 --2- 172.21.15.2:0/1548082857 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f4500 0x555930404790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.120+0000 7f9a9239b700 1 -- 172.21.15.2:0/1548082857 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f50c0 msgr2=0x5559303ff760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:59.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.120+0000 7f9a9239b700 1 --2- 172.21.15.2:0/1548082857 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f50c0 0x5559303ff760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.126 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.120+0000 7f9a9239b700 1 -- 172.21.15.2:0/1548082857 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5559301c65e0 con 0x5559303f3bb0 2023-11-16T16:48:59.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.120+0000 7f9a91b9a700 1 --2- 172.21.15.2:0/1548082857 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f4500 0x555930404790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:59.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.121+0000 7f9a9239b700 1 --2- 172.21.15.2:0/1548082857 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f3bb0 0x5559303ff220 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9a880ca690 tx=0x7f9a8806c050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d03805785b01a467 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:59.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.121+0000 7f9a75ffb700 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2023-11-16T16:48:59.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.121+0000 7f9a75ffb700 1 -- 172.21.15.2:0/1548082857 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a880caae0 con 0x5559303f3bb0 2023-11-16T16:48:59.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.121+0000 7f9a75ffb700 1 -- 172.21.15.2:0/1548082857 <== mon.2 v2:172.21.15.141:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f9a880690b0 con 0x5559303f3bb0 2023-11-16T16:48:59.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.121+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/1548082857 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9a7c01e040 con 0x5559303f3bb0 2023-11-16T16:48:59.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.121+0000 7f9a75ffb700 1 -- 172.21.15.2:0/1548082857 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a88069270 con 0x5559303f3bb0 2023-11-16T16:48:59.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.121+0000 7f9aa7fc7b80 10 client.25181 fetch_fsmap learned FSMap version 26 2023-11-16T16:48:59.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.121+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/1548082857 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55593015bc10 con 0x5559303f3bb0 2023-11-16T16:48:59.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.121+0000 7f9a75ffb700 1 -- 172.21.15.2:0/1548082857 <== mon.2 v2:172.21.15.141:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a7c01e040 con 0x5559303f3bb0 2023-11-16T16:48:59.127 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.122+0000 7f9a75ffb700 1 -- 172.21.15.2:0/1548082857 <== mon.2 v2:172.21.15.141:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f9a88099480 con 0x5559303f3bb0 2023-11-16T16:48:59.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.122+0000 7f9aa7fc7b80 10 client.25181 fetch_fsmap finished waiting for FSMap version 26 2023-11-16T16:48:59.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.122+0000 7f9aa7fc7b80 10 client.25181 resolve_mds: resolved name 'i' to daemon mds.i 2023-11-16T16:48:59.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.122+0000 7f9aa7fc7b80 20 client.25181 populate_metadata read hostname 'smithi002' 2023-11-16T16:48:59.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.122+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/1548082857 >> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] conn(0x5559304613e0 0x55593045dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.122+0000 7f9aa7fc7b80 4 client.25181 mds_command: new command op to 24544 tid=0[{"prefix": "status"}] 2023-11-16T16:48:59.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.122+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/1548082857 --> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555930313610 con 0x5559304613e0 2023-11-16T16:48:59.128 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.122+0000 7f9a91b9a700 1 --2- 172.21.15.2:0/1548082857 >> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] conn(0x5559304613e0 0x55593045dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.123+0000 7f9a91b9a700 1 --2- 172.21.15.2:0/1548082857 >> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] conn(0x5559304613e0 0x55593045dcb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=296596a0737259a8 server_cookie=98c2225b7fb5292f in_seq=0 out_seq=0 2023-11-16T16:48:59.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.123+0000 7f9a75ffb700 10 client.25181 ms_handle_connect on v2:172.21.15.155:6838/1158457009 2023-11-16T16:48:59.129 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.123+0000 7f9a75ffb700 1 -- 172.21.15.2:0/1548082857 <== mds.1 v2:172.21.15.155:6838/1158457009 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x555930313610 con 0x5559304613e0 2023-11-16T16:48:59.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.123+0000 7f9a75ffb700 10 client.25181 handle_command_reply: tid=0 2023-11-16T16:48:59.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.123+0000 7f9aa7fc7b80 1 client.25181 shutdown 2023-11-16T16:48:59.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.123+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/1548082857 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f3bb0 msgr2=0x5559303ff220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:59.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/1548082857 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f3bb0 0x5559303ff220 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9a880ca690 tx=0x7f9a8806c050 comp rx=0 tx=0).stop 2023-11-16T16:48:59.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/1548082857 shutdown_connections 2023-11-16T16:48:59.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/1548082857 >> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] conn(0x5559304613e0 0x55593045dcb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/1548082857 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5559303f50c0 0x5559303ff760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9a75ffb700 0 client.25181 ms_handle_reset on v2:172.21.15.155:6838/1158457009 2023-11-16T16:48:59.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/1548082857 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5559303f3bb0 0x5559303ff220 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9aa7fc7b80 1 --2- 172.21.15.2:0/1548082857 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5559303f4500 0x555930404790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.130 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/1548082857 >> 172.21.15.2:0/1548082857 conn(0x5559303e9310 msgr2=0x555930407b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:59.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/1548082857 shutdown_connections 2023-11-16T16:48:59.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9aa7fc7b80 1 -- 172.21.15.2:0/1548082857 wait complete. 2023-11-16T16:48:59.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9aa7fc7b80 20 client.25181 trim_cache size 0 max 16384 2023-11-16T16:48:59.131 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T16:48:59.133 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "875fcd24-849e-11ee-8dbe-212e2dc638e7", 2023-11-16T16:48:59.133 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 1, 2023-11-16T16:48:59.133 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24544, 2023-11-16T16:48:59.133 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2023-11-16T16:48:59.133 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2023-11-16T16:48:59.133 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2023-11-16T16:48:59.134 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 68.234423762000006, 2023-11-16T16:48:59.134 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 26, 2023-11-16T16:48:59.134 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 93, 2023-11-16T16:48:59.134 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 91, 2023-11-16T16:48:59.134 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 107.520176545 2023-11-16T16:48:59.134 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T16:48:59.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9a945ff700 1 -- 172.21.15.2:0/34349480 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9a780a11e0 msgr2=0x7f9a780a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:59.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9a945ff700 1 --2- 172.21.15.2:0/34349480 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9a780a11e0 0x7f9a780a3690 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9a84012f00 tx=0x7f9a8400b040 comp rx=0 tx=0).stop 2023-11-16T16:48:59.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9a945ff700 1 -- 172.21.15.2:0/34349480 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a8c073550 msgr2=0x7f9a8c1981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:59.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9a945ff700 1 --2- 172.21.15.2:0/34349480 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a8c073550 0x7f9a8c1981e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800e970 tx=0x7f9a8800ec80 comp rx=0 tx=0).stop 2023-11-16T16:48:59.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9a945ff700 1 -- 172.21.15.2:0/34349480 shutdown_connections 2023-11-16T16:48:59.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9a945ff700 1 --2- 172.21.15.2:0/34349480 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9a780a11e0 0x7f9a780a3690 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9a945ff700 1 --2- 172.21.15.2:0/34349480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a8c108100 0x7f9a8c19f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9a945ff700 1 --2- 172.21.15.2:0/34349480 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9a8c073550 0x7f9a8c1981e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9a945ff700 1 --2- 172.21.15.2:0/34349480 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a8c074b30 0x7f9a8c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.124+0000 7f9a945ff700 1 -- 172.21.15.2:0/34349480 >> 172.21.15.2:0/34349480 conn(0x7f9a8c0fb820 msgr2=0x7f9a8c0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:59.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.125+0000 7f9a945ff700 1 -- 172.21.15.2:0/34349480 shutdown_connections 2023-11-16T16:48:59.136 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.125+0000 7f9a945ff700 1 -- 172.21.15.2:0/34349480 wait complete. 2023-11-16T16:48:59.142 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:58 smithi002 ceph-mon[128712]: pgmap v330: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s; 5.4 KiB/s, 1 keys/s, 0 objects/s recovering 2023-11-16T16:48:59.142 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:58 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3319900235' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:59.142 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:48:58 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3473697270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:48:59.145 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-11-16T16:48:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:58 smithi141 ceph-mon[131184]: pgmap v330: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s; 5.4 KiB/s, 1 keys/s, 0 objects/s recovering 2023-11-16T16:48:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:58 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3319900235' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:48:58 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3473697270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:48:59.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.349+0000 7f31203ff700 1 -- 172.21.15.2:0/3350895141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31181080f0 msgr2=0x7f311810a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:59.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.349+0000 7f31203ff700 1 --2- 172.21.15.2:0/3350895141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31181080f0 0x7f311810a570 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f311000b0a0 tx=0x7f311000b3b0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.349+0000 7f31203ff700 1 -- 172.21.15.2:0/3350895141 shutdown_connections 2023-11-16T16:48:59.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.349+0000 7f31203ff700 1 --2- 172.21.15.2:0/3350895141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31181080f0 0x7f311810a570 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.349+0000 7f31203ff700 1 --2- 172.21.15.2:0/3350895141 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3118074b30 0x7f3118072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.351 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.349+0000 7f31203ff700 1 --2- 172.21.15.2:0/3350895141 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3118073550 0x7f31180739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.349+0000 7f31203ff700 1 -- 172.21.15.2:0/3350895141 >> 172.21.15.2:0/3350895141 conn(0x7f31180fb810 msgr2=0x7f31180fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:59.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.349+0000 7f31203ff700 1 -- 172.21.15.2:0/3350895141 shutdown_connections 2023-11-16T16:48:59.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.349+0000 7f31203ff700 1 -- 172.21.15.2:0/3350895141 wait complete. 2023-11-16T16:48:59.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.350+0000 7f31203ff700 1 Processor -- start 2023-11-16T16:48:59.352 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.351+0000 7f31203ff700 1 -- start start 2023-11-16T16:48:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.351+0000 7f31203ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3118073550 0x7f31181982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.351+0000 7f31203ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3118074b30 0x7f3118198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.351+0000 7f31203ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31181080f0 0x7f311819f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.351+0000 7f31203ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311819fe50 con 0x7f31181080f0 2023-11-16T16:48:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.351+0000 7f31203ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311819ff90 con 0x7f3118073550 2023-11-16T16:48:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.351+0000 7f31203ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31181a0120 con 0x7f3118074b30 2023-11-16T16:48:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.352+0000 7f311e99c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31181080f0 0x7f311819f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.352+0000 7f311e99c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31181080f0 0x7f311819f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44444/0 (socket says 172.21.15.2:44444) 2023-11-16T16:48:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.352+0000 7f311e99c700 1 -- 172.21.15.2:0/693629628 learned_addr learned my addr 172.21.15.2:0/693629628 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:59.356 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.352+0000 7f311e19b700 1 --2- 172.21.15.2:0/693629628 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3118073550 0x7f31181982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.356 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.352+0000 7f311d99a700 1 --2- 172.21.15.2:0/693629628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3118074b30 0x7f3118198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.356 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.352+0000 7f311e99c700 1 -- 172.21.15.2:0/693629628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3118074b30 msgr2=0x7f3118198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:59.356 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.352+0000 7f311e99c700 1 --2- 172.21.15.2:0/693629628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3118074b30 0x7f3118198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.356 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.352+0000 7f311e99c700 1 -- 172.21.15.2:0/693629628 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3118073550 msgr2=0x7f31181982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:59.356 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.352+0000 7f311e99c700 1 --2- 172.21.15.2:0/693629628 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3118073550 0x7f31181982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.356 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.352+0000 7f311e99c700 1 -- 172.21.15.2:0/693629628 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3110009d00 con 0x7f31181080f0 2023-11-16T16:48:59.356 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.352+0000 7f311e19b700 1 --2- 172.21.15.2:0/693629628 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3118073550 0x7f31181982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:59.356 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.352+0000 7f311e99c700 1 --2- 172.21.15.2:0/693629628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31181080f0 0x7f311819f880 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3110009fd0 tx=0x7f311000bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:59.356 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.352+0000 7f310f7fe700 1 -- 172.21.15.2:0/693629628 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311000e040 con 0x7f31181080f0 2023-11-16T16:48:59.357 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.353+0000 7f310f7fe700 1 -- 172.21.15.2:0/693629628 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f31100094a0 con 0x7f31181080f0 2023-11-16T16:48:59.357 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.353+0000 7f310f7fe700 1 -- 172.21.15.2:0/693629628 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311001b6c0 con 0x7f31181080f0 2023-11-16T16:48:59.357 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.353+0000 7f31203ff700 1 -- 172.21.15.2:0/693629628 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311810de10 con 0x7f31181080f0 2023-11-16T16:48:59.357 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.353+0000 7f31203ff700 1 -- 172.21.15.2:0/693629628 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311810dc80 con 0x7f31181080f0 2023-11-16T16:48:59.357 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.356+0000 7f310f7fe700 1 -- 172.21.15.2:0/693629628 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f311001d470 con 0x7f31181080f0 2023-11-16T16:48:59.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.357+0000 7f3133df0b80 1 Processor -- start 2023-11-16T16:48:59.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.357+0000 7f3133df0b80 1 -- start start 2023-11-16T16:48:59.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.357+0000 7f310f7fe700 1 --2- 172.21.15.2:0/693629628 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31040a11e0 0x7f31040a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.358+0000 7f3133df0b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb6bb0 0x563345eb6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.358+0000 7f3133df0b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb7500 0x563345eb7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.358+0000 7f3133df0b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb80c0 0x563345eb8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.358+0000 7f3133df0b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x563345dbc010 con 0x563345eb7500 2023-11-16T16:48:59.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.358+0000 7f3133df0b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x563345e0e850 con 0x563345eb80c0 2023-11-16T16:48:59.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.358+0000 7f3133df0b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x563345e07ce0 con 0x563345eb6bb0 2023-11-16T16:48:59.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.358+0000 7f310f7fe700 1 -- 172.21.15.2:0/693629628 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f31100cdff0 con 0x7f31181080f0 2023-11-16T16:48:59.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.358+0000 7f311d99a700 1 --2- 172.21.15.2:0/693629628 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31040a11e0 0x7f31040a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.358+0000 7f311e19b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb6bb0 0x563345eb6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.358+0000 7f311e19b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb6bb0 0x563345eb6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:58820/0 (socket says 172.21.15.2:58820) 2023-11-16T16:48:59.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.358+0000 7f311e19b700 1 -- 172.21.15.2:0/2559343082 learned_addr learned my addr 172.21.15.2:0/2559343082 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:59.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.358+0000 7f311d99a700 1 --2- 172.21.15.2:0/2559343082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb7500 0x563345eb7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.358+0000 7f311e99c700 1 --2- 172.21.15.2:0/2559343082 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb80c0 0x563345eb8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.359+0000 7f311e19b700 1 -- 172.21.15.2:0/2559343082 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb80c0 msgr2=0x563345eb8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:59.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.359+0000 7f311e19b700 1 --2- 172.21.15.2:0/2559343082 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb80c0 0x563345eb8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.359+0000 7f311e19b700 1 -- 172.21.15.2:0/2559343082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb7500 msgr2=0x563345eb7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:59.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.359+0000 7f311e19b700 1 --2- 172.21.15.2:0/2559343082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb7500 0x563345eb7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.359+0000 7f311e19b700 1 -- 172.21.15.2:0/2559343082 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563345c895e0 con 0x563345eb6bb0 2023-11-16T16:48:59.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.359+0000 7f311e99c700 1 --2- 172.21.15.2:0/2559343082 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb80c0 0x563345eb8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:48:59.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.359+0000 7f311e19b700 1 --2- 172.21.15.2:0/2559343082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb6bb0 0x563345eb6fc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f31140115f0 tx=0x7f3114011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a509e51ac51b7fa3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:59.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.359+0000 7f311d99a700 1 --2- 172.21.15.2:0/693629628 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31040a11e0 0x7f31040a3690 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3118074850 tx=0x7f3108009bf0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:59.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.359+0000 7f310cbfe700 1 -- 172.21.15.2:0/2559343082 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114004230 con 0x563345eb6bb0 2023-11-16T16:48:59.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.359+0000 7f310cbfe700 1 -- 172.21.15.2:0/2559343082 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3114004390 con 0x563345eb6bb0 2023-11-16T16:48:59.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.359+0000 7f310cbfe700 1 -- 172.21.15.2:0/2559343082 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114019800 con 0x563345eb6bb0 2023-11-16T16:48:59.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.360+0000 7f3133df0b80 1 -- 172.21.15.2:0/2559343082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb6bb0 msgr2=0x563345eb6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:59.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.360+0000 7f3133df0b80 1 --2- 172.21.15.2:0/2559343082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb6bb0 0x563345eb6fc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f31140115f0 tx=0x7f3114011900 comp rx=0 tx=0).stop 2023-11-16T16:48:59.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.360+0000 7f3133df0b80 1 -- 172.21.15.2:0/2559343082 shutdown_connections 2023-11-16T16:48:59.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.360+0000 7f3133df0b80 1 --2- 172.21.15.2:0/2559343082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb6bb0 0x563345eb6fc0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.360+0000 7f3133df0b80 1 --2- 172.21.15.2:0/2559343082 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb80c0 0x563345eb8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.360+0000 7f3133df0b80 1 --2- 172.21.15.2:0/2559343082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb7500 0x563345eb7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.360+0000 7f3133df0b80 1 -- 172.21.15.2:0/2559343082 >> 172.21.15.2:0/2559343082 conn(0x563345eac310 msgr2=0x563345eb1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:59.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.360+0000 7f3133df0b80 1 -- 172.21.15.2:0/2559343082 shutdown_connections 2023-11-16T16:48:59.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.360+0000 7f3133df0b80 1 -- 172.21.15.2:0/2559343082 wait complete. 2023-11-16T16:48:59.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.361+0000 7f3133df0b80 1 -- start start 2023-11-16T16:48:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.361+0000 7f3133df0b80 1 Processor -- start 2023-11-16T16:48:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.361+0000 7f3133df0b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb6bb0 0x563345ec0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.361+0000 7f3133df0b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb7500 0x563345ec0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.361+0000 7f3133df0b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb80c0 0x563345ec7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.361+0000 7f3133df0b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x563345da9890 con 0x563345eb6bb0 2023-11-16T16:48:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.361+0000 7f3133df0b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x563345da9530 con 0x563345eb7500 2023-11-16T16:48:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.361+0000 7f3133df0b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x563345c8b8d0 con 0x563345eb80c0 2023-11-16T16:48:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.362+0000 7f3133df0b80 10 client.0 _ll_register_callbacks cb 0x563345dc7730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:48:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.362+0000 7f311e19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb6bb0 0x563345ec0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.362+0000 7f311d99a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb7500 0x563345ec0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.362+0000 7f311d99a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb7500 0x563345ec0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:35932/0 (socket says 172.21.15.2:35932) 2023-11-16T16:48:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.362+0000 7f311e19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb6bb0 0x563345ec0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44458/0 (socket says 172.21.15.2:44458) 2023-11-16T16:48:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.362+0000 7f311d99a700 1 -- 172.21.15.2:0/3325609108 learned_addr learned my addr 172.21.15.2:0/3325609108 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.362+0000 7f311e99c700 1 --2- 172.21.15.2:0/3325609108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb80c0 0x563345ec7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.362+0000 7f311d99a700 1 -- 172.21.15.2:0/3325609108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb80c0 msgr2=0x563345ec7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.362+0000 7f311d99a700 1 --2- 172.21.15.2:0/3325609108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb80c0 0x563345ec7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.362+0000 7f311d99a700 1 -- 172.21.15.2:0/3325609108 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb6bb0 msgr2=0x563345ec0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.362+0000 7f311d99a700 1 --2- 172.21.15.2:0/3325609108 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb6bb0 0x563345ec0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.362+0000 7f311d99a700 1 -- 172.21.15.2:0/3325609108 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3114009d50 con 0x563345eb7500 2023-11-16T16:48:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.362+0000 7f311d99a700 1 --2- 172.21.15.2:0/3325609108 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb7500 0x563345ec0a60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3108004590 tx=0x7f310801d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2534dce2ce8b23f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:59.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.362+0000 7f3101ffb700 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2023-11-16T16:48:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.363+0000 7f3101ffb700 1 -- 172.21.15.2:0/3325609108 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3108016040 con 0x563345eb7500 2023-11-16T16:48:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.363+0000 7f3133df0b80 1 -- 172.21.15.2:0/3325609108 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563345ea9ad0 con 0x563345eb7500 2023-11-16T16:48:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.363+0000 7f3101ffb700 1 -- 172.21.15.2:0/3325609108 <== mon.2 v2:172.21.15.141:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f3108003890 con 0x563345eb7500 2023-11-16T16:48:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.363+0000 7f3101ffb700 1 -- 172.21.15.2:0/3325609108 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3108011e50 con 0x563345eb7500 2023-11-16T16:48:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.363+0000 7f3133df0b80 10 client.25193 fetch_fsmap learned FSMap version 26 2023-11-16T16:48:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.363+0000 7f3133df0b80 1 -- 172.21.15.2:0/3325609108 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x563345dce140 con 0x563345eb7500 2023-11-16T16:48:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.363+0000 7f3101ffb700 1 -- 172.21.15.2:0/3325609108 <== mon.2 v2:172.21.15.141:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563345ea9ad0 con 0x563345eb7500 2023-11-16T16:48:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.364+0000 7f3101ffb700 1 -- 172.21.15.2:0/3325609108 <== mon.2 v2:172.21.15.141:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f3108026030 con 0x563345eb7500 2023-11-16T16:48:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.364+0000 7f3133df0b80 10 client.25193 fetch_fsmap finished waiting for FSMap version 26 2023-11-16T16:48:59.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.364+0000 7f3133df0b80 10 client.25193 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T16:48:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.364+0000 7f3133df0b80 20 client.25193 populate_metadata read hostname 'smithi002' 2023-11-16T16:48:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.364+0000 7f3133df0b80 1 --2- 172.21.15.2:0/3325609108 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x563345ecb7f0 0x563345ecaba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.364+0000 7f3133df0b80 4 client.25193 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:48:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.364+0000 7f3133df0b80 1 -- 172.21.15.2:0/3325609108 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563345dca050 con 0x563345ecb7f0 2023-11-16T16:48:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.364+0000 7f311e19b700 1 --2- 172.21.15.2:0/3325609108 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x563345ecb7f0 0x563345ecaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.365+0000 7f311e19b700 1 --2- 172.21.15.2:0/3325609108 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x563345ecb7f0 0x563345ecaba0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5b0f97914d4d621 server_cookie=8fd67f13d70a56de in_seq=0 out_seq=0 2023-11-16T16:48:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.365+0000 7f3101ffb700 10 client.25193 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:48:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.365+0000 7f3101ffb700 1 -- 172.21.15.2:0/3325609108 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x563345dca050 con 0x563345ecb7f0 2023-11-16T16:48:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.365+0000 7f3101ffb700 10 client.25193 handle_command_reply: tid=0 2023-11-16T16:48:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.365+0000 7f3133df0b80 1 client.25193 shutdown 2023-11-16T16:48:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.366+0000 7f3133df0b80 1 -- 172.21.15.2:0/3325609108 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb7500 msgr2=0x563345ec0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.366+0000 7f3133df0b80 1 --2- 172.21.15.2:0/3325609108 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb7500 0x563345ec0a60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3108004590 tx=0x7f310801d000 comp rx=0 tx=0).stop 2023-11-16T16:48:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.366+0000 7f3133df0b80 1 -- 172.21.15.2:0/3325609108 shutdown_connections 2023-11-16T16:48:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.366+0000 7f3133df0b80 1 --2- 172.21.15.2:0/3325609108 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x563345ecb7f0 0x563345ecaba0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.366+0000 7f3133df0b80 1 --2- 172.21.15.2:0/3325609108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb80c0 0x563345ec7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.366+0000 7f3133df0b80 1 --2- 172.21.15.2:0/3325609108 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb6bb0 0x563345ec0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.366+0000 7f3101ffb700 0 client.25193 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:48:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.366+0000 7f3133df0b80 1 --2- 172.21.15.2:0/3325609108 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb7500 0x563345ec0a60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.366+0000 7f3133df0b80 1 -- 172.21.15.2:0/3325609108 >> 172.21.15.2:0/3325609108 conn(0x563345eac310 msgr2=0x563345eb0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.366+0000 7f3133df0b80 1 -- 172.21.15.2:0/3325609108 shutdown_connections 2023-11-16T16:48:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.366+0000 7f3133df0b80 1 -- 172.21.15.2:0/3325609108 wait complete. 2023-11-16T16:48:59.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.366+0000 7f3133df0b80 20 client.25193 trim_cache size 0 max 16384 2023-11-16T16:48:59.374 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:58 smithi155 ceph-mon[131607]: pgmap v330: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s; 5.4 KiB/s, 1 keys/s, 0 objects/s recovering 2023-11-16T16:48:59.374 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:58 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3319900235' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:48:59.374 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:48:58 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3473697270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:48:59.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.384+0000 7f3133df0b80 1 Processor -- start 2023-11-16T16:48:59.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.384+0000 7f3133df0b80 1 -- start start 2023-11-16T16:48:59.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.385+0000 7f3133df0b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb6bb0 0x563345ecaba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.385+0000 7f3133df0b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb7500 0x563345ecb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.385+0000 7f3133df0b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb80c0 0x563345ec7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.385+0000 7f3133df0b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x563345dc5fa0 con 0x563345eb6bb0 2023-11-16T16:48:59.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.385+0000 7f3133df0b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x563345c8da30 con 0x563345eb80c0 2023-11-16T16:48:59.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.385+0000 7f3133df0b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x563345eb7f40 con 0x563345eb7500 2023-11-16T16:48:59.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.385+0000 7f311e19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb6bb0 0x563345ecaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.385+0000 7f311e19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb6bb0 0x563345ecaba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44464/0 (socket says 172.21.15.2:44464) 2023-11-16T16:48:59.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.385+0000 7f311e19b700 1 -- 172.21.15.2:0/977946662 learned_addr learned my addr 172.21.15.2:0/977946662 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:59.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.385+0000 7f311d99a700 1 --2- 172.21.15.2:0/977946662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb7500 0x563345ecb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.385+0000 7f311e99c700 1 --2- 172.21.15.2:0/977946662 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb80c0 0x563345ec7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.385+0000 7f311e19b700 1 -- 172.21.15.2:0/977946662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb7500 msgr2=0x563345ecb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:59.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.385+0000 7f311e19b700 1 --2- 172.21.15.2:0/977946662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb7500 0x563345ecb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.385+0000 7f311e19b700 1 -- 172.21.15.2:0/977946662 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb80c0 msgr2=0x563345ec7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:59.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.385+0000 7f311e19b700 1 --2- 172.21.15.2:0/977946662 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb80c0 0x563345ec7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.385+0000 7f311e19b700 1 -- 172.21.15.2:0/977946662 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563345dcde90 con 0x563345eb6bb0 2023-11-16T16:48:59.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.386+0000 7f311e19b700 1 --2- 172.21.15.2:0/977946662 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb6bb0 0x563345ecaba0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3114004360 tx=0x7f31140058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9250076cbef14e22 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:59.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.386+0000 7f310d3ff700 1 -- 172.21.15.2:0/977946662 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114029070 con 0x563345eb6bb0 2023-11-16T16:48:59.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.386+0000 7f310d3ff700 1 -- 172.21.15.2:0/977946662 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3114004500 con 0x563345eb6bb0 2023-11-16T16:48:59.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.386+0000 7f310d3ff700 1 -- 172.21.15.2:0/977946662 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311402aa50 con 0x563345eb6bb0 2023-11-16T16:48:59.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.386+0000 7f3133df0b80 1 -- 172.21.15.2:0/977946662 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb6bb0 msgr2=0x563345ecaba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:59.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.387+0000 7f3133df0b80 1 --2- 172.21.15.2:0/977946662 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb6bb0 0x563345ecaba0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3114004360 tx=0x7f31140058e0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.387+0000 7f3133df0b80 1 -- 172.21.15.2:0/977946662 shutdown_connections 2023-11-16T16:48:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.387+0000 7f3133df0b80 1 --2- 172.21.15.2:0/977946662 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb80c0 0x563345ec7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.387+0000 7f3133df0b80 1 --2- 172.21.15.2:0/977946662 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb6bb0 0x563345ecaba0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.387+0000 7f3133df0b80 1 --2- 172.21.15.2:0/977946662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb7500 0x563345ecb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.387+0000 7f3133df0b80 1 -- 172.21.15.2:0/977946662 >> 172.21.15.2:0/977946662 conn(0x563345eac310 msgr2=0x563345ebe440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.387+0000 7f3133df0b80 1 -- 172.21.15.2:0/977946662 shutdown_connections 2023-11-16T16:48:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.387+0000 7f3133df0b80 1 -- 172.21.15.2:0/977946662 wait complete. 2023-11-16T16:48:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.387+0000 7f3133df0b80 1 -- start start 2023-11-16T16:48:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.387+0000 7f3133df0b80 1 Processor -- start 2023-11-16T16:48:59.394 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.388+0000 7f3133df0b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb6bb0 0x563345ec2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.388+0000 7f3133df0b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb7500 0x563345ec2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.388+0000 7f3133df0b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb80c0 0x563345f1f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.388+0000 7f3133df0b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x563345f1fc50 con 0x563345eb7500 2023-11-16T16:48:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.388+0000 7f3133df0b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x563345f1fd90 con 0x563345eb6bb0 2023-11-16T16:48:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.388+0000 7f3133df0b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x563345f1ff00 con 0x563345eb80c0 2023-11-16T16:48:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.388+0000 7f3133df0b80 10 client.0 _ll_register_callbacks cb 0x563345dc7730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:48:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.388+0000 7f311d99a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb7500 0x563345ec2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.388+0000 7f311d99a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb7500 0x563345ec2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44472/0 (socket says 172.21.15.2:44472) 2023-11-16T16:48:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.388+0000 7f311d99a700 1 -- 172.21.15.2:0/3929077655 learned_addr learned my addr 172.21.15.2:0/3929077655 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:48:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.388+0000 7f311e99c700 1 --2- 172.21.15.2:0/3929077655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb80c0 0x563345f1f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.395 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.388+0000 7f311e19b700 1 --2- 172.21.15.2:0/3929077655 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb6bb0 0x563345ec2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.388+0000 7f311d99a700 1 -- 172.21.15.2:0/3929077655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb80c0 msgr2=0x563345f1f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:59.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.388+0000 7f311d99a700 1 --2- 172.21.15.2:0/3929077655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb80c0 0x563345f1f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.388+0000 7f311d99a700 1 -- 172.21.15.2:0/3929077655 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb6bb0 msgr2=0x563345ec2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:48:59.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.388+0000 7f311d99a700 1 --2- 172.21.15.2:0/3929077655 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb6bb0 0x563345ec2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.388+0000 7f311d99a700 1 -- 172.21.15.2:0/3929077655 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563345c895e0 con 0x563345eb7500 2023-11-16T16:48:59.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.389+0000 7f311d99a700 1 --2- 172.21.15.2:0/3929077655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb7500 0x563345ec2f50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3108004560 tx=0x7f31080286b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8760ebedbaf1cad server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:48:59.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.389+0000 7f3101ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:48:59.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.389+0000 7f3101ffb700 1 -- 172.21.15.2:0/3929077655 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3108016040 con 0x563345eb7500 2023-11-16T16:48:59.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.389+0000 7f3133df0b80 1 -- 172.21.15.2:0/3929077655 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3114004390 con 0x563345eb7500 2023-11-16T16:48:59.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.389+0000 7f3101ffb700 1 -- 172.21.15.2:0/3929077655 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f3108028b60 con 0x563345eb7500 2023-11-16T16:48:59.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.389+0000 7f3101ffb700 1 -- 172.21.15.2:0/3929077655 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3108009ec0 con 0x563345eb7500 2023-11-16T16:48:59.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.389+0000 7f3133df0b80 10 client.15168 fetch_fsmap learned FSMap version 26 2023-11-16T16:48:59.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.389+0000 7f3133df0b80 1 -- 172.21.15.2:0/3929077655 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x563345c1ec10 con 0x563345eb7500 2023-11-16T16:48:59.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.390+0000 7f3101ffb700 1 -- 172.21.15.2:0/3929077655 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3114004390 con 0x563345eb7500 2023-11-16T16:48:59.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.390+0000 7f3101ffb700 1 -- 172.21.15.2:0/3929077655 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f310800f830 con 0x563345eb7500 2023-11-16T16:48:59.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.390+0000 7f3133df0b80 10 client.15168 fetch_fsmap finished waiting for FSMap version 26 2023-11-16T16:48:59.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.390+0000 7f3133df0b80 10 client.15168 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T16:48:59.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.390+0000 7f3133df0b80 20 client.15168 populate_metadata read hostname 'smithi002' 2023-11-16T16:48:59.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.390+0000 7f3133df0b80 1 --2- 172.21.15.2:0/3929077655 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x563345f24cf0 0x563345f215a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:48:59.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.390+0000 7f3133df0b80 4 client.15168 mds_command: new command op to 24590 tid=0[{"prefix": "status"}] 2023-11-16T16:48:59.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.390+0000 7f3133df0b80 1 -- 172.21.15.2:0/3929077655 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563345dd6610 con 0x563345f24cf0 2023-11-16T16:48:59.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.390+0000 7f311e19b700 1 --2- 172.21.15.2:0/3929077655 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x563345f24cf0 0x563345f215a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:48:59.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.391+0000 7f311e19b700 1 --2- 172.21.15.2:0/3929077655 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x563345f24cf0 0x563345f215a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f87f4558f56bffc server_cookie=20b14b22acb3044e in_seq=0 out_seq=0 2023-11-16T16:48:59.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.391+0000 7f3101ffb700 10 client.15168 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:48:59.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.391+0000 7f3101ffb700 1 -- 172.21.15.2:0/3929077655 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x563345dd6610 con 0x563345f24cf0 2023-11-16T16:48:59.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.391+0000 7f3101ffb700 10 client.15168 handle_command_reply: tid=0 2023-11-16T16:48:59.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.391+0000 7f3133df0b80 1 client.15168 shutdown 2023-11-16T16:48:59.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.391+0000 7f3133df0b80 1 -- 172.21.15.2:0/3929077655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb7500 msgr2=0x563345ec2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:59.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.391+0000 7f3133df0b80 1 --2- 172.21.15.2:0/3929077655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb7500 0x563345ec2f50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3108004560 tx=0x7f31080286b0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.391+0000 7f3133df0b80 1 -- 172.21.15.2:0/3929077655 shutdown_connections 2023-11-16T16:48:59.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.391+0000 7f3133df0b80 1 --2- 172.21.15.2:0/3929077655 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x563345f24cf0 0x563345f215a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.391+0000 7f3133df0b80 1 --2- 172.21.15.2:0/3929077655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563345eb80c0 0x563345f1f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.391+0000 7f3133df0b80 1 --2- 172.21.15.2:0/3929077655 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563345eb6bb0 0x563345ec2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.391+0000 7f3133df0b80 1 --2- 172.21.15.2:0/3929077655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563345eb7500 0x563345ec2f50 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.391+0000 7f3101ffb700 0 client.15168 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:48:59.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.391+0000 7f3133df0b80 1 -- 172.21.15.2:0/3929077655 >> 172.21.15.2:0/3929077655 conn(0x563345eac310 msgr2=0x563345ecaba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:59.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.391+0000 7f3133df0b80 1 -- 172.21.15.2:0/3929077655 shutdown_connections 2023-11-16T16:48:59.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.391+0000 7f3133df0b80 1 -- 172.21.15.2:0/3929077655 wait complete. 2023-11-16T16:48:59.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.392+0000 7f3133df0b80 20 client.15168 trim_cache size 0 max 16384 2023-11-16T16:48:59.399 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T16:48:59.400 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "875fcd24-849e-11ee-8dbe-212e2dc638e7", 2023-11-16T16:48:59.400 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2023-11-16T16:48:59.400 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24590, 2023-11-16T16:48:59.400 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2023-11-16T16:48:59.400 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2023-11-16T16:48:59.400 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2023-11-16T16:48:59.400 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 93.040509278000002, 2023-11-16T16:48:59.400 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 26, 2023-11-16T16:48:59.400 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 93, 2023-11-16T16:48:59.400 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 91, 2023-11-16T16:48:59.400 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 99.087135373999999 2023-11-16T16:48:59.400 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T16:48:59.401 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.392+0000 7f31203ff700 1 -- 172.21.15.2:0/693629628 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31040a11e0 msgr2=0x7f31040a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:59.401 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.392+0000 7f31203ff700 1 --2- 172.21.15.2:0/693629628 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31040a11e0 0x7f31040a3690 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3118074850 tx=0x7f3108009bf0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.401 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.392+0000 7f31203ff700 1 -- 172.21.15.2:0/693629628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31181080f0 msgr2=0x7f311819f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:48:59.402 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.392+0000 7f31203ff700 1 --2- 172.21.15.2:0/693629628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31181080f0 0x7f311819f880 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3110009fd0 tx=0x7f311000bce0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.402 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.392+0000 7f31203ff700 1 -- 172.21.15.2:0/693629628 shutdown_connections 2023-11-16T16:48:59.402 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.392+0000 7f31203ff700 1 --2- 172.21.15.2:0/693629628 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31040a11e0 0x7f31040a3690 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.402 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.392+0000 7f31203ff700 1 --2- 172.21.15.2:0/693629628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31181080f0 0x7f311819f880 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.402 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.392+0000 7f31203ff700 1 --2- 172.21.15.2:0/693629628 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3118073550 0x7f31181982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.402 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.392+0000 7f31203ff700 1 --2- 172.21.15.2:0/693629628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3118074b30 0x7f3118198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:48:59.402 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.392+0000 7f31203ff700 1 -- 172.21.15.2:0/693629628 >> 172.21.15.2:0/693629628 conn(0x7f31180fb810 msgr2=0x7f31180fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:48:59.402 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.392+0000 7f31203ff700 1 -- 172.21.15.2:0/693629628 shutdown_connections 2023-11-16T16:48:59.402 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:48:59.392+0000 7f31203ff700 1 -- 172.21.15.2:0/693629628 wait complete. 2023-11-16T16:48:59.412 INFO:tasks.cephadm:Setup complete, yielding 2023-11-16T16:48:59.413 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-11-16T16:48:59.422 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T16:48:59.422 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch status' 2023-11-16T16:49:00.282 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:49:00.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:00 smithi141 ceph-mon[131184]: pgmap v331: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s; 5.4 KiB/s, 0 objects/s recovering 2023-11-16T16:49:00.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:00 smithi155 ceph-mon[131607]: pgmap v331: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s; 5.4 KiB/s, 0 objects/s recovering 2023-11-16T16:49:00.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:00 smithi002 ceph-mon[128712]: pgmap v331: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s; 5.4 KiB/s, 0 objects/s recovering 2023-11-16T16:49:01.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.834+0000 7f8105d39700 1 -- 172.21.15.2:0/3767805531 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81000fe970 msgr2=0x7f81000fed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:01.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.834+0000 7f8105d39700 1 --2- 172.21.15.2:0/3767805531 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81000fe970 0x7f81000fed80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f80f4009990 tx=0x7f80f4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:01.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.834+0000 7f8105d39700 1 -- 172.21.15.2:0/3767805531 shutdown_connections 2023-11-16T16:49:01.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.834+0000 7f8105d39700 1 --2- 172.21.15.2:0/3767805531 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81000ffc70 0x7f81001000e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:01.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.834+0000 7f8105d39700 1 --2- 172.21.15.2:0/3767805531 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81000fe970 0x7f81000fed80 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:01.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.834+0000 7f8105d39700 1 --2- 172.21.15.2:0/3767805531 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81000ff2c0 0x7f81000ff730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:01.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.834+0000 7f8105d39700 1 -- 172.21.15.2:0/3767805531 >> 172.21.15.2:0/3767805531 conn(0x7f81000fa620 msgr2=0x7f81000fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:01.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.834+0000 7f8105d39700 1 -- 172.21.15.2:0/3767805531 shutdown_connections 2023-11-16T16:49:01.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.834+0000 7f8105d39700 1 -- 172.21.15.2:0/3767805531 wait complete. 2023-11-16T16:49:01.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.835+0000 7f8105d39700 1 Processor -- start 2023-11-16T16:49:01.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.836+0000 7f8105d39700 1 -- start start 2023-11-16T16:49:01.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.836+0000 7f8105d39700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81000fe970 0x7f8100198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:01.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.836+0000 7f8105d39700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81000ff2c0 0x7f8100198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:01.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.836+0000 7f8105d39700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81000ffc70 0x7f810019f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:01.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.836+0000 7f8105d39700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810019fbe0 con 0x7f81000fe970 2023-11-16T16:49:01.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.836+0000 7f8105d39700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810019fd20 con 0x7f81000ff2c0 2023-11-16T16:49:01.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.836+0000 7f8105d39700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810019feb0 con 0x7f81000ffc70 2023-11-16T16:49:01.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.836+0000 7f8104d37700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81000fe970 0x7f8100198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:01.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.836+0000 7f8104d37700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81000fe970 0x7f8100198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44498/0 (socket says 172.21.15.2:44498) 2023-11-16T16:49:01.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.836+0000 7f8104d37700 1 -- 172.21.15.2:0/1997688084 learned_addr learned my addr 172.21.15.2:0/1997688084 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:01.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.836+0000 7f8105538700 1 --2- 172.21.15.2:0/1997688084 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81000ffc70 0x7f810019f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:01.839 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.837+0000 7f80fffff700 1 --2- 172.21.15.2:0/1997688084 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81000ff2c0 0x7f8100198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:01.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.837+0000 7f8104d37700 1 -- 172.21.15.2:0/1997688084 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81000ffc70 msgr2=0x7f810019f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:01.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.837+0000 7f8104d37700 1 --2- 172.21.15.2:0/1997688084 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81000ffc70 0x7f810019f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:01.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.837+0000 7f8104d37700 1 -- 172.21.15.2:0/1997688084 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81000ff2c0 msgr2=0x7f8100198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:01.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.837+0000 7f8104d37700 1 --2- 172.21.15.2:0/1997688084 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81000ff2c0 0x7f8100198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:01.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.837+0000 7f8104d37700 1 -- 172.21.15.2:0/1997688084 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f40095f0 con 0x7f81000fe970 2023-11-16T16:49:01.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.837+0000 7f8104d37700 1 --2- 172.21.15.2:0/1997688084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81000fe970 0x7f8100198050 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f80f4004770 tx=0x7f80f4009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:01.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.837+0000 7f80fdffb700 1 -- 172.21.15.2:0/1997688084 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f401d070 con 0x7f81000fe970 2023-11-16T16:49:01.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.837+0000 7f80fdffb700 1 -- 172.21.15.2:0/1997688084 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f80f4005030 con 0x7f81000fe970 2023-11-16T16:49:01.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.837+0000 7f80fdffb700 1 -- 172.21.15.2:0/1997688084 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f4004b50 con 0x7f81000fe970 2023-11-16T16:49:01.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.838+0000 7f8105d39700 1 -- 172.21.15.2:0/1997688084 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810010dc80 con 0x7f81000fe970 2023-11-16T16:49:01.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.838+0000 7f8105d39700 1 -- 172.21.15.2:0/1997688084 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f810010daf0 con 0x7f81000fe970 2023-11-16T16:49:01.841 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.839+0000 7f8105d39700 1 -- 172.21.15.2:0/1997688084 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8100109750 con 0x7f81000fe970 2023-11-16T16:49:01.842 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.841+0000 7f80fdffb700 1 -- 172.21.15.2:0/1997688084 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f80f4004cf0 con 0x7f81000fe970 2023-11-16T16:49:01.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.843+0000 7f80fdffb700 1 --2- 172.21.15.2:0/1997688084 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f80f00a1260 0x7f80f00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:01.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.843+0000 7f80fdffb700 1 -- 172.21.15.2:0/1997688084 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f80f40cdf70 con 0x7f81000fe970 2023-11-16T16:49:01.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.843+0000 7f80fffff700 1 --2- 172.21.15.2:0/1997688084 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f80f00a1260 0x7f80f00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:01.845 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.844+0000 7f80fffff700 1 --2- 172.21.15.2:0/1997688084 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f80f00a1260 0x7f80f00a3710 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f80ec009c60 tx=0x7f80ec009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:01.849 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:01.847+0000 7f80fdffb700 1 -- 172.21.15.2:0/1997688084 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f80f405a730 con 0x7f81000fe970 2023-11-16T16:49:02.072 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:02.070+0000 7f8105d39700 1 -- 172.21.15.2:0/1997688084 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8100061c80 con 0x7f80f00a1260 2023-11-16T16:49:02.073 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:02.071+0000 7f80fdffb700 1 -- 172.21.15.2:0/1997688084 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f8100061c80 con 0x7f80f00a1260 2023-11-16T16:49:02.073 INFO:teuthology.orchestra.run.smithi002.stdout:Backend: cephadm 2023-11-16T16:49:02.073 INFO:teuthology.orchestra.run.smithi002.stdout:Available: Yes 2023-11-16T16:49:02.073 INFO:teuthology.orchestra.run.smithi002.stdout:Paused: No 2023-11-16T16:49:02.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:02.073+0000 7f8105d39700 1 -- 172.21.15.2:0/1997688084 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f80f00a1260 msgr2=0x7f80f00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:02.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:02.073+0000 7f8105d39700 1 --2- 172.21.15.2:0/1997688084 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f80f00a1260 0x7f80f00a3710 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f80ec009c60 tx=0x7f80ec009260 comp rx=0 tx=0).stop 2023-11-16T16:49:02.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:02.073+0000 7f8105d39700 1 -- 172.21.15.2:0/1997688084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81000fe970 msgr2=0x7f8100198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:02.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:02.073+0000 7f8105d39700 1 --2- 172.21.15.2:0/1997688084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81000fe970 0x7f8100198050 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f80f4004770 tx=0x7f80f4009580 comp rx=0 tx=0).stop 2023-11-16T16:49:02.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:02.073+0000 7f8105d39700 1 -- 172.21.15.2:0/1997688084 shutdown_connections 2023-11-16T16:49:02.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:02.073+0000 7f8105d39700 1 --2- 172.21.15.2:0/1997688084 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f80f00a1260 0x7f80f00a3710 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:02.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:02.073+0000 7f8105d39700 1 --2- 172.21.15.2:0/1997688084 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81000ffc70 0x7f810019f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:02.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:02.073+0000 7f8105d39700 1 --2- 172.21.15.2:0/1997688084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81000fe970 0x7f8100198050 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:02.075 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:02.073+0000 7f8105d39700 1 --2- 172.21.15.2:0/1997688084 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81000ff2c0 0x7f8100198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:02.076 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:02.073+0000 7f8105d39700 1 -- 172.21.15.2:0/1997688084 >> 172.21.15.2:0/1997688084 conn(0x7f81000fa620 msgr2=0x7f81000fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:02.076 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:02.073+0000 7f8105d39700 1 -- 172.21.15.2:0/1997688084 shutdown_connections 2023-11-16T16:49:02.076 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:02.073+0000 7f8105d39700 1 -- 172.21.15.2:0/1997688084 wait complete. 2023-11-16T16:49:02.579 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch ps' 2023-11-16T16:49:03.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:02 smithi155 ceph-mon[131607]: pgmap v332: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-11-16T16:49:03.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:02 smithi155 ceph-mon[131607]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:03.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:02 smithi002 ceph-mon[128712]: pgmap v332: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-11-16T16:49:03.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:02 smithi002 ceph-mon[128712]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:03.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:02 smithi141 ceph-mon[131184]: pgmap v332: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-11-16T16:49:03.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:02 smithi141 ceph-mon[131184]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:03.444 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:49:04.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.933+0000 7f69495a9700 1 -- 172.21.15.2:0/4206260292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69441014f0 msgr2=0x7f6944101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:04.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.933+0000 7f69495a9700 1 --2- 172.21.15.2:0/4206260292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69441014f0 0x7f6944101940 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f69380098f0 tx=0x7f6938009c00 comp rx=0 tx=0).stop 2023-11-16T16:49:04.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.933+0000 7f69495a9700 1 -- 172.21.15.2:0/4206260292 shutdown_connections 2023-11-16T16:49:04.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.933+0000 7f69495a9700 1 --2- 172.21.15.2:0/4206260292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69441002f0 0x7f6944100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:04.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.933+0000 7f69495a9700 1 --2- 172.21.15.2:0/4206260292 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6944101e80 0x7f6944108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:04.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.933+0000 7f69495a9700 1 --2- 172.21.15.2:0/4206260292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69441014f0 0x7f6944101940 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:04.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.933+0000 7f69495a9700 1 -- 172.21.15.2:0/4206260292 >> 172.21.15.2:0/4206260292 conn(0x7f69440fb900 msgr2=0x7f69440fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:04.936 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.933+0000 7f69495a9700 1 -- 172.21.15.2:0/4206260292 shutdown_connections 2023-11-16T16:49:04.936 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.933+0000 7f69495a9700 1 -- 172.21.15.2:0/4206260292 wait complete. 2023-11-16T16:49:04.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.934+0000 7f69495a9700 1 Processor -- start 2023-11-16T16:49:04.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.934+0000 7f69495a9700 1 -- start start 2023-11-16T16:49:04.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.934+0000 7f69495a9700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69441002f0 0x7f6944074890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:04.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.934+0000 7f69495a9700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f69441014f0 0x7f6944072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:04.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.934+0000 7f69495a9700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6944101e80 0x7f6944073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:04.937 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.934+0000 7f69495a9700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6944073ba0 con 0x7f6944101e80 2023-11-16T16:49:04.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.934+0000 7f69495a9700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6944073d10 con 0x7f69441014f0 2023-11-16T16:49:04.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.934+0000 7f69495a9700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69441a0100 con 0x7f69441002f0 2023-11-16T16:49:04.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.935+0000 7f69427fc700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f69441014f0 0x7f6944072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:04.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.935+0000 7f6942ffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69441002f0 0x7f6944074890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:04.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.935+0000 7f69427fc700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f69441014f0 0x7f6944072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:33154/0 (socket says 172.21.15.2:33154) 2023-11-16T16:49:04.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.935+0000 7f69427fc700 1 -- 172.21.15.2:0/274860662 learned_addr learned my addr 172.21.15.2:0/274860662 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:04.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.935+0000 7f69437fe700 1 --2- 172.21.15.2:0/274860662 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6944101e80 0x7f6944073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:04.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.935+0000 7f69427fc700 1 -- 172.21.15.2:0/274860662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69441002f0 msgr2=0x7f6944074890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:04.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.935+0000 7f69427fc700 1 --2- 172.21.15.2:0/274860662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69441002f0 0x7f6944074890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:04.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.935+0000 7f69427fc700 1 -- 172.21.15.2:0/274860662 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6944101e80 msgr2=0x7f6944073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:04.938 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.935+0000 7f69427fc700 1 --2- 172.21.15.2:0/274860662 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6944101e80 0x7f6944073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:04.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.935+0000 7f69427fc700 1 -- 172.21.15.2:0/274860662 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69340095f0 con 0x7f69441014f0 2023-11-16T16:49:04.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.935+0000 7f69427fc700 1 --2- 172.21.15.2:0/274860662 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f69441014f0 0x7f6944072ee0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6938009fd0 tx=0x7f693800b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:04.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.936+0000 7f692bfff700 1 -- 172.21.15.2:0/274860662 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693801c070 con 0x7f69441014f0 2023-11-16T16:49:04.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.936+0000 7f692bfff700 1 -- 172.21.15.2:0/274860662 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f693800f7d0 con 0x7f69441014f0 2023-11-16T16:49:04.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.936+0000 7f69495a9700 1 -- 172.21.15.2:0/274860662 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6938009520 con 0x7f69441014f0 2023-11-16T16:49:04.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.936+0000 7f692bfff700 1 -- 172.21.15.2:0/274860662 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6938004ad0 con 0x7f69441014f0 2023-11-16T16:49:04.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.936+0000 7f69495a9700 1 -- 172.21.15.2:0/274860662 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f694410bc90 con 0x7f69441014f0 2023-11-16T16:49:04.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.937+0000 7f69495a9700 1 -- 172.21.15.2:0/274860662 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69441078e0 con 0x7f69441014f0 2023-11-16T16:49:04.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.939+0000 7f692bfff700 1 -- 172.21.15.2:0/274860662 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f6938020070 con 0x7f69441014f0 2023-11-16T16:49:04.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.940+0000 7f692bfff700 1 --2- 172.21.15.2:0/274860662 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f692c0a11e0 0x7f692c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:04.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.940+0000 7f692bfff700 1 -- 172.21.15.2:0/274860662 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f69380cda80 con 0x7f69441014f0 2023-11-16T16:49:04.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.940+0000 7f6942ffd700 1 --2- 172.21.15.2:0/274860662 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f692c0a11e0 0x7f692c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:04.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.941+0000 7f6942ffd700 1 --2- 172.21.15.2:0/274860662 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f692c0a11e0 0x7f692c0a3690 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6944101350 tx=0x7f6934009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:04.944 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:04.943+0000 7f692bfff700 1 -- 172.21.15.2:0/274860662 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6938096d40 con 0x7f69441014f0 2023-11-16T16:49:05.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:04 smithi002 ceph-mon[128712]: pgmap v333: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-11-16T16:49:05.161 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:05.159+0000 7f69495a9700 1 -- 172.21.15.2:0/274860662 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6944060f30 con 0x7f692c0a11e0 2023-11-16T16:49:05.175 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:05.173+0000 7f692bfff700 1 -- 172.21.15.2:0/274860662 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7f6944060f30 con 0x7f692c0a11e0 2023-11-16T16:49:05.175 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-11-16T16:49:05.175 INFO:teuthology.orchestra.run.smithi002.stdout:mds.a smithi002 running (107s) 21s ago 107s 13.7M - 18.0.0-3905-g5e4a743a d5b1f852d682 816b8eda420f 2023-11-16T16:49:05.175 INFO:teuthology.orchestra.run.smithi002.stdout:mds.b smithi141 running (105s) 33s ago 104s 30.9M - 18.0.0-3905-g5e4a743a d5b1f852d682 971ae64327c4 2023-11-16T16:49:05.175 INFO:teuthology.orchestra.run.smithi002.stdout:mds.c smithi155 running (118s) 33s ago 117s 17.9M - 18.0.0-3905-g5e4a743a d5b1f852d682 78b25505c4e9 2023-11-16T16:49:05.175 INFO:teuthology.orchestra.run.smithi002.stdout:mds.d smithi002 running (2m) 21s ago 2m 13.8M - 18.0.0-3905-g5e4a743a d5b1f852d682 915282f55dd1 2023-11-16T16:49:05.176 INFO:teuthology.orchestra.run.smithi002.stdout:mds.e smithi141 running (2m) 33s ago 2m 18.8M - 18.0.0-3905-g5e4a743a d5b1f852d682 475a2474adfc 2023-11-16T16:49:05.176 INFO:teuthology.orchestra.run.smithi002.stdout:mds.f smithi155 running (2m) 33s ago 2m 17.9M - 18.0.0-3905-g5e4a743a d5b1f852d682 eba0770ed574 2023-11-16T16:49:05.176 INFO:teuthology.orchestra.run.smithi002.stdout:mds.g smithi002 running (110s) 21s ago 110s 13.7M - 18.0.0-3905-g5e4a743a d5b1f852d682 e82d6215a6d0 2023-11-16T16:49:05.176 INFO:teuthology.orchestra.run.smithi002.stdout:mds.h smithi141 running (116s) 33s ago 115s 20.5M - 18.0.0-3905-g5e4a743a d5b1f852d682 8ca4ca040eac 2023-11-16T16:49:05.176 INFO:teuthology.orchestra.run.smithi002.stdout:mds.i smithi155 running (113s) 33s ago 113s 18.8M - 18.0.0-3905-g5e4a743a d5b1f852d682 00ca969bfe73 2023-11-16T16:49:05.176 INFO:teuthology.orchestra.run.smithi002.stdout:mds.j smithi002 running (2m) 21s ago 2m 13.8M - 18.0.0-3905-g5e4a743a d5b1f852d682 cf1bb173fe07 2023-11-16T16:49:05.176 INFO:teuthology.orchestra.run.smithi002.stdout:mds.k smithi141 running (2m) 33s ago 2m 14.2M - 18.0.0-3905-g5e4a743a d5b1f852d682 4b16fdad1472 2023-11-16T16:49:05.176 INFO:teuthology.orchestra.run.smithi002.stdout:mds.l smithi155 running (2m) 33s ago 2m 17.6M - 18.0.0-3905-g5e4a743a d5b1f852d682 9aaccb5bb9bf 2023-11-16T16:49:05.176 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi002 *:9283,8765 running (9m) 21s ago 9m 521M - 18.0.0-3905-g5e4a743a d5b1f852d682 98af492427e6 2023-11-16T16:49:05.176 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi141 *:8443,8765 running (7m) 33s ago 7m 448M - 18.0.0-3905-g5e4a743a d5b1f852d682 a93e0f19fa7b 2023-11-16T16:49:05.176 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.z smithi155 *:8443,8765 running (7m) 33s ago 7m 447M - 18.0.0-3905-g5e4a743a d5b1f852d682 2f95791bfcd5 2023-11-16T16:49:05.176 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (9m) 21s ago 9m 96.0M 2048M 18.0.0-3905-g5e4a743a d5b1f852d682 c8cc70f11b42 2023-11-16T16:49:05.176 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi141 running (7m) 33s ago 7m 65.5M 2048M 18.0.0-3905-g5e4a743a d5b1f852d682 78c734a04948 2023-11-16T16:49:05.177 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi155 running (7m) 33s ago 7m 69.8M 2048M 18.0.0-3905-g5e4a743a d5b1f852d682 5dbc5b655938 2023-11-16T16:49:05.177 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (6m) 21s ago 6m 71.2M 4319M 18.0.0-3905-g5e4a743a d5b1f852d682 6f6657375717 2023-11-16T16:49:05.177 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi141 running (6m) 33s ago 6m 78.4M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 9114a6f26adb 2023-11-16T16:49:05.177 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi155 running (6m) 33s ago 6m 74.4M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 c48d9ec5a001 2023-11-16T16:49:05.177 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (5m) 21s ago 5m 65.9M 4319M 18.0.0-3905-g5e4a743a d5b1f852d682 6aee30e87c44 2023-11-16T16:49:05.177 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi141 running (5m) 33s ago 5m 64.4M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 008f8b513c23 2023-11-16T16:49:05.177 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi155 running (4m) 33s ago 4m 70.2M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 7667905f7690 2023-11-16T16:49:05.177 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi002 running (4m) 21s ago 4m 61.7M 4319M 18.0.0-3905-g5e4a743a d5b1f852d682 23308c82a636 2023-11-16T16:49:05.177 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi141 running (4m) 33s ago 4m 65.7M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 c395444e37f7 2023-11-16T16:49:05.177 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8 smithi155 running (3m) 33s ago 3m 62.3M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 4ba4478d7815 2023-11-16T16:49:05.177 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9 smithi002 running (3m) 21s ago 3m 59.9M 4319M 18.0.0-3905-g5e4a743a d5b1f852d682 22ddbee54376 2023-11-16T16:49:05.177 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10 smithi141 running (2m) 33s ago 2m 62.3M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 faa0d15da3af 2023-11-16T16:49:05.177 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11 smithi155 running (2m) 33s ago 2m 55.0M 4292M 18.0.0-3905-g5e4a743a d5b1f852d682 e4c83809cc77 2023-11-16T16:49:05.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:05.175+0000 7f69495a9700 1 -- 172.21.15.2:0/274860662 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f692c0a11e0 msgr2=0x7f692c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:05.178 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:05.175+0000 7f69495a9700 1 --2- 172.21.15.2:0/274860662 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f692c0a11e0 0x7f692c0a3690 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6944101350 tx=0x7f6934009310 comp rx=0 tx=0).stop 2023-11-16T16:49:05.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:05.175+0000 7f69495a9700 1 -- 172.21.15.2:0/274860662 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f69441014f0 msgr2=0x7f6944072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:05.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:05.175+0000 7f69495a9700 1 --2- 172.21.15.2:0/274860662 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f69441014f0 0x7f6944072ee0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6938009fd0 tx=0x7f693800b790 comp rx=0 tx=0).stop 2023-11-16T16:49:05.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:05.176+0000 7f69495a9700 1 -- 172.21.15.2:0/274860662 shutdown_connections 2023-11-16T16:49:05.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:05.176+0000 7f69495a9700 1 --2- 172.21.15.2:0/274860662 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f692c0a11e0 0x7f692c0a3690 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:05.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:05.176+0000 7f69495a9700 1 --2- 172.21.15.2:0/274860662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69441002f0 0x7f6944074890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:05.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:05.176+0000 7f69495a9700 1 --2- 172.21.15.2:0/274860662 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6944101e80 0x7f6944073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:05.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:05.176+0000 7f69495a9700 1 --2- 172.21.15.2:0/274860662 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f69441014f0 0x7f6944072ee0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:05.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:05.176+0000 7f69495a9700 1 -- 172.21.15.2:0/274860662 >> 172.21.15.2:0/274860662 conn(0x7f69440fb900 msgr2=0x7f6944104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:05.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:05.176+0000 7f69495a9700 1 -- 172.21.15.2:0/274860662 shutdown_connections 2023-11-16T16:49:05.179 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:05.176+0000 7f69495a9700 1 -- 172.21.15.2:0/274860662 wait complete. 2023-11-16T16:49:05.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:04 smithi141 ceph-mon[131184]: pgmap v333: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-11-16T16:49:05.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:04 smithi155 ceph-mon[131607]: pgmap v333: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-11-16T16:49:05.643 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch ls' 2023-11-16T16:49:06.512 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:49:07.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:06 smithi002 ceph-mon[128712]: from='client.25211 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:07.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:06 smithi002 ceph-mon[128712]: pgmap v334: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2023-11-16T16:49:07.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:06 smithi141 ceph-mon[131184]: from='client.25211 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:07.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:06 smithi141 ceph-mon[131184]: pgmap v334: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2023-11-16T16:49:07.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:06 smithi155 ceph-mon[131607]: from='client.25211 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:07.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:06 smithi155 ceph-mon[131607]: pgmap v334: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2023-11-16T16:49:08.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.056+0000 7f7269f94700 1 -- 172.21.15.2:0/3185819524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72641014f0 msgr2=0x7f7264101940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:08.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.056+0000 7f7269f94700 1 --2- 172.21.15.2:0/3185819524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72641014f0 0x7f7264101940 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7258009990 tx=0x7f7258009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:08.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.056+0000 7f7269f94700 1 -- 172.21.15.2:0/3185819524 shutdown_connections 2023-11-16T16:49:08.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.056+0000 7f7269f94700 1 --2- 172.21.15.2:0/3185819524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7264101e80 0x7f7264108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:08.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.056+0000 7f7269f94700 1 --2- 172.21.15.2:0/3185819524 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f72641002f0 0x7f7264100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:08.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.056+0000 7f7269f94700 1 --2- 172.21.15.2:0/3185819524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72641014f0 0x7f7264101940 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:08.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.056+0000 7f7269f94700 1 -- 172.21.15.2:0/3185819524 >> 172.21.15.2:0/3185819524 conn(0x7f72640fb900 msgr2=0x7f72640fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:08.059 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.056+0000 7f7269f94700 1 -- 172.21.15.2:0/3185819524 shutdown_connections 2023-11-16T16:49:08.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.056+0000 7f7269f94700 1 -- 172.21.15.2:0/3185819524 wait complete. 2023-11-16T16:49:08.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.057+0000 7f7269f94700 1 Processor -- start 2023-11-16T16:49:08.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.058+0000 7f7269f94700 1 -- start start 2023-11-16T16:49:08.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.058+0000 7f7269f94700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72641002f0 0x7f7264198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:08.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.058+0000 7f7269f94700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72641014f0 0x7f72641987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:08.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.058+0000 7f7269f94700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7264101e80 0x7f726419f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:08.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.058+0000 7f7269f94700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726419fe00 con 0x7f72641002f0 2023-11-16T16:49:08.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.058+0000 7f7269f94700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726419ff40 con 0x7f7264101e80 2023-11-16T16:49:08.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.058+0000 7f7269f94700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72641a00d0 con 0x7f72641014f0 2023-11-16T16:49:08.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.058+0000 7f72637fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72641002f0 0x7f7264198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:08.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.058+0000 7f7262ffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72641014f0 0x7f72641987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:08.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.058+0000 7f72637fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72641002f0 0x7f7264198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59454/0 (socket says 172.21.15.2:59454) 2023-11-16T16:49:08.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.058+0000 7f72637fe700 1 -- 172.21.15.2:0/1236067643 learned_addr learned my addr 172.21.15.2:0/1236067643 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:08.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.059+0000 7f7263fff700 1 --2- 172.21.15.2:0/1236067643 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7264101e80 0x7f726419f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:08.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.059+0000 7f7262ffd700 1 -- 172.21.15.2:0/1236067643 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7264101e80 msgr2=0x7f726419f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:08.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.059+0000 7f7262ffd700 1 --2- 172.21.15.2:0/1236067643 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7264101e80 0x7f726419f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:08.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.059+0000 7f7262ffd700 1 -- 172.21.15.2:0/1236067643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72641002f0 msgr2=0x7f7264198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:08.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.059+0000 7f7262ffd700 1 --2- 172.21.15.2:0/1236067643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72641002f0 0x7f7264198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:08.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.059+0000 7f7262ffd700 1 -- 172.21.15.2:0/1236067643 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72580095f0 con 0x7f72641014f0 2023-11-16T16:49:08.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.059+0000 7f7262ffd700 1 --2- 172.21.15.2:0/1236067643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72641014f0 0x7f72641987b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7258009990 tx=0x7f7258009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:08.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.059+0000 7f7260ff9700 1 -- 172.21.15.2:0/1236067643 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725801d070 con 0x7f72641014f0 2023-11-16T16:49:08.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.059+0000 7f7269f94700 1 -- 172.21.15.2:0/1236067643 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f726410bc90 con 0x7f72641014f0 2023-11-16T16:49:08.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.060+0000 7f7269f94700 1 -- 172.21.15.2:0/1236067643 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f726410bb00 con 0x7f72641014f0 2023-11-16T16:49:08.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.060+0000 7f7260ff9700 1 -- 172.21.15.2:0/1236067643 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7258021410 con 0x7f72641014f0 2023-11-16T16:49:08.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.060+0000 7f7260ff9700 1 -- 172.21.15.2:0/1236067643 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7258005720 con 0x7f72641014f0 2023-11-16T16:49:08.063 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.061+0000 7f7269f94700 1 -- 172.21.15.2:0/1236067643 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7250005320 con 0x7f72641014f0 2023-11-16T16:49:08.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.063+0000 7f7260ff9700 1 -- 172.21.15.2:0/1236067643 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f725800fad0 con 0x7f72641014f0 2023-11-16T16:49:08.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.064+0000 7f7260ff9700 1 --2- 172.21.15.2:0/1236067643 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f724c0a1260 0x7f724c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:08.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.064+0000 7f7260ff9700 1 -- 172.21.15.2:0/1236067643 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f72580ce1b0 con 0x7f72641014f0 2023-11-16T16:49:08.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.064+0000 7f72637fe700 1 --2- 172.21.15.2:0/1236067643 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f724c0a1260 0x7f724c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:08.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.065+0000 7f72637fe700 1 --2- 172.21.15.2:0/1236067643 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f724c0a1260 0x7f724c0a3710 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7254005950 tx=0x7f7254009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:08.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.066+0000 7f7260ff9700 1 -- 172.21.15.2:0/1236067643 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7258097470 con 0x7f72641014f0 2023-11-16T16:49:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.282+0000 7f7269f94700 1 -- 172.21.15.2:0/1236067643 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7250000bf0 con 0x7f724c0a1260 2023-11-16T16:49:08.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.288+0000 7f7260ff9700 1 -- 172.21.15.2:0/1236067643 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f7250000bf0 con 0x7f724c0a1260 2023-11-16T16:49:08.290 INFO:teuthology.orchestra.run.smithi002.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-11-16T16:49:08.290 INFO:teuthology.orchestra.run.smithi002.stdout:mds.all 12/12 36s ago 2m smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi141=b;smithi141=e;smithi141=h;smithi141=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2023-11-16T16:49:08.290 INFO:teuthology.orchestra.run.smithi002.stdout:mgr 3/3 36s ago 7m smithi002=x;smithi141=y;smithi155=z;count:3 2023-11-16T16:49:08.290 INFO:teuthology.orchestra.run.smithi002.stdout:mon 3/3 36s ago 7m smithi002:172.21.15.2=a;smithi141:172.21.15.141=b;smithi155:172.21.15.155=c;count:3 2023-11-16T16:49:08.290 INFO:teuthology.orchestra.run.smithi002.stdout:osd 12 36s ago - 2023-11-16T16:49:08.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.290+0000 7f7269f94700 1 -- 172.21.15.2:0/1236067643 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f724c0a1260 msgr2=0x7f724c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:08.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.290+0000 7f7269f94700 1 --2- 172.21.15.2:0/1236067643 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f724c0a1260 0x7f724c0a3710 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7254005950 tx=0x7f7254009260 comp rx=0 tx=0).stop 2023-11-16T16:49:08.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.290+0000 7f7269f94700 1 -- 172.21.15.2:0/1236067643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72641014f0 msgr2=0x7f72641987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:08.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.290+0000 7f7269f94700 1 --2- 172.21.15.2:0/1236067643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72641014f0 0x7f72641987b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7258009990 tx=0x7f7258009580 comp rx=0 tx=0).stop 2023-11-16T16:49:08.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.290+0000 7f7269f94700 1 -- 172.21.15.2:0/1236067643 shutdown_connections 2023-11-16T16:49:08.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.290+0000 7f7269f94700 1 --2- 172.21.15.2:0/1236067643 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f724c0a1260 0x7f724c0a3710 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:08.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.290+0000 7f7269f94700 1 --2- 172.21.15.2:0/1236067643 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7264101e80 0x7f726419f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:08.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.290+0000 7f7269f94700 1 --2- 172.21.15.2:0/1236067643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72641002f0 0x7f7264198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:08.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.290+0000 7f7269f94700 1 --2- 172.21.15.2:0/1236067643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72641014f0 0x7f72641987b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:08.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.290+0000 7f7269f94700 1 -- 172.21.15.2:0/1236067643 >> 172.21.15.2:0/1236067643 conn(0x7f72640fb900 msgr2=0x7f7264106e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:08.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.290+0000 7f7269f94700 1 -- 172.21.15.2:0/1236067643 shutdown_connections 2023-11-16T16:49:08.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:08.290+0000 7f7269f94700 1 -- 172.21.15.2:0/1236067643 wait complete. 2023-11-16T16:49:08.807 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch host ls' 2023-11-16T16:49:09.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:08 smithi002 ceph-mon[128712]: pgmap v335: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T16:49:09.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:08 smithi141 ceph-mon[131184]: pgmap v335: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T16:49:09.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:08 smithi155 ceph-mon[131607]: pgmap v335: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T16:49:09.674 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:49:10.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:09 smithi002 ceph-mon[128712]: from='client.25192 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:10.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:09 smithi141 ceph-mon[131184]: from='client.25192 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:10.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:09 smithi155 ceph-mon[131607]: from='client.25192 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:10.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:10 smithi002 ceph-mon[128712]: pgmap v336: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T16:49:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:10 smithi141 ceph-mon[131184]: pgmap v336: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T16:49:11.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.315+0000 7f81d25a7700 1 -- 172.21.15.2:0/2058504536 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81cc1034a0 msgr2=0x7f81cc105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:11.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.315+0000 7f81d25a7700 1 --2- 172.21.15.2:0/2058504536 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81cc1034a0 0x7f81cc105880 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0098f0 tx=0x7f81bc009c00 comp rx=0 tx=0).stop 2023-11-16T16:49:11.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.315+0000 7f81d25a7700 1 -- 172.21.15.2:0/2058504536 shutdown_connections 2023-11-16T16:49:11.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.315+0000 7f81d25a7700 1 --2- 172.21.15.2:0/2058504536 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81cc100b80 0x7f81cc102f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:11.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.315+0000 7f81d25a7700 1 --2- 172.21.15.2:0/2058504536 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81cc105dc0 0x7f81cc108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:11.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.315+0000 7f81d25a7700 1 --2- 172.21.15.2:0/2058504536 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81cc1034a0 0x7f81cc105880 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:11.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.315+0000 7f81d25a7700 1 -- 172.21.15.2:0/2058504536 >> 172.21.15.2:0/2058504536 conn(0x7f81cc0fa620 msgr2=0x7f81cc0fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:11.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.315+0000 7f81d25a7700 1 -- 172.21.15.2:0/2058504536 shutdown_connections 2023-11-16T16:49:11.318 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.315+0000 7f81d25a7700 1 -- 172.21.15.2:0/2058504536 wait complete. 2023-11-16T16:49:11.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.316+0000 7f81d25a7700 1 Processor -- start 2023-11-16T16:49:11.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.317+0000 7f81d25a7700 1 -- start start 2023-11-16T16:49:11.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.317+0000 7f81d25a7700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81cc100b80 0x7f81cc195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:11.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.317+0000 7f81d25a7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81cc1034a0 0x7f81cc196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:11.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.317+0000 7f81d25a7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81cc105dc0 0x7f81cc19d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:11.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.317+0000 7f81d25a7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc19da70 con 0x7f81cc1034a0 2023-11-16T16:49:11.320 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.317+0000 7f81d25a7700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc19dbb0 con 0x7f81cc100b80 2023-11-16T16:49:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.317+0000 7f81d25a7700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81cc19dd40 con 0x7f81cc105dc0 2023-11-16T16:49:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.317+0000 7f81d0da4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81cc1034a0 0x7f81cc196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.317+0000 7f81d0da4700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81cc1034a0 0x7f81cc196420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59468/0 (socket says 172.21.15.2:59468) 2023-11-16T16:49:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.317+0000 7f81d0da4700 1 -- 172.21.15.2:0/1306022582 learned_addr learned my addr 172.21.15.2:0/1306022582 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.317+0000 7f81d15a5700 1 --2- 172.21.15.2:0/1306022582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81cc100b80 0x7f81cc195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.317+0000 7f81d1da6700 1 --2- 172.21.15.2:0/1306022582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81cc105dc0 0x7f81cc19d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.318+0000 7f81d0da4700 1 -- 172.21.15.2:0/1306022582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81cc105dc0 msgr2=0x7f81cc19d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.318+0000 7f81d0da4700 1 --2- 172.21.15.2:0/1306022582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81cc105dc0 0x7f81cc19d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.318+0000 7f81d0da4700 1 -- 172.21.15.2:0/1306022582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81cc100b80 msgr2=0x7f81cc195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.318+0000 7f81d0da4700 1 --2- 172.21.15.2:0/1306022582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81cc100b80 0x7f81cc195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.318+0000 7f81d0da4700 1 -- 172.21.15.2:0/1306022582 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c4009c40 con 0x7f81cc1034a0 2023-11-16T16:49:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.318+0000 7f81d1da6700 1 --2- 172.21.15.2:0/1306022582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81cc105dc0 0x7f81cc19d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:49:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.318+0000 7f81d0da4700 1 --2- 172.21.15.2:0/1306022582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81cc1034a0 0x7f81cc196420 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009230 tx=0x7f81bc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.318+0000 7f81c27fc700 1 -- 172.21.15.2:0/1306022582 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc01c070 con 0x7f81cc1034a0 2023-11-16T16:49:11.322 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.318+0000 7f81c27fc700 1 -- 172.21.15.2:0/1306022582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f81bc00f7d0 con 0x7f81cc1034a0 2023-11-16T16:49:11.322 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.318+0000 7f81d25a7700 1 -- 172.21.15.2:0/1306022582 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81bc009520 con 0x7f81cc1034a0 2023-11-16T16:49:11.322 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.318+0000 7f81c27fc700 1 -- 172.21.15.2:0/1306022582 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc004d70 con 0x7f81cc1034a0 2023-11-16T16:49:11.322 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.318+0000 7f81d25a7700 1 -- 172.21.15.2:0/1306022582 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81cc10bb10 con 0x7f81cc1034a0 2023-11-16T16:49:11.322 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.320+0000 7f81d25a7700 1 -- 172.21.15.2:0/1306022582 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81cc10bca0 con 0x7f81cc1034a0 2023-11-16T16:49:11.323 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.321+0000 7f81c27fc700 1 -- 172.21.15.2:0/1306022582 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f81bc005590 con 0x7f81cc1034a0 2023-11-16T16:49:11.324 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.323+0000 7f81c27fc700 1 --2- 172.21.15.2:0/1306022582 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f81b80a1260 0x7f81b80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:11.324 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.323+0000 7f81c27fc700 1 -- 172.21.15.2:0/1306022582 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f81bc0ce1b0 con 0x7f81cc1034a0 2023-11-16T16:49:11.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.323+0000 7f81d15a5700 1 --2- 172.21.15.2:0/1306022582 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f81b80a1260 0x7f81b80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:11.325 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.324+0000 7f81d15a5700 1 --2- 172.21.15.2:0/1306022582 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f81b80a1260 0x7f81b80a3710 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f81c8005fd0 tx=0x7f81c8009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:11.329 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.327+0000 7f81c27fc700 1 -- 172.21.15.2:0/1306022582 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f81bc097470 con 0x7f81cc1034a0 2023-11-16T16:49:11.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:10 smithi155 ceph-mon[131607]: pgmap v336: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T16:49:11.542 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.540+0000 7f81d25a7700 1 -- 172.21.15.2:0/1306022582 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f81cc061900 con 0x7f81b80a1260 2023-11-16T16:49:11.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.542+0000 7f81c27fc700 1 -- 172.21.15.2:0/1306022582 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f81cc061900 con 0x7f81b80a1260 2023-11-16T16:49:11.543 INFO:teuthology.orchestra.run.smithi002.stdout:HOST ADDR LABELS STATUS 2023-11-16T16:49:11.543 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 172.21.15.2 2023-11-16T16:49:11.544 INFO:teuthology.orchestra.run.smithi002.stdout:smithi141 172.21.15.141 2023-11-16T16:49:11.544 INFO:teuthology.orchestra.run.smithi002.stdout:smithi155 172.21.15.155 2023-11-16T16:49:11.544 INFO:teuthology.orchestra.run.smithi002.stdout:3 hosts in cluster 2023-11-16T16:49:11.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.544+0000 7f81d25a7700 1 -- 172.21.15.2:0/1306022582 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f81b80a1260 msgr2=0x7f81b80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:11.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.544+0000 7f81d25a7700 1 --2- 172.21.15.2:0/1306022582 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f81b80a1260 0x7f81b80a3710 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f81c8005fd0 tx=0x7f81c8009310 comp rx=0 tx=0).stop 2023-11-16T16:49:11.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.544+0000 7f81d25a7700 1 -- 172.21.15.2:0/1306022582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81cc1034a0 msgr2=0x7f81cc196420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:11.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.544+0000 7f81d25a7700 1 --2- 172.21.15.2:0/1306022582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81cc1034a0 0x7f81cc196420 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009230 tx=0x7f81bc00b840 comp rx=0 tx=0).stop 2023-11-16T16:49:11.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.544+0000 7f81d25a7700 1 -- 172.21.15.2:0/1306022582 shutdown_connections 2023-11-16T16:49:11.546 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.544+0000 7f81d25a7700 1 --2- 172.21.15.2:0/1306022582 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f81b80a1260 0x7f81b80a3710 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:11.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.544+0000 7f81d25a7700 1 --2- 172.21.15.2:0/1306022582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81cc105dc0 0x7f81cc19d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:11.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.544+0000 7f81d25a7700 1 --2- 172.21.15.2:0/1306022582 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81cc100b80 0x7f81cc195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:11.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.544+0000 7f81d25a7700 1 --2- 172.21.15.2:0/1306022582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81cc1034a0 0x7f81cc196420 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:11.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.544+0000 7f81d25a7700 1 -- 172.21.15.2:0/1306022582 >> 172.21.15.2:0/1306022582 conn(0x7f81cc0fa620 msgr2=0x7f81cc1017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:11.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.544+0000 7f81d25a7700 1 -- 172.21.15.2:0/1306022582 shutdown_connections 2023-11-16T16:49:11.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:11.544+0000 7f81d25a7700 1 -- 172.21.15.2:0/1306022582 wait complete. 2023-11-16T16:49:11.972 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph orch device ls' 2023-11-16T16:49:12.835 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:49:13.140 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:12 smithi002 ceph-mon[128712]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:13.140 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:12 smithi002 ceph-mon[128712]: pgmap v337: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T16:49:13.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:12 smithi141 ceph-mon[131184]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:13.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:12 smithi141 ceph-mon[131184]: pgmap v337: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T16:49:13.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:12 smithi155 ceph-mon[131607]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:13.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:12 smithi155 ceph-mon[131607]: pgmap v337: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T16:49:14.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.424+0000 7f35a17a7700 1 -- 172.21.15.2:0/4291785295 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c1002f0 msgr2=0x7f359c100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:14.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.424+0000 7f35a17a7700 1 --2- 172.21.15.2:0/4291785295 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c1002f0 0x7f359c100700 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f358c009990 tx=0x7f358c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.425+0000 7f35a17a7700 1 -- 172.21.15.2:0/4291785295 shutdown_connections 2023-11-16T16:49:14.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.425+0000 7f35a17a7700 1 --2- 172.21.15.2:0/4291785295 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f359c101e80 0x7f359c108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.425+0000 7f35a17a7700 1 --2- 172.21.15.2:0/4291785295 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c1002f0 0x7f359c100700 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.425+0000 7f35a17a7700 1 --2- 172.21.15.2:0/4291785295 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f359c1014f0 0x7f359c101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.425+0000 7f35a17a7700 1 -- 172.21.15.2:0/4291785295 >> 172.21.15.2:0/4291785295 conn(0x7f359c0fb900 msgr2=0x7f359c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:14.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.425+0000 7f35a17a7700 1 -- 172.21.15.2:0/4291785295 shutdown_connections 2023-11-16T16:49:14.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.425+0000 7f35a17a7700 1 -- 172.21.15.2:0/4291785295 wait complete. 2023-11-16T16:49:14.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.426+0000 7f35a17a7700 1 Processor -- start 2023-11-16T16:49:14.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.426+0000 7f35a17a7700 1 -- start start 2023-11-16T16:49:14.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.427+0000 7f35a17a7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c1002f0 0x7f359c1960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:14.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.427+0000 7f35a17a7700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f359c1014f0 0x7f359c1965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:14.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.427+0000 7f35a17a7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f359c101e80 0x7f359c19d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:14.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.427+0000 7f35a17a7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c19dc30 con 0x7f359c1002f0 2023-11-16T16:49:14.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.427+0000 7f35a17a7700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c19dd70 con 0x7f359c1014f0 2023-11-16T16:49:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.427+0000 7f35a17a7700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c19df00 con 0x7f359c101e80 2023-11-16T16:49:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.427+0000 7f359affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c1002f0 0x7f359c1960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.427+0000 7f359affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c1002f0 0x7f359c1960a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45028/0 (socket says 172.21.15.2:45028) 2023-11-16T16:49:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.427+0000 7f359affd700 1 -- 172.21.15.2:0/1284108124 learned_addr learned my addr 172.21.15.2:0/1284108124 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.427+0000 7f359a7fc700 1 --2- 172.21.15.2:0/1284108124 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f359c1014f0 0x7f359c1965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.427+0000 7f359b7fe700 1 --2- 172.21.15.2:0/1284108124 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f359c101e80 0x7f359c19d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.428+0000 7f359affd700 1 -- 172.21.15.2:0/1284108124 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f359c101e80 msgr2=0x7f359c19d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.428+0000 7f359affd700 1 --2- 172.21.15.2:0/1284108124 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f359c101e80 0x7f359c19d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.428+0000 7f359affd700 1 -- 172.21.15.2:0/1284108124 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f359c1014f0 msgr2=0x7f359c1965e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.428+0000 7f359affd700 1 --2- 172.21.15.2:0/1284108124 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f359c1014f0 0x7f359c1965e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.428+0000 7f359affd700 1 -- 172.21.15.2:0/1284108124 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c0095f0 con 0x7f359c1002f0 2023-11-16T16:49:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.428+0000 7f359affd700 1 --2- 172.21.15.2:0/1284108124 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c1002f0 0x7f359c1960a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f358c004740 tx=0x7f358c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:14.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.428+0000 7f3583fff700 1 -- 172.21.15.2:0/1284108124 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c01d070 con 0x7f359c1002f0 2023-11-16T16:49:14.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.428+0000 7f35a17a7700 1 -- 172.21.15.2:0/1284108124 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359c10bcb0 con 0x7f359c1002f0 2023-11-16T16:49:14.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.428+0000 7f3583fff700 1 -- 172.21.15.2:0/1284108124 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f358c021410 con 0x7f359c1002f0 2023-11-16T16:49:14.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.428+0000 7f3583fff700 1 -- 172.21.15.2:0/1284108124 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c0175f0 con 0x7f359c1002f0 2023-11-16T16:49:14.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.428+0000 7f35a17a7700 1 -- 172.21.15.2:0/1284108124 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359c10bb20 con 0x7f359c1002f0 2023-11-16T16:49:14.431 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.430+0000 7f35a17a7700 1 -- 172.21.15.2:0/1284108124 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359c107940 con 0x7f359c1002f0 2023-11-16T16:49:14.432 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.431+0000 7f3583fff700 1 -- 172.21.15.2:0/1284108124 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 135765+0+0 (secure 0 0 0) 0x7f358c021580 con 0x7f359c1002f0 2023-11-16T16:49:14.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.433+0000 7f3583fff700 1 --2- 172.21.15.2:0/1284108124 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f35840a11e0 0x7f35840a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:14.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.433+0000 7f3583fff700 1 -- 172.21.15.2:0/1284108124 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f358c0cdf50 con 0x7f359c1002f0 2023-11-16T16:49:14.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.433+0000 7f359a7fc700 1 --2- 172.21.15.2:0/1284108124 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f35840a11e0 0x7f35840a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:14.434 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.434+0000 7f359a7fc700 1 --2- 172.21.15.2:0/1284108124 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f35840a11e0 0x7f35840a3690 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3590009c60 tx=0x7f3590009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:14.437 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.437+0000 7f3583fff700 1 -- 172.21.15.2:0/1284108124 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f358c097330 con 0x7f359c1002f0 2023-11-16T16:49:14.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.653+0000 7f35a17a7700 1 -- 172.21.15.2:0/1284108124 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f359c061c80 con 0x7f35840a11e0 2023-11-16T16:49:14.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.655+0000 7f3583fff700 1 -- 172.21.15.2:0/1284108124 <== mgr.14150 v2:172.21.15.2:6800/1793067160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7f359c061c80 con 0x7f35840a11e0 2023-11-16T16:49:14.656 INFO:teuthology.orchestra.run.smithi002.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-11-16T16:49:14.656 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300EV480DGN 447G No 3m ago LVM detected, locked 2023-11-16T16:49:14.656 INFO:teuthology.orchestra.run.smithi002.stdout:smithi141 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6242001S400BGN 372G No 3m ago LVM detected, locked 2023-11-16T16:49:14.656 INFO:teuthology.orchestra.run.smithi002.stdout:smithi155 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GW400BGN 372G No 2m ago LVM detected, locked 2023-11-16T16:49:14.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.657+0000 7f35a17a7700 1 -- 172.21.15.2:0/1284108124 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f35840a11e0 msgr2=0x7f35840a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:14.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.657+0000 7f35a17a7700 1 --2- 172.21.15.2:0/1284108124 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f35840a11e0 0x7f35840a3690 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3590009c60 tx=0x7f3590009260 comp rx=0 tx=0).stop 2023-11-16T16:49:14.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.657+0000 7f35a17a7700 1 -- 172.21.15.2:0/1284108124 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c1002f0 msgr2=0x7f359c1960a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:14.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.657+0000 7f35a17a7700 1 --2- 172.21.15.2:0/1284108124 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c1002f0 0x7f359c1960a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f358c004740 tx=0x7f358c009580 comp rx=0 tx=0).stop 2023-11-16T16:49:14.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.657+0000 7f35a17a7700 1 -- 172.21.15.2:0/1284108124 shutdown_connections 2023-11-16T16:49:14.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.657+0000 7f35a17a7700 1 --2- 172.21.15.2:0/1284108124 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f35840a11e0 0x7f35840a3690 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.657+0000 7f35a17a7700 1 --2- 172.21.15.2:0/1284108124 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f359c101e80 0x7f359c19d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.657+0000 7f35a17a7700 1 --2- 172.21.15.2:0/1284108124 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359c1002f0 0x7f359c1960a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.657+0000 7f35a17a7700 1 --2- 172.21.15.2:0/1284108124 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f359c1014f0 0x7f359c1965e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:14.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.657+0000 7f35a17a7700 1 -- 172.21.15.2:0/1284108124 >> 172.21.15.2:0/1284108124 conn(0x7f359c0fb900 msgr2=0x7f359c104a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:14.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.657+0000 7f35a17a7700 1 -- 172.21.15.2:0/1284108124 shutdown_connections 2023-11-16T16:49:14.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:14.658+0000 7f35a17a7700 1 -- 172.21.15.2:0/1284108124 wait complete. 2023-11-16T16:49:15.092 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:14 smithi002 ceph-mon[128712]: pgmap v338: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:49:15.165 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-11-16T16:49:15.174 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T16:49:15.174 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph fs dump' 2023-11-16T16:49:15.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:14 smithi141 ceph-mon[131184]: pgmap v338: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:49:15.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:14 smithi155 ceph-mon[131607]: pgmap v338: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:49:15.940 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:49:16.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:15 smithi002 ceph-mon[128712]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:16.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:15 smithi002 ceph-mon[128712]: mgrmap e18: x(active, since 8m), standbys: z, y 2023-11-16T16:49:16.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:15 smithi141 ceph-mon[131184]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:16.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:15 smithi141 ceph-mon[131184]: mgrmap e18: x(active, since 8m), standbys: z, y 2023-11-16T16:49:16.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:15 smithi155 ceph-mon[131607]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:16.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:15 smithi155 ceph-mon[131607]: mgrmap e18: x(active, since 8m), standbys: z, y 2023-11-16T16:49:17.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:16 smithi002 ceph-mon[128712]: pgmap v339: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2023-11-16T16:49:17.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:16 smithi141 ceph-mon[131184]: pgmap v339: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2023-11-16T16:49:17.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:16 smithi155 ceph-mon[131607]: pgmap v339: 129 pgs: 129 active+clean; 656 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2023-11-16T16:49:17.468 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.466+0000 7fa72b67c700 1 -- 172.21.15.2:0/3274771785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa724101f00 msgr2=0x7fa7241087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:17.468 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.466+0000 7fa72b67c700 1 --2- 172.21.15.2:0/3274771785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa724101f00 0x7fa7241087c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa71c009f90 tx=0x7fa71c00b330 comp rx=0 tx=0).stop 2023-11-16T16:49:17.468 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.466+0000 7fa72b67c700 1 -- 172.21.15.2:0/3274771785 shutdown_connections 2023-11-16T16:49:17.468 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.466+0000 7fa72b67c700 1 --2- 172.21.15.2:0/3274771785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa724100370 0x7fa724100780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.468 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.466+0000 7fa72b67c700 1 --2- 172.21.15.2:0/3274771785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa724101f00 0x7fa7241087c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.466+0000 7fa72b67c700 1 --2- 172.21.15.2:0/3274771785 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa724101570 0x7fa7241019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.466+0000 7fa72b67c700 1 -- 172.21.15.2:0/3274771785 >> 172.21.15.2:0/3274771785 conn(0x7fa7240fb9a0 msgr2=0x7fa7240fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:17.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.467+0000 7fa72b67c700 1 -- 172.21.15.2:0/3274771785 shutdown_connections 2023-11-16T16:49:17.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.467+0000 7fa72b67c700 1 -- 172.21.15.2:0/3274771785 wait complete. 2023-11-16T16:49:17.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.468+0000 7fa72b67c700 1 Processor -- start 2023-11-16T16:49:17.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.468+0000 7fa72b67c700 1 -- start start 2023-11-16T16:49:17.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.468+0000 7fa72b67c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa724100370 0x7fa724198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:17.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.468+0000 7fa72b67c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa724101570 0x7fa724198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:17.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.468+0000 7fa72b67c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa724101f00 0x7fa72419f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:17.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.468+0000 7fa72b67c700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72419fec0 con 0x7fa724101f00 2023-11-16T16:49:17.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.468+0000 7fa72b67c700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7241a0000 con 0x7fa724101570 2023-11-16T16:49:17.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.468+0000 7fa72b67c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7241a0190 con 0x7fa724100370 2023-11-16T16:49:17.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.469+0000 7fa729c19700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa724101f00 0x7fa72419f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:17.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.469+0000 7fa728c17700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa724101570 0x7fa724198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:17.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.469+0000 7fa729418700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa724100370 0x7fa724198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.469+0000 7fa728c17700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa724101570 0x7fa724198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:44952/0 (socket says 172.21.15.2:44952) 2023-11-16T16:49:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.469+0000 7fa729c19700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa724101f00 0x7fa72419f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45058/0 (socket says 172.21.15.2:45058) 2023-11-16T16:49:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.469+0000 7fa728c17700 1 -- 172.21.15.2:0/3024667457 learned_addr learned my addr 172.21.15.2:0/3024667457 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.469+0000 7fa729c19700 1 -- 172.21.15.2:0/3024667457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa724100370 msgr2=0x7fa724198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.469+0000 7fa729c19700 1 --2- 172.21.15.2:0/3024667457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa724100370 0x7fa724198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.469+0000 7fa729c19700 1 -- 172.21.15.2:0/3024667457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa724101570 msgr2=0x7fa724198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:17.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.469+0000 7fa729c19700 1 --2- 172.21.15.2:0/3024667457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa724101570 0x7fa724198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.469+0000 7fa729c19700 1 -- 172.21.15.2:0/3024667457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa720009520 con 0x7fa724101f00 2023-11-16T16:49:17.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.469+0000 7fa729c19700 1 --2- 172.21.15.2:0/3024667457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa724101f00 0x7fa72419f8f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa71c009f90 tx=0x7fa71c004c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:17.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.469+0000 7fa71a7fc700 1 -- 172.21.15.2:0/3024667457 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71c00e040 con 0x7fa724101f00 2023-11-16T16:49:17.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.470+0000 7fa72b67c700 1 -- 172.21.15.2:0/3024667457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7140095f0 con 0x7fa724101f00 2023-11-16T16:49:17.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.470+0000 7fa71a7fc700 1 -- 172.21.15.2:0/3024667457 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa71c00bbf0 con 0x7fa724101f00 2023-11-16T16:49:17.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.470+0000 7fa72b67c700 1 -- 172.21.15.2:0/3024667457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa71c009c40 con 0x7fa724101f00 2023-11-16T16:49:17.475 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.470+0000 7fa71a7fc700 1 -- 172.21.15.2:0/3024667457 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71c005240 con 0x7fa724101f00 2023-11-16T16:49:17.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.471+0000 7fa72b67c700 1 -- 172.21.15.2:0/3024667457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa72410bb80 con 0x7fa724101f00 2023-11-16T16:49:17.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.473+0000 7fa71a7fc700 1 -- 172.21.15.2:0/3024667457 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fa71c01d470 con 0x7fa724101f00 2023-11-16T16:49:17.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.474+0000 7fa71a7fc700 1 --2- 172.21.15.2:0/3024667457 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa7100a1150 0x7fa7100a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:17.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.474+0000 7fa71a7fc700 1 -- 172.21.15.2:0/3024667457 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fa71c0cdd50 con 0x7fa724101f00 2023-11-16T16:49:17.476 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.474+0000 7fa729418700 1 --2- 172.21.15.2:0/3024667457 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa7100a1150 0x7fa7100a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:17.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.475+0000 7fa729418700 1 --2- 172.21.15.2:0/3024667457 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa7100a1150 0x7fa7100a3600 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa7241013d0 tx=0x7fa720009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:17.479 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.478+0000 7fa71a7fc700 1 -- 172.21.15.2:0/3024667457 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa71c0971b0 con 0x7fa724101f00 2023-11-16T16:49:17.732 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.732+0000 7fa72b67c700 1 -- 172.21.15.2:0/3024667457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa72410bd70 con 0x7fa724101f00 2023-11-16T16:49:17.733 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.732+0000 7fa71a7fc700 1 -- 172.21.15.2:0/3024667457 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+2920 (secure 0 0 0) 0x7fa71c09a7d0 con 0x7fa724101f00 2023-11-16T16:49:17.733 INFO:teuthology.orchestra.run.smithi002.stdout:e26 2023-11-16T16:49:17.733 INFO:teuthology.orchestra.run.smithi002.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-11-16T16:49:17.733 INFO:teuthology.orchestra.run.smithi002.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T16:49:17.734 INFO:teuthology.orchestra.run.smithi002.stdout:legacy client fscid: 1 2023-11-16T16:49:17.734 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:49:17.734 INFO:teuthology.orchestra.run.smithi002.stdout:Filesystem 'cephfs' (1) 2023-11-16T16:49:17.734 INFO:teuthology.orchestra.run.smithi002.stdout:fs_name cephfs 2023-11-16T16:49:17.734 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 26 2023-11-16T16:49:17.734 INFO:teuthology.orchestra.run.smithi002.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2023-11-16T16:49:17.734 INFO:teuthology.orchestra.run.smithi002.stdout:created 2023-11-16T16:47:26.349176+0000 2023-11-16T16:49:17.734 INFO:teuthology.orchestra.run.smithi002.stdout:modified 2023-11-16T16:48:05.774072+0000 2023-11-16T16:49:17.734 INFO:teuthology.orchestra.run.smithi002.stdout:tableserver 0 2023-11-16T16:49:17.734 INFO:teuthology.orchestra.run.smithi002.stdout:root 0 2023-11-16T16:49:17.734 INFO:teuthology.orchestra.run.smithi002.stdout:session_timeout 300 2023-11-16T16:49:17.734 INFO:teuthology.orchestra.run.smithi002.stdout:session_autoclose 300 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi002.stdout:max_file_size 1099511627776 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi002.stdout:max_xattr_size 65536 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi002.stdout:required_client_features {} 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure 0 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure_osd_epoch 0 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi002.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi002.stdout:max_mds 3 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi002.stdout:in 0,1,2 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi002.stdout:up {0=24590,1=24544,2=24542} 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi002.stdout:failed 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi002.stdout:damaged 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi002.stdout:stopped 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi002.stdout:data_pools [3,4] 2023-11-16T16:49:17.735 INFO:teuthology.orchestra.run.smithi002.stdout:metadata_pool 2 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi002.stdout:inline_data disabled 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi002.stdout:balancer 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi002.stdout:bal_rank_mask -1 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi002.stdout:standby_count_wanted 0 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.b{0:24590} state up:active seq 3 export targets 1,2 addr [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.c{0:24523} state up:standby-replay seq 1 addr [v2:172.21.15.155:6836/1607756751,v1:172.21.15.155:6837/1607756751] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.i{1:24544} state up:active seq 11 addr [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.f{1:24511} state up:standby-replay seq 1 addr [v2:172.21.15.155:6834/2527313191,v1:172.21.15.155:6835/2527313191] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.h{2:24542} state up:active seq 12 addr [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.e{2:24506} state up:standby-replay seq 1 addr [v2:172.21.15.141:6834/3516869188,v1:172.21.15.141:6835/3516869188] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi002.stdout:Standby daemons: 2023-11-16T16:49:17.736 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:49:17.737 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.j{-1:14469} state up:standby seq 1 addr [v2:172.21.15.2:6834/4133614271,v1:172.21.15.2:6835/4133614271] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:17.737 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.d{-1:14481} state up:standby seq 1 addr [v2:172.21.15.2:6836/1610211225,v1:172.21.15.2:6837/1610211225] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:17.737 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.g{-1:14538} state up:standby seq 1 addr [v2:172.21.15.2:6838/2583261818,v1:172.21.15.2:6839/2583261818] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:17.737 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.a{-1:14550} state up:standby seq 1 addr [v2:172.21.15.2:6840/3170991923,v1:172.21.15.2:6841/3170991923] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:17.737 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.l{-1:24454} state up:standby seq 1 addr [v2:172.21.15.155:6832/4280048431,v1:172.21.15.155:6833/4280048431] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:17.737 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.k{-1:24470} state up:standby seq 1 addr [v2:172.21.15.141:6832/3738610702,v1:172.21.15.141:6833/3738610702] compat {c=[1],r=[1],i=[7ff]}] 2023-11-16T16:49:17.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.734+0000 7fa72b67c700 1 -- 172.21.15.2:0/3024667457 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa7100a1150 msgr2=0x7fa7100a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:17.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.734+0000 7fa72b67c700 1 --2- 172.21.15.2:0/3024667457 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa7100a1150 0x7fa7100a3600 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa7241013d0 tx=0x7fa720009260 comp rx=0 tx=0).stop 2023-11-16T16:49:17.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.734+0000 7fa72b67c700 1 -- 172.21.15.2:0/3024667457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa724101f00 msgr2=0x7fa72419f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:17.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.734+0000 7fa72b67c700 1 --2- 172.21.15.2:0/3024667457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa724101f00 0x7fa72419f8f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa71c009f90 tx=0x7fa71c004c60 comp rx=0 tx=0).stop 2023-11-16T16:49:17.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.735+0000 7fa72b67c700 1 -- 172.21.15.2:0/3024667457 shutdown_connections 2023-11-16T16:49:17.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.735+0000 7fa72b67c700 1 --2- 172.21.15.2:0/3024667457 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa7100a1150 0x7fa7100a3600 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.735+0000 7fa72b67c700 1 --2- 172.21.15.2:0/3024667457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa724100370 0x7fa724198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.735+0000 7fa72b67c700 1 --2- 172.21.15.2:0/3024667457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa724101f00 0x7fa72419f8f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.735+0000 7fa72b67c700 1 --2- 172.21.15.2:0/3024667457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa724101570 0x7fa724198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:17.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.735+0000 7fa72b67c700 1 -- 172.21.15.2:0/3024667457 >> 172.21.15.2:0/3024667457 conn(0x7fa7240fb9a0 msgr2=0x7fa7240fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:17.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.735+0000 7fa72b67c700 1 -- 172.21.15.2:0/3024667457 shutdown_connections 2023-11-16T16:49:17.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:17.735+0000 7fa72b67c700 1 -- 172.21.15.2:0/3024667457 wait complete. 2023-11-16T16:49:17.739 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 26 2023-11-16T16:49:18.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:17 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3024667457' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-11-16T16:49:18.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:17 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3024667457' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-11-16T16:49:18.340 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- bash -c 'ceph osd dump' 2023-11-16T16:49:18.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:17 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3024667457' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-11-16T16:49:19.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:18 smithi002 ceph-mon[128712]: pgmap v340: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 6 op/s 2023-11-16T16:49:19.205 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:49:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:18 smithi141 ceph-mon[131184]: pgmap v340: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 6 op/s 2023-11-16T16:49:19.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:18 smithi155 ceph-mon[131607]: pgmap v340: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 6 op/s 2023-11-16T16:49:20.764 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:20 smithi141 ceph-mon[131184]: pgmap v341: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2023-11-16T16:49:20.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:20 smithi155 ceph-mon[131607]: pgmap v341: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2023-11-16T16:49:20.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.872+0000 7fccebc65700 1 -- 172.21.15.2:0/161869857 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcce41002f0 msgr2=0x7fcce4100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:20.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.872+0000 7fccebc65700 1 --2- 172.21.15.2:0/161869857 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcce41002f0 0x7fcce4100700 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcce0009990 tx=0x7fcce0009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.873 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.872+0000 7fccebc65700 1 -- 172.21.15.2:0/161869857 shutdown_connections 2023-11-16T16:49:20.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.872+0000 7fccebc65700 1 --2- 172.21.15.2:0/161869857 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcce4101e80 0x7fcce4108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.872+0000 7fccebc65700 1 --2- 172.21.15.2:0/161869857 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcce41002f0 0x7fcce4100700 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.872+0000 7fccebc65700 1 --2- 172.21.15.2:0/161869857 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcce41014f0 0x7fcce4101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.872+0000 7fccebc65700 1 -- 172.21.15.2:0/161869857 >> 172.21.15.2:0/161869857 conn(0x7fcce40fb900 msgr2=0x7fcce40fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:20.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.872+0000 7fccebc65700 1 -- 172.21.15.2:0/161869857 shutdown_connections 2023-11-16T16:49:20.874 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.872+0000 7fccebc65700 1 -- 172.21.15.2:0/161869857 wait complete. 2023-11-16T16:49:20.875 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.873+0000 7fccebc65700 1 Processor -- start 2023-11-16T16:49:20.875 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.874+0000 7fccebc65700 1 -- start start 2023-11-16T16:49:20.875 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.874+0000 7fccebc65700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcce41002f0 0x7fcce4074890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.875 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.874+0000 7fccebc65700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcce41014f0 0x7fcce4072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.875 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.874+0000 7fccebc65700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcce4101e80 0x7fcce4073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.874+0000 7fccebc65700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce4199f00 con 0x7fcce41014f0 2023-11-16T16:49:20.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.874+0000 7fccebc65700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce4073960 con 0x7fcce4101e80 2023-11-16T16:49:20.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.874+0000 7fccebc65700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce4073b20 con 0x7fcce41002f0 2023-11-16T16:49:20.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.874+0000 7fcce9200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcce41014f0 0x7fcce4072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.874+0000 7fccea202700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcce4101e80 0x7fcce4073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.874+0000 7fcce9200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcce41014f0 0x7fcce4072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45076/0 (socket says 172.21.15.2:45076) 2023-11-16T16:49:20.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.874+0000 7fcce9200700 1 -- 172.21.15.2:0/1924350947 learned_addr learned my addr 172.21.15.2:0/1924350947 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:20.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.874+0000 7fcce9a01700 1 --2- 172.21.15.2:0/1924350947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcce41002f0 0x7fcce4074890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.875+0000 7fcce9200700 1 -- 172.21.15.2:0/1924350947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcce41002f0 msgr2=0x7fcce4074890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:20.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.875+0000 7fcce9200700 1 --2- 172.21.15.2:0/1924350947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcce41002f0 0x7fcce4074890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.875+0000 7fcce9200700 1 -- 172.21.15.2:0/1924350947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcce4101e80 msgr2=0x7fcce4073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:20.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.875+0000 7fcce9200700 1 --2- 172.21.15.2:0/1924350947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcce4101e80 0x7fcce4073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:20.876 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.875+0000 7fcce9200700 1 -- 172.21.15.2:0/1924350947 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce00095f0 con 0x7fcce41014f0 2023-11-16T16:49:20.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.875+0000 7fccea202700 1 --2- 172.21.15.2:0/1924350947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcce4101e80 0x7fcce4073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:20.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.875+0000 7fcce9200700 1 --2- 172.21.15.2:0/1924350947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcce41014f0 0x7fcce4072ee0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fccd400e9a0 tx=0x7fccd400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:20.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.875+0000 7fccdaffd700 1 -- 172.21.15.2:0/1924350947 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd4009790 con 0x7fcce41014f0 2023-11-16T16:49:20.877 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.875+0000 7fccdaffd700 1 -- 172.21.15.2:0/1924350947 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fccd4010430 con 0x7fcce41014f0 2023-11-16T16:49:20.878 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.875+0000 7fccebc65700 1 -- 172.21.15.2:0/1924350947 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce410bc90 con 0x7fcce41014f0 2023-11-16T16:49:20.878 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.875+0000 7fccdaffd700 1 -- 172.21.15.2:0/1924350947 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd4018680 con 0x7fcce41014f0 2023-11-16T16:49:20.878 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.875+0000 7fccebc65700 1 -- 172.21.15.2:0/1924350947 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce410bb00 con 0x7fcce41014f0 2023-11-16T16:49:20.878 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.876+0000 7fccebc65700 1 -- 172.21.15.2:0/1924350947 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce41a4bb0 con 0x7fcce41014f0 2023-11-16T16:49:20.879 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.878+0000 7fccdaffd700 1 -- 172.21.15.2:0/1924350947 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fccd40105a0 con 0x7fcce41014f0 2023-11-16T16:49:20.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.880+0000 7fccdaffd700 1 --2- 172.21.15.2:0/1924350947 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fccd00a1150 0x7fccd00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:20.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.880+0000 7fccdaffd700 1 -- 172.21.15.2:0/1924350947 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fccd4014070 con 0x7fcce41014f0 2023-11-16T16:49:20.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.880+0000 7fcce9a01700 1 --2- 172.21.15.2:0/1924350947 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fccd00a1150 0x7fccd00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:20.881 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.881+0000 7fcce9a01700 1 --2- 172.21.15.2:0/1924350947 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fccd00a1150 0x7fccd00a3600 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcce000b2e0 tx=0x7fcce0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:20.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:20.884+0000 7fccdaffd700 1 -- 172.21.15.2:0/1924350947 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fccd4051730 con 0x7fcce41014f0 2023-11-16T16:49:20.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:20 smithi002 ceph-mon[128712]: pgmap v341: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2023-11-16T16:49:21.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.094+0000 7fccebc65700 1 -- 172.21.15.2:0/1924350947 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7fcce4107280 con 0x7fcce41014f0 2023-11-16T16:49:21.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.095+0000 7fccdaffd700 1 -- 172.21.15.2:0/1924350947 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) v1 ==== 56+0+5677 (secure 0 0 0) 0x7fccd40d3080 con 0x7fcce41014f0 2023-11-16T16:49:21.096 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 93 2023-11-16T16:49:21.097 INFO:teuthology.orchestra.run.smithi002.stdout:fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 2023-11-16T16:49:21.097 INFO:teuthology.orchestra.run.smithi002.stdout:created 2023-11-16T16:39:43.029249+0000 2023-11-16T16:49:21.097 INFO:teuthology.orchestra.run.smithi002.stdout:modified 2023-11-16T16:48:45.491739+0000 2023-11-16T16:49:21.097 INFO:teuthology.orchestra.run.smithi002.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-11-16T16:49:21.097 INFO:teuthology.orchestra.run.smithi002.stdout:crush_version 27 2023-11-16T16:49:21.097 INFO:teuthology.orchestra.run.smithi002.stdout:full_ratio 0.95 2023-11-16T16:49:21.097 INFO:teuthology.orchestra.run.smithi002.stdout:backfillfull_ratio 0.9 2023-11-16T16:49:21.097 INFO:teuthology.orchestra.run.smithi002.stdout:nearfull_ratio 0.85 2023-11-16T16:49:21.097 INFO:teuthology.orchestra.run.smithi002.stdout:require_min_compat_client luminous 2023-11-16T16:49:21.097 INFO:teuthology.orchestra.run.smithi002.stdout:min_compat_client luminous 2023-11-16T16:49:21.097 INFO:teuthology.orchestra.run.smithi002.stdout:require_osd_release reef 2023-11-16T16:49:21.097 INFO:teuthology.orchestra.run.smithi002.stdout:stretch_mode_enabled false 2023-11-16T16:49:21.097 INFO:teuthology.orchestra.run.smithi002.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2023-11-16T16:49:21.098 INFO:teuthology.orchestra.run.smithi002.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2023-11-16T16:49:21.098 INFO:teuthology.orchestra.run.smithi002.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2023-11-16T16:49:21.098 INFO:teuthology.orchestra.run.smithi002.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2023-11-16T16:49:21.098 INFO:teuthology.orchestra.run.smithi002.stdout:max_osd 12 2023-11-16T16:49:21.098 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 up in weight 1 up_from 8 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6802/3417839294,v1:172.21.15.2:6803/3417839294] [v2:172.21.15.2:6804/3417839294,v1:172.21.15.2:6805/3417839294] exists,up bfde47f9-e428-42b2-a86a-d1a39a7f8084 2023-11-16T16:49:21.098 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 up in weight 1 up_from 13 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.141:6800/1040978026,v1:172.21.15.141:6801/1040978026] [v2:172.21.15.141:6802/1040978026,v1:172.21.15.141:6803/1040978026] exists,up fdef1dd6-b1c9-484c-9942-395e0213aa49 2023-11-16T16:49:21.098 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6800/844890823,v1:172.21.15.155:6801/844890823] [v2:172.21.15.155:6802/844890823,v1:172.21.15.155:6803/844890823] exists,up 1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae 2023-11-16T16:49:21.098 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6810/2757259480,v1:172.21.15.2:6811/2757259480] [v2:172.21.15.2:6812/2757259480,v1:172.21.15.2:6813/2757259480] exists,up 374aea7d-57ed-49b1-921a-c78d31478c37 2023-11-16T16:49:21.098 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.141:6808/1025476846,v1:172.21.15.141:6809/1025476846] [v2:172.21.15.141:6810/1025476846,v1:172.21.15.141:6811/1025476846] exists,up 4a4b549e-4eb4-4f67-829c-c37004403b76 2023-11-16T16:49:21.098 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 up in weight 1 up_from 38 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6808/2988032938,v1:172.21.15.155:6809/2988032938] [v2:172.21.15.155:6810/2988032938,v1:172.21.15.155:6811/2988032938] exists,up 35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd 2023-11-16T16:49:21.098 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6818/2881363640,v1:172.21.15.2:6819/2881363640] [v2:172.21.15.2:6820/2881363640,v1:172.21.15.2:6821/2881363640] exists,up e574f450-ac2d-445f-bcef-61ccdc6f6985 2023-11-16T16:49:21.098 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 up in weight 1 up_from 51 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.141:6816/372138580,v1:172.21.15.141:6817/372138580] [v2:172.21.15.141:6818/372138580,v1:172.21.15.141:6819/372138580] exists,up 7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46 2023-11-16T16:49:21.098 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8 up in weight 1 up_from 59 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6816/3808747539,v1:172.21.15.155:6817/3808747539] [v2:172.21.15.155:6818/3808747539,v1:172.21.15.155:6819/3808747539] exists,up fd6d5d76-9af2-49f8-a769-3ef0fb0599dc 2023-11-16T16:49:21.098 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9 up in weight 1 up_from 67 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6826/3506759823,v1:172.21.15.2:6827/3506759823] [v2:172.21.15.2:6828/3506759823,v1:172.21.15.2:6829/3506759823] exists,up 0a2692c3-2f49-4706-9a2b-0cb85882387d 2023-11-16T16:49:21.099 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10 up in weight 1 up_from 72 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.141:6824/1084001742,v1:172.21.15.141:6825/1084001742] [v2:172.21.15.141:6826/1084001742,v1:172.21.15.141:6827/1084001742] exists,up 4c9f53c3-c1bb-45ce-a05e-94d49b0c654f 2023-11-16T16:49:21.099 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11 up in weight 1 up_from 80 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6824/1791652726,v1:172.21.15.155:6825/1791652726] [v2:172.21.15.155:6826/1791652726,v1:172.21.15.155:6827/1791652726] exists,up 52f27807-1649-4c8d-9182-d161fa526d54 2023-11-16T16:49:21.099 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 2.0 [8,4] 2023-11-16T16:49:21.099 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 2.a [2,11] 2023-11-16T16:49:21.099 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 2.13 [2,11] 2023-11-16T16:49:21.099 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 2.14 [2,4] 2023-11-16T16:49:21.099 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.2 [3,9] 2023-11-16T16:49:21.099 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.6 [4,8] 2023-11-16T16:49:21.099 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.c [3,9] 2023-11-16T16:49:21.099 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.14 [4,6] 2023-11-16T16:49:21.099 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.31 [4,8] 2023-11-16T16:49:21.099 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.33 [5,9] 2023-11-16T16:49:21.099 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 4.7 [5,8] 2023-11-16T16:49:21.100 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 4.c [9,10] 2023-11-16T16:49:21.100 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 4.f [9,8] 2023-11-16T16:49:21.100 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 4.16 [9,8] 2023-11-16T16:49:21.100 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/2137541200 expires 2023-11-17T16:40:43.705118+0000 2023-11-16T16:49:21.100 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/2525877070 expires 2023-11-17T16:40:43.705118+0000 2023-11-16T16:49:21.100 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/30571981 expires 2023-11-17T16:40:16.538166+0000 2023-11-16T16:49:21.100 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6800/3359672074 expires 2023-11-17T16:40:43.705118+0000 2023-11-16T16:49:21.100 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/500255408 expires 2023-11-17T16:40:16.538166+0000 2023-11-16T16:49:21.100 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/981081063 expires 2023-11-17T16:40:43.705118+0000 2023-11-16T16:49:21.100 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6800/3453602765 expires 2023-11-17T16:40:16.538166+0000 2023-11-16T16:49:21.100 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6801/3359672074 expires 2023-11-17T16:40:43.705118+0000 2023-11-16T16:49:21.100 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6801/3453602765 expires 2023-11-17T16:40:16.538166+0000 2023-11-16T16:49:21.100 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/3818551797 expires 2023-11-17T16:40:16.538166+0000 2023-11-16T16:49:21.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.098+0000 7fccebc65700 1 -- 172.21.15.2:0/1924350947 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fccd00a1150 msgr2=0x7fccd00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:21.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.098+0000 7fccebc65700 1 --2- 172.21.15.2:0/1924350947 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fccd00a1150 0x7fccd00a3600 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcce000b2e0 tx=0x7fcce0005fb0 comp rx=0 tx=0).stop 2023-11-16T16:49:21.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.098+0000 7fccebc65700 1 -- 172.21.15.2:0/1924350947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcce41014f0 msgr2=0x7fcce4072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:21.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.098+0000 7fccebc65700 1 --2- 172.21.15.2:0/1924350947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcce41014f0 0x7fcce4072ee0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fccd400e9a0 tx=0x7fccd400ecb0 comp rx=0 tx=0).stop 2023-11-16T16:49:21.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.098+0000 7fccebc65700 1 -- 172.21.15.2:0/1924350947 shutdown_connections 2023-11-16T16:49:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.098+0000 7fccebc65700 1 --2- 172.21.15.2:0/1924350947 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fccd00a1150 0x7fccd00a3600 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.098+0000 7fccebc65700 1 --2- 172.21.15.2:0/1924350947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcce41002f0 0x7fcce4074890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.098+0000 7fccebc65700 1 --2- 172.21.15.2:0/1924350947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcce4101e80 0x7fcce4073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.098+0000 7fccebc65700 1 --2- 172.21.15.2:0/1924350947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcce41014f0 0x7fcce4072ee0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.098+0000 7fccebc65700 1 -- 172.21.15.2:0/1924350947 >> 172.21.15.2:0/1924350947 conn(0x7fcce40fb900 msgr2=0x7fcce40fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.098+0000 7fccebc65700 1 -- 172.21.15.2:0/1924350947 shutdown_connections 2023-11-16T16:49:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.098+0000 7fccebc65700 1 -- 172.21.15.2:0/1924350947 wait complete. 2023-11-16T16:49:21.607 INFO:teuthology.run_tasks:Running task fs.ready... 2023-11-16T16:49:21.619 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:49:21.764 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:21 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1924350947' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-11-16T16:49:21.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.828+0000 7f75347ff700 1 -- 172.21.15.2:0/1421550602 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f752c101480 msgr2=0x7f752c1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:21.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.828+0000 7f75347ff700 1 --2- 172.21.15.2:0/1421550602 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f752c101480 0x7f752c1018d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f751c009990 tx=0x7f751c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:21.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.828+0000 7f75347ff700 1 -- 172.21.15.2:0/1421550602 shutdown_connections 2023-11-16T16:49:21.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.828+0000 7f75347ff700 1 --2- 172.21.15.2:0/1421550602 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f752c100290 0x7f752c1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:21.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.828+0000 7f75347ff700 1 --2- 172.21.15.2:0/1421550602 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f752c101e10 0x7f752c1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:21.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.828+0000 7f75347ff700 1 --2- 172.21.15.2:0/1421550602 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f752c101480 0x7f752c1018d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:21.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.828+0000 7f75347ff700 1 -- 172.21.15.2:0/1421550602 >> 172.21.15.2:0/1421550602 conn(0x7f752c0fb870 msgr2=0x7f752c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:21.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.828+0000 7f75347ff700 1 -- 172.21.15.2:0/1421550602 shutdown_connections 2023-11-16T16:49:21.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.828+0000 7f75347ff700 1 -- 172.21.15.2:0/1421550602 wait complete. 2023-11-16T16:49:21.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.829+0000 7f75347ff700 1 Processor -- start 2023-11-16T16:49:21.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.829+0000 7f75347ff700 1 -- start start 2023-11-16T16:49:21.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.830+0000 7f75347ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f752c100290 0x7f752c195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:21.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.830+0000 7f75347ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f752c101480 0x7f752c1964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:21.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.830+0000 7f75347ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f752c101e10 0x7f752c19d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:21.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.830+0000 7f75347ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752c19db10 con 0x7f752c100290 2023-11-16T16:49:21.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.830+0000 7f75347ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752c19dc50 con 0x7f752c101480 2023-11-16T16:49:21.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.830+0000 7f75347ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752c19dde0 con 0x7f752c101e10 2023-11-16T16:49:21.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.830+0000 7f753259b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f752c100290 0x7f752c195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:21.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.830+0000 7f753259b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f752c100290 0x7f752c195f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45096/0 (socket says 172.21.15.2:45096) 2023-11-16T16:49:21.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.830+0000 7f753259b700 1 -- 172.21.15.2:0/658576569 learned_addr learned my addr 172.21.15.2:0/658576569 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:21.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.830+0000 7f7532d9c700 1 --2- 172.21.15.2:0/658576569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f752c101e10 0x7f752c19d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:21.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.830+0000 7f7531d9a700 1 --2- 172.21.15.2:0/658576569 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f752c101480 0x7f752c1964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:21.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.830+0000 7f753259b700 1 -- 172.21.15.2:0/658576569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f752c101e10 msgr2=0x7f752c19d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:21.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.830+0000 7f753259b700 1 --2- 172.21.15.2:0/658576569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f752c101e10 0x7f752c19d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:21.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.830+0000 7f753259b700 1 -- 172.21.15.2:0/658576569 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f752c101480 msgr2=0x7f752c1964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:21.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.831+0000 7f753259b700 1 --2- 172.21.15.2:0/658576569 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f752c101480 0x7f752c1964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:21.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.831+0000 7f753259b700 1 -- 172.21.15.2:0/658576569 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751c0095f0 con 0x7f752c100290 2023-11-16T16:49:21.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.831+0000 7f753259b700 1 --2- 172.21.15.2:0/658576569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f752c100290 0x7f752c195f90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f752800e970 tx=0x7f752800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:21.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.831+0000 7f75237fe700 1 -- 172.21.15.2:0/658576569 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7528009790 con 0x7f752c100290 2023-11-16T16:49:21.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.831+0000 7f75237fe700 1 -- 172.21.15.2:0/658576569 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f752800c7e0 con 0x7f752c100290 2023-11-16T16:49:21.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.831+0000 7f75347ff700 1 -- 172.21.15.2:0/658576569 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f752c10bc50 con 0x7f752c100290 2023-11-16T16:49:21.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.831+0000 7f75237fe700 1 -- 172.21.15.2:0/658576569 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7528003c70 con 0x7f752c100290 2023-11-16T16:49:21.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.831+0000 7f75347ff700 1 -- 172.21.15.2:0/658576569 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f752c10bac0 con 0x7f752c100290 2023-11-16T16:49:21.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.832+0000 7f75347ff700 1 -- 172.21.15.2:0/658576569 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f752c19e4e0 con 0x7f752c100290 2023-11-16T16:49:21.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.834+0000 7f75237fe700 1 -- 172.21.15.2:0/658576569 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f752800cc50 con 0x7f752c100290 2023-11-16T16:49:21.836 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.835+0000 7f75237fe700 1 --2- 172.21.15.2:0/658576569 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f75180a1150 0x7f75180a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:21.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.836+0000 7f75237fe700 1 -- 172.21.15.2:0/658576569 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f7528014070 con 0x7f752c100290 2023-11-16T16:49:21.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.836+0000 7f7531d9a700 1 --2- 172.21.15.2:0/658576569 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f75180a1150 0x7f75180a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:21.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.836+0000 7f7531d9a700 1 --2- 172.21.15.2:0/658576569 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f75180a1150 0x7f75180a3600 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f751c00b2e0 tx=0x7f751c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:21.840 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:21.839+0000 7f75237fe700 1 -- 172.21.15.2:0/658576569 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f75280961f0 con 0x7f752c100290 2023-11-16T16:49:21.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:21 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1924350947' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-11-16T16:49:21.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:21 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1924350947' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-11-16T16:49:22.055 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.055+0000 7f75347ff700 1 -- 172.21.15.2:0/658576569 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f752c1071d0 con 0x7f752c100290 2023-11-16T16:49:22.057 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.057+0000 7f75237fe700 1 -- 172.21.15.2:0/658576569 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24567 (secure 0 0 0) 0x7f7528003dd0 con 0x7f752c100290 2023-11-16T16:49:22.058 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:49:22.058 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":93,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:48:45.491739+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:24.458452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:25.786005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:29.313181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:49:22.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.059+0000 7f75347ff700 1 -- 172.21.15.2:0/658576569 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f75180a1150 msgr2=0x7f75180a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:22.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.059+0000 7f75347ff700 1 --2- 172.21.15.2:0/658576569 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f75180a1150 0x7f75180a3600 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f751c00b2e0 tx=0x7f751c005fb0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.059+0000 7f75347ff700 1 -- 172.21.15.2:0/658576569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f752c100290 msgr2=0x7f752c195f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:22.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.059+0000 7f75347ff700 1 --2- 172.21.15.2:0/658576569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f752c100290 0x7f752c195f90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f752800e970 tx=0x7f752800ec80 comp rx=0 tx=0).stop 2023-11-16T16:49:22.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.059+0000 7f75347ff700 1 -- 172.21.15.2:0/658576569 shutdown_connections 2023-11-16T16:49:22.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.059+0000 7f75347ff700 1 --2- 172.21.15.2:0/658576569 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f75180a1150 0x7f75180a3600 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.060 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.059+0000 7f75347ff700 1 --2- 172.21.15.2:0/658576569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f752c101e10 0x7f752c19d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.059+0000 7f75347ff700 1 --2- 172.21.15.2:0/658576569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f752c100290 0x7f752c195f90 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.059+0000 7f75347ff700 1 --2- 172.21.15.2:0/658576569 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f752c101480 0x7f752c1964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.059+0000 7f75347ff700 1 -- 172.21.15.2:0/658576569 >> 172.21.15.2:0/658576569 conn(0x7f752c0fb870 msgr2=0x7f752c0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:22.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.059+0000 7f75347ff700 1 -- 172.21.15.2:0/658576569 shutdown_connections 2023-11-16T16:49:22.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.059+0000 7f75347ff700 1 -- 172.21.15.2:0/658576569 wait complete. 2023-11-16T16:49:22.080 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:04.306584+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T16:47:24.458452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T16:47:25.786005+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T16:47:29.313181+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T16:49:22.080 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:49:22.288 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.287+0000 7f8124fff700 1 -- 172.21.15.2:0/1120484301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81201002c0 msgr2=0x7f81201006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:22.288 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.287+0000 7f8124fff700 1 --2- 172.21.15.2:0/1120484301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81201002c0 0x7f81201006d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8110009990 tx=0x7f8110009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.288 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.287+0000 7f8124fff700 1 -- 172.21.15.2:0/1120484301 shutdown_connections 2023-11-16T16:49:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.287+0000 7f8124fff700 1 --2- 172.21.15.2:0/1120484301 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8120101e40 0x7f8120108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.287+0000 7f8124fff700 1 --2- 172.21.15.2:0/1120484301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81201002c0 0x7f81201006d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.287+0000 7f8124fff700 1 --2- 172.21.15.2:0/1120484301 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f81201014b0 0x7f8120101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.287+0000 7f8124fff700 1 -- 172.21.15.2:0/1120484301 >> 172.21.15.2:0/1120484301 conn(0x7f81200fb8a0 msgr2=0x7f81200fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.287+0000 7f8124fff700 1 -- 172.21.15.2:0/1120484301 shutdown_connections 2023-11-16T16:49:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.287+0000 7f8124fff700 1 -- 172.21.15.2:0/1120484301 wait complete. 2023-11-16T16:49:22.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.288+0000 7f8124fff700 1 Processor -- start 2023-11-16T16:49:22.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.288+0000 7f8124fff700 1 -- start start 2023-11-16T16:49:22.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.288+0000 7f8124fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81201002c0 0x7f8120198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:22.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.288+0000 7f8124fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81201014b0 0x7f81201987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:22.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.289+0000 7f8124fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8120101e40 0x7f812019f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:22.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.289+0000 7f8124fff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812019fe00 con 0x7f81201014b0 2023-11-16T16:49:22.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.289+0000 7f8124fff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812019ff40 con 0x7f8120101e40 2023-11-16T16:49:22.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.289+0000 7f8124fff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81201a00d0 con 0x7f81201002c0 2023-11-16T16:49:22.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.289+0000 7f811ed9d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8120101e40 0x7f812019f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:22.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.289+0000 7f8117fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81201014b0 0x7f81201987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:22.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.289+0000 7f811ed9d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8120101e40 0x7f812019f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:45006/0 (socket says 172.21.15.2:45006) 2023-11-16T16:49:22.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.289+0000 7f811ed9d700 1 -- 172.21.15.2:0/3732218477 learned_addr learned my addr 172.21.15.2:0/3732218477 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:22.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.289+0000 7f811e59c700 1 --2- 172.21.15.2:0/3732218477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81201002c0 0x7f8120198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:22.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.289+0000 7f811ed9d700 1 -- 172.21.15.2:0/3732218477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81201002c0 msgr2=0x7f8120198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:22.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.289+0000 7f811ed9d700 1 --2- 172.21.15.2:0/3732218477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81201002c0 0x7f8120198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.289+0000 7f811ed9d700 1 -- 172.21.15.2:0/3732218477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81201014b0 msgr2=0x7f81201987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:22.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.289+0000 7f811ed9d700 1 --2- 172.21.15.2:0/3732218477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81201014b0 0x7f81201987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.289+0000 7f811ed9d700 1 -- 172.21.15.2:0/3732218477 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81100095f0 con 0x7f8120101e40 2023-11-16T16:49:22.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.289+0000 7f8117fff700 1 --2- 172.21.15.2:0/3732218477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81201014b0 0x7f81201987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:22.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.290+0000 7f811ed9d700 1 --2- 172.21.15.2:0/3732218477 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8120101e40 0x7f812019f830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8118007ee0 tx=0x7f811800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:22.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.290+0000 7f81177fe700 1 -- 172.21.15.2:0/3732218477 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8118017070 con 0x7f8120101e40 2023-11-16T16:49:22.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.290+0000 7f8124fff700 1 -- 172.21.15.2:0/3732218477 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f812010bc90 con 0x7f8120101e40 2023-11-16T16:49:22.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.290+0000 7f81177fe700 1 -- 172.21.15.2:0/3732218477 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f811800f040 con 0x7f8120101e40 2023-11-16T16:49:22.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.290+0000 7f81177fe700 1 -- 172.21.15.2:0/3732218477 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8118005550 con 0x7f8120101e40 2023-11-16T16:49:22.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.290+0000 7f8124fff700 1 -- 172.21.15.2:0/3732218477 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f812010bb00 con 0x7f8120101e40 2023-11-16T16:49:22.294 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.291+0000 7f8124fff700 1 -- 172.21.15.2:0/3732218477 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81201929f0 con 0x7f8120101e40 2023-11-16T16:49:22.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.293+0000 7f81177fe700 1 -- 172.21.15.2:0/3732218477 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f811800db20 con 0x7f8120101e40 2023-11-16T16:49:22.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.295+0000 7f81177fe700 1 --2- 172.21.15.2:0/3732218477 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f810c0a1150 0x7f810c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:22.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.295+0000 7f81177fe700 1 -- 172.21.15.2:0/3732218477 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f81180cca10 con 0x7f8120101e40 2023-11-16T16:49:22.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.295+0000 7f811e59c700 1 --2- 172.21.15.2:0/3732218477 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f810c0a1150 0x7f810c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:22.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.295+0000 7f811e59c700 1 --2- 172.21.15.2:0/3732218477 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f810c0a1150 0x7f810c0a3600 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f811000b2e0 tx=0x7f8110005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:22.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.297+0000 7f81177fe700 1 -- 172.21.15.2:0/3732218477 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8118095de0 con 0x7f8120101e40 2023-11-16T16:49:22.506 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.506+0000 7f8124fff700 1 -- 172.21.15.2:0/3732218477 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f812002cfe0 con 0x7f8120101e40 2023-11-16T16:49:22.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.511+0000 7f81177fe700 1 -- 172.21.15.2:0/3732218477 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8118099400 con 0x7f8120101e40 2023-11-16T16:49:22.512 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T16:49:22.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.513+0000 7f8124fff700 1 -- 172.21.15.2:0/3732218477 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f810c0a1150 msgr2=0x7f810c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:22.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.513+0000 7f8124fff700 1 --2- 172.21.15.2:0/3732218477 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f810c0a1150 0x7f810c0a3600 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f811000b2e0 tx=0x7f8110005fb0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.513+0000 7f8124fff700 1 -- 172.21.15.2:0/3732218477 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8120101e40 msgr2=0x7f812019f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:22.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.513+0000 7f8124fff700 1 --2- 172.21.15.2:0/3732218477 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8120101e40 0x7f812019f830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8118007ee0 tx=0x7f811800d3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.513+0000 7f8124fff700 1 -- 172.21.15.2:0/3732218477 shutdown_connections 2023-11-16T16:49:22.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.513+0000 7f8124fff700 1 --2- 172.21.15.2:0/3732218477 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f810c0a1150 0x7f810c0a3600 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.513+0000 7f8124fff700 1 --2- 172.21.15.2:0/3732218477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81201002c0 0x7f8120198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.513+0000 7f8124fff700 1 --2- 172.21.15.2:0/3732218477 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8120101e40 0x7f812019f830 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.513+0000 7f8124fff700 1 --2- 172.21.15.2:0/3732218477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81201014b0 0x7f81201987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.513+0000 7f8124fff700 1 -- 172.21.15.2:0/3732218477 >> 172.21.15.2:0/3732218477 conn(0x7f81200fb8a0 msgr2=0x7f8120106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.513+0000 7f8124fff700 1 -- 172.21.15.2:0/3732218477 shutdown_connections 2023-11-16T16:49:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.513+0000 7f8124fff700 1 -- 172.21.15.2:0/3732218477 wait complete. 2023-11-16T16:49:22.533 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T16:49:22.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.739+0000 7fc484bff700 1 -- 172.21.15.2:0/867142735 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc480101e40 msgr2=0x7fc480108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:22.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.739+0000 7fc484bff700 1 --2- 172.21.15.2:0/867142735 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc480101e40 0x7fc480108700 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc47800b0a0 tx=0x7fc47800b3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.739+0000 7fc484bff700 1 -- 172.21.15.2:0/867142735 shutdown_connections 2023-11-16T16:49:22.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.739+0000 7fc484bff700 1 --2- 172.21.15.2:0/867142735 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc480101e40 0x7fc480108700 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.739+0000 7fc484bff700 1 --2- 172.21.15.2:0/867142735 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc4801002c0 0x7fc4801006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.739+0000 7fc484bff700 1 --2- 172.21.15.2:0/867142735 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc4801014b0 0x7fc480101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.739+0000 7fc484bff700 1 -- 172.21.15.2:0/867142735 >> 172.21.15.2:0/867142735 conn(0x7fc4800fb8a0 msgr2=0x7fc4800fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:22.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.739+0000 7fc484bff700 1 -- 172.21.15.2:0/867142735 shutdown_connections 2023-11-16T16:49:22.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.739+0000 7fc484bff700 1 -- 172.21.15.2:0/867142735 wait complete. 2023-11-16T16:49:22.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.740+0000 7fc484bff700 1 Processor -- start 2023-11-16T16:49:22.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.740+0000 7fc484bff700 1 -- start start 2023-11-16T16:49:22.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.740+0000 7fc484bff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4801002c0 0x7fc4801982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:22.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.740+0000 7fc484bff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc4801014b0 0x7fc480198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:22.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.740+0000 7fc484bff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc480101e40 0x7fc48019f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:22.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.741+0000 7fc484bff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48019fe60 con 0x7fc4801002c0 2023-11-16T16:49:22.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.741+0000 7fc484bff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48019ffa0 con 0x7fc480101e40 2023-11-16T16:49:22.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.741+0000 7fc484bff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4801a0130 con 0x7fc4801014b0 2023-11-16T16:49:22.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.741+0000 7fc47e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4801002c0 0x7fc4801982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:22.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.741+0000 7fc47e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4801002c0 0x7fc4801982e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45128/0 (socket says 172.21.15.2:45128) 2023-11-16T16:49:22.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.741+0000 7fc47e59c700 1 -- 172.21.15.2:0/766863779 learned_addr learned my addr 172.21.15.2:0/766863779 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:22.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.741+0000 7fc47dd9b700 1 --2- 172.21.15.2:0/766863779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc4801014b0 0x7fc480198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:22.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.741+0000 7fc47ed9d700 1 --2- 172.21.15.2:0/766863779 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc480101e40 0x7fc48019f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:22.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.741+0000 7fc47e59c700 1 -- 172.21.15.2:0/766863779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc4801014b0 msgr2=0x7fc480198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:22.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.741+0000 7fc47e59c700 1 --2- 172.21.15.2:0/766863779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc4801014b0 0x7fc480198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.741+0000 7fc47e59c700 1 -- 172.21.15.2:0/766863779 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc480101e40 msgr2=0x7fc48019f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:22.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.741+0000 7fc47e59c700 1 --2- 172.21.15.2:0/766863779 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc480101e40 0x7fc48019f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.741+0000 7fc47e59c700 1 -- 172.21.15.2:0/766863779 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc478009d00 con 0x7fc4801002c0 2023-11-16T16:49:22.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.741+0000 7fc47e59c700 1 --2- 172.21.15.2:0/766863779 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4801002c0 0x7fc4801982e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc47000e970 tx=0x7fc47000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:22.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.742+0000 7fc46f7fe700 1 -- 172.21.15.2:0/766863779 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc470009790 con 0x7fc4801002c0 2023-11-16T16:49:22.746 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.742+0000 7fc484bff700 1 -- 172.21.15.2:0/766863779 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc48010bc50 con 0x7fc4801002c0 2023-11-16T16:49:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.742+0000 7fc46f7fe700 1 -- 172.21.15.2:0/766863779 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc47000eec0 con 0x7fc4801002c0 2023-11-16T16:49:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.742+0000 7fc46f7fe700 1 -- 172.21.15.2:0/766863779 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc470003cd0 con 0x7fc4801002c0 2023-11-16T16:49:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.742+0000 7fc484bff700 1 -- 172.21.15.2:0/766863779 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc48010bac0 con 0x7fc4801002c0 2023-11-16T16:49:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.743+0000 7fc484bff700 1 -- 172.21.15.2:0/766863779 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4801a0880 con 0x7fc4801002c0 2023-11-16T16:49:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.745+0000 7fc46f7fe700 1 -- 172.21.15.2:0/766863779 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fc47000ccd0 con 0x7fc4801002c0 2023-11-16T16:49:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.746+0000 7fc46f7fe700 1 --2- 172.21.15.2:0/766863779 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc4680a1150 0x7fc4680a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.746+0000 7fc46f7fe700 1 -- 172.21.15.2:0/766863779 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fc470014070 con 0x7fc4801002c0 2023-11-16T16:49:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.746+0000 7fc47dd9b700 1 --2- 172.21.15.2:0/766863779 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc4680a1150 0x7fc4680a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:22.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.747+0000 7fc47dd9b700 1 --2- 172.21.15.2:0/766863779 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc4680a1150 0x7fc4680a3600 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc474009c60 tx=0x7fc474009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:22.751 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.750+0000 7fc46f7fe700 1 -- 172.21.15.2:0/766863779 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc470096280 con 0x7fc4801002c0 2023-11-16T16:49:22.764 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:22 smithi141 ceph-mon[131184]: pgmap v342: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2023-11-16T16:49:22.764 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:22 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/658576569' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:22.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:22 smithi155 ceph-mon[131607]: pgmap v342: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2023-11-16T16:49:22.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:22 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/658576569' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:22.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:22 smithi002 ceph-mon[128712]: pgmap v342: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2023-11-16T16:49:22.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:22 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/658576569' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:22.965 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.964+0000 7fc484bff700 1 -- 172.21.15.2:0/766863779 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc4801a0e30 con 0x7fc4801002c0 2023-11-16T16:49:22.966 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.964+0000 7fc46f7fe700 1 -- 172.21.15.2:0/766863779 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc4700998a0 con 0x7fc4801002c0 2023-11-16T16:49:22.966 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T16:49:22.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.967+0000 7fc484bff700 1 -- 172.21.15.2:0/766863779 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc4680a1150 msgr2=0x7fc4680a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:22.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.967+0000 7fc484bff700 1 --2- 172.21.15.2:0/766863779 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc4680a1150 0x7fc4680a3600 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc474009c60 tx=0x7fc474009260 comp rx=0 tx=0).stop 2023-11-16T16:49:22.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.967+0000 7fc484bff700 1 -- 172.21.15.2:0/766863779 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4801002c0 msgr2=0x7fc4801982e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:22.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.967+0000 7fc484bff700 1 --2- 172.21.15.2:0/766863779 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4801002c0 0x7fc4801982e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc47000e970 tx=0x7fc47000ec80 comp rx=0 tx=0).stop 2023-11-16T16:49:22.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.967+0000 7fc484bff700 1 -- 172.21.15.2:0/766863779 shutdown_connections 2023-11-16T16:49:22.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.967+0000 7fc484bff700 1 --2- 172.21.15.2:0/766863779 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fc4680a1150 0x7fc4680a3600 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.967+0000 7fc484bff700 1 --2- 172.21.15.2:0/766863779 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc480101e40 0x7fc48019f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.967+0000 7fc484bff700 1 --2- 172.21.15.2:0/766863779 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4801002c0 0x7fc4801982e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.967+0000 7fc484bff700 1 --2- 172.21.15.2:0/766863779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc4801014b0 0x7fc480198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:22.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.967+0000 7fc484bff700 1 -- 172.21.15.2:0/766863779 >> 172.21.15.2:0/766863779 conn(0x7fc4800fb8a0 msgr2=0x7fc480106d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:22.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.967+0000 7fc484bff700 1 -- 172.21.15.2:0/766863779 shutdown_connections 2023-11-16T16:49:22.970 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:22.967+0000 7fc484bff700 1 -- 172.21.15.2:0/766863779 wait complete. 2023-11-16T16:49:22.988 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T16:49:23.194 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.192+0000 7fb0f35ff700 1 -- 172.21.15.2:0/638678855 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0ec1014b0 msgr2=0x7fb0ec101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:23.194 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.192+0000 7fb0f35ff700 1 --2- 172.21.15.2:0/638678855 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0ec1014b0 0x7fb0ec101900 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc009990 tx=0x7fb0dc009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:23.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.192+0000 7fb0f35ff700 1 -- 172.21.15.2:0/638678855 shutdown_connections 2023-11-16T16:49:23.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.192+0000 7fb0f35ff700 1 --2- 172.21.15.2:0/638678855 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0ec1002c0 0x7fb0ec1006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:23.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.192+0000 7fb0f35ff700 1 --2- 172.21.15.2:0/638678855 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0ec101e40 0x7fb0ec108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:23.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.192+0000 7fb0f35ff700 1 --2- 172.21.15.2:0/638678855 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0ec1014b0 0x7fb0ec101900 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:23.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.192+0000 7fb0f35ff700 1 -- 172.21.15.2:0/638678855 >> 172.21.15.2:0/638678855 conn(0x7fb0ec0fb8a0 msgr2=0x7fb0ec0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:23.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.192+0000 7fb0f35ff700 1 -- 172.21.15.2:0/638678855 shutdown_connections 2023-11-16T16:49:23.195 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.193+0000 7fb0f35ff700 1 -- 172.21.15.2:0/638678855 wait complete. 2023-11-16T16:49:23.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.193+0000 7fb0f35ff700 1 Processor -- start 2023-11-16T16:49:23.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.194+0000 7fb0f35ff700 1 -- start start 2023-11-16T16:49:23.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.194+0000 7fb0f35ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0ec1002c0 0x7fb0ec0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:23.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.194+0000 7fb0f35ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0ec1014b0 0x7fb0ec072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:23.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.194+0000 7fb0f35ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0ec101e40 0x7fb0ec073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:23.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.194+0000 7fb0f35ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec073bc0 con 0x7fb0ec1002c0 2023-11-16T16:49:23.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.194+0000 7fb0f35ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec073d30 con 0x7fb0ec1014b0 2023-11-16T16:49:23.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.194+0000 7fb0f35ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec1a0090 con 0x7fb0ec101e40 2023-11-16T16:49:23.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.194+0000 7fb0f139b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0ec1002c0 0x7fb0ec0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:23.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.194+0000 7fb0f139b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0ec1002c0 0x7fb0ec0748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45150/0 (socket says 172.21.15.2:45150) 2023-11-16T16:49:23.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.194+0000 7fb0f139b700 1 -- 172.21.15.2:0/977913997 learned_addr learned my addr 172.21.15.2:0/977913997 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:23.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.194+0000 7fb0f1b9c700 1 --2- 172.21.15.2:0/977913997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0ec101e40 0x7fb0ec073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:23.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.194+0000 7fb0f0b9a700 1 --2- 172.21.15.2:0/977913997 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0ec1014b0 0x7fb0ec072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:23.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.195+0000 7fb0f139b700 1 -- 172.21.15.2:0/977913997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0ec101e40 msgr2=0x7fb0ec073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:23.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.195+0000 7fb0f139b700 1 --2- 172.21.15.2:0/977913997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0ec101e40 0x7fb0ec073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:23.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.195+0000 7fb0f139b700 1 -- 172.21.15.2:0/977913997 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0ec1014b0 msgr2=0x7fb0ec072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:23.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.195+0000 7fb0f139b700 1 --2- 172.21.15.2:0/977913997 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0ec1014b0 0x7fb0ec072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:23.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.195+0000 7fb0f139b700 1 -- 172.21.15.2:0/977913997 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0dc0095f0 con 0x7fb0ec1002c0 2023-11-16T16:49:23.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.195+0000 7fb0f139b700 1 --2- 172.21.15.2:0/977913997 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0ec1002c0 0x7fb0ec0748a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb0e800d980 tx=0x7fb0e800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:23.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.195+0000 7fb0e27fc700 1 -- 172.21.15.2:0/977913997 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e800f890 con 0x7fb0ec1002c0 2023-11-16T16:49:23.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.195+0000 7fb0e27fc700 1 -- 172.21.15.2:0/977913997 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0e8010460 con 0x7fb0ec1002c0 2023-11-16T16:49:23.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.195+0000 7fb0e27fc700 1 -- 172.21.15.2:0/977913997 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e80195c0 con 0x7fb0ec1002c0 2023-11-16T16:49:23.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.195+0000 7fb0f35ff700 1 -- 172.21.15.2:0/977913997 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ec10bc30 con 0x7fb0ec1002c0 2023-11-16T16:49:23.199 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.195+0000 7fb0f35ff700 1 -- 172.21.15.2:0/977913997 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ec10baa0 con 0x7fb0ec1002c0 2023-11-16T16:49:23.199 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.196+0000 7fb0f35ff700 1 -- 172.21.15.2:0/977913997 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0ec107850 con 0x7fb0ec1002c0 2023-11-16T16:49:23.200 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.198+0000 7fb0e27fc700 1 -- 172.21.15.2:0/977913997 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fb0e80105d0 con 0x7fb0ec1002c0 2023-11-16T16:49:23.201 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.200+0000 7fb0e27fc700 1 --2- 172.21.15.2:0/977913997 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb0d80a1150 0x7fb0d80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:23.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.200+0000 7fb0e27fc700 1 -- 172.21.15.2:0/977913997 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fb0e80cca30 con 0x7fb0ec1002c0 2023-11-16T16:49:23.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.200+0000 7fb0f0b9a700 1 --2- 172.21.15.2:0/977913997 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb0d80a1150 0x7fb0d80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:23.202 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.201+0000 7fb0f0b9a700 1 --2- 172.21.15.2:0/977913997 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb0d80a1150 0x7fb0d80a3600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc00b5c0 tx=0x7fb0dc005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:23.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.204+0000 7fb0e27fc700 1 -- 172.21.15.2:0/977913997 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb0e8095dc0 con 0x7fb0ec1002c0 2023-11-16T16:49:23.417 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.415+0000 7fb0f35ff700 1 -- 172.21.15.2:0/977913997 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb0ec061700 con 0x7fb0ec1002c0 2023-11-16T16:49:32.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.415+0000 7fb0e27fc700 1 -- 172.21.15.2:0/977913997 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb0e80200a0 con 0x7fb0ec1002c0 2023-11-16T16:49:32.978 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2023-11-16T16:49:32.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.418+0000 7fb0f35ff700 1 -- 172.21.15.2:0/977913997 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb0d80a1150 msgr2=0x7fb0d80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:32.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.418+0000 7fb0f35ff700 1 --2- 172.21.15.2:0/977913997 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb0d80a1150 0x7fb0d80a3600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc00b5c0 tx=0x7fb0dc005960 comp rx=0 tx=0).stop 2023-11-16T16:49:32.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.418+0000 7fb0f35ff700 1 -- 172.21.15.2:0/977913997 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0ec1002c0 msgr2=0x7fb0ec0748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:32.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.418+0000 7fb0f35ff700 1 --2- 172.21.15.2:0/977913997 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0ec1002c0 0x7fb0ec0748a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb0e800d980 tx=0x7fb0e800dc90 comp rx=0 tx=0).stop 2023-11-16T16:49:32.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.418+0000 7fb0f35ff700 1 -- 172.21.15.2:0/977913997 shutdown_connections 2023-11-16T16:49:32.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.418+0000 7fb0f35ff700 1 --2- 172.21.15.2:0/977913997 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fb0d80a1150 0x7fb0d80a3600 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:32.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.418+0000 7fb0f35ff700 1 --2- 172.21.15.2:0/977913997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0ec101e40 0x7fb0ec073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:32.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.418+0000 7fb0f35ff700 1 --2- 172.21.15.2:0/977913997 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0ec1002c0 0x7fb0ec0748a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:32.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.418+0000 7fb0f35ff700 1 --2- 172.21.15.2:0/977913997 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb0ec1014b0 0x7fb0ec072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:32.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.418+0000 7fb0f35ff700 1 -- 172.21.15.2:0/977913997 >> 172.21.15.2:0/977913997 conn(0x7fb0ec0fb8a0 msgr2=0x7fb0ec104990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:32.983 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.418+0000 7fb0f35ff700 1 -- 172.21.15.2:0/977913997 shutdown_connections 2023-11-16T16:49:32.983 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:23.418+0000 7fb0f35ff700 1 -- 172.21.15.2:0/977913997 wait complete. 2023-11-16T16:49:32.984 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T16:49:32.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:23 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3732218477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:32.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:23 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/766863779' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:32.985 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:23 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/977913997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:32.986 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:24 smithi002 ceph-mon[128712]: pgmap v343: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2023-11-16T16:49:32.986 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:26 smithi002 ceph-mon[128712]: pgmap v344: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2023-11-16T16:49:32.986 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:28 smithi002 ceph-mon[128712]: pgmap v345: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T16:49:32.986 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:30 smithi002 ceph-mon[128712]: pgmap v346: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 0 B/s wr, 7 op/s 2023-11-16T16:49:32.988 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:23 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3732218477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:32.988 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:23 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/766863779' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:32.988 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:23 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/977913997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:32.989 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:24 smithi155 ceph-mon[131607]: pgmap v343: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2023-11-16T16:49:32.989 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:26 smithi155 ceph-mon[131607]: pgmap v344: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2023-11-16T16:49:32.989 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:28 smithi155 ceph-mon[131607]: pgmap v345: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T16:49:32.989 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:30 smithi155 ceph-mon[131607]: pgmap v346: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 0 B/s wr, 7 op/s 2023-11-16T16:49:32.989 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:23 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3732218477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:32.989 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:23 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/766863779' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:32.989 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:23 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/977913997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:32.989 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:24 smithi141 ceph-mon[131184]: pgmap v343: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2023-11-16T16:49:32.990 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:26 smithi141 ceph-mon[131184]: pgmap v344: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 8 op/s 2023-11-16T16:49:32.990 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:28 smithi141 ceph-mon[131184]: pgmap v345: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T16:49:32.990 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:30 smithi141 ceph-mon[131184]: pgmap v346: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 0 B/s wr, 7 op/s 2023-11-16T16:49:33.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:32 smithi002 ceph-mon[128712]: pgmap v347: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T16:49:33.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.203+0000 7fa649dff700 1 -- 172.21.15.2:0/602145861 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6441002c0 msgr2=0x7fa6441006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:33.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.203+0000 7fa649dff700 1 --2- 172.21.15.2:0/602145861 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6441002c0 0x7fa6441006d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa634009990 tx=0x7fa634009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.203+0000 7fa649dff700 1 -- 172.21.15.2:0/602145861 shutdown_connections 2023-11-16T16:49:33.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.203+0000 7fa649dff700 1 --2- 172.21.15.2:0/602145861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa644101e40 0x7fa644108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.203+0000 7fa649dff700 1 --2- 172.21.15.2:0/602145861 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6441002c0 0x7fa6441006d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.203+0000 7fa649dff700 1 --2- 172.21.15.2:0/602145861 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6441014b0 0x7fa644101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.203+0000 7fa649dff700 1 -- 172.21.15.2:0/602145861 >> 172.21.15.2:0/602145861 conn(0x7fa6440fb8a0 msgr2=0x7fa6440fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:33.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.203+0000 7fa649dff700 1 -- 172.21.15.2:0/602145861 shutdown_connections 2023-11-16T16:49:33.205 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.203+0000 7fa649dff700 1 -- 172.21.15.2:0/602145861 wait complete. 2023-11-16T16:49:33.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.204+0000 7fa649dff700 1 Processor -- start 2023-11-16T16:49:33.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.204+0000 7fa649dff700 1 -- start start 2023-11-16T16:49:33.207 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.204+0000 7fa649dff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6441002c0 0x7fa644196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:33.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.204+0000 7fa649dff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6441014b0 0x7fa6441965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:33.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.204+0000 7fa649dff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa644101e40 0x7fa64419d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:33.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.204+0000 7fa649dff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64419dc10 con 0x7fa644101e40 2023-11-16T16:49:33.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.204+0000 7fa649dff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64419dd50 con 0x7fa6441002c0 2023-11-16T16:49:33.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.204+0000 7fa649dff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64419dee0 con 0x7fa6441014b0 2023-11-16T16:49:33.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.205+0000 7fa643fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa644101e40 0x7fa64419d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:33.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.205+0000 7fa643fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa644101e40 0x7fa64419d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37912/0 (socket says 172.21.15.2:37912) 2023-11-16T16:49:33.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.205+0000 7fa6437fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6441002c0 0x7fa644196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:33.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.205+0000 7fa643fff700 1 -- 172.21.15.2:0/1195009529 learned_addr learned my addr 172.21.15.2:0/1195009529 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:33.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.205+0000 7fa642ffd700 1 --2- 172.21.15.2:0/1195009529 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6441014b0 0x7fa6441965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:33.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.205+0000 7fa643fff700 1 -- 172.21.15.2:0/1195009529 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6441014b0 msgr2=0x7fa6441965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:33.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.205+0000 7fa643fff700 1 --2- 172.21.15.2:0/1195009529 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6441014b0 0x7fa6441965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.208 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.205+0000 7fa643fff700 1 -- 172.21.15.2:0/1195009529 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6441002c0 msgr2=0x7fa644196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:33.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.205+0000 7fa643fff700 1 --2- 172.21.15.2:0/1195009529 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6441002c0 0x7fa644196090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.205+0000 7fa643fff700 1 -- 172.21.15.2:0/1195009529 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6340095f0 con 0x7fa644101e40 2023-11-16T16:49:33.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.205+0000 7fa642ffd700 1 --2- 172.21.15.2:0/1195009529 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6441014b0 0x7fa6441965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:49:33.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.205+0000 7fa643fff700 1 --2- 172.21.15.2:0/1195009529 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa644101e40 0x7fa64419d640 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa63c007ee0 tx=0x7fa63c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:33.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.206+0000 7fa640ff9700 1 -- 172.21.15.2:0/1195009529 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c017070 con 0x7fa644101e40 2023-11-16T16:49:33.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.206+0000 7fa649dff700 1 -- 172.21.15.2:0/1195009529 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64410bc90 con 0x7fa644101e40 2023-11-16T16:49:33.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.206+0000 7fa640ff9700 1 -- 172.21.15.2:0/1195009529 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa63c00f040 con 0x7fa644101e40 2023-11-16T16:49:33.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.206+0000 7fa640ff9700 1 -- 172.21.15.2:0/1195009529 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c005740 con 0x7fa644101e40 2023-11-16T16:49:33.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.206+0000 7fa649dff700 1 -- 172.21.15.2:0/1195009529 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa64410bb00 con 0x7fa644101e40 2023-11-16T16:49:33.209 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.207+0000 7fa649dff700 1 -- 172.21.15.2:0/1195009529 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa644107850 con 0x7fa644101e40 2023-11-16T16:49:33.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.209+0000 7fa640ff9700 1 -- 172.21.15.2:0/1195009529 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fa63c00dc60 con 0x7fa644101e40 2023-11-16T16:49:33.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.210+0000 7fa640ff9700 1 --2- 172.21.15.2:0/1195009529 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa62c0a1150 0x7fa62c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:33.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.210+0000 7fa640ff9700 1 -- 172.21.15.2:0/1195009529 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fa63c0cc420 con 0x7fa644101e40 2023-11-16T16:49:33.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.211+0000 7fa6437fe700 1 --2- 172.21.15.2:0/1195009529 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa62c0a1150 0x7fa62c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:33.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.211+0000 7fa6437fe700 1 --2- 172.21.15.2:0/1195009529 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa62c0a1150 0x7fa62c0a3600 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa63400b5c0 tx=0x7fa634005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:33.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.215+0000 7fa640ff9700 1 -- 172.21.15.2:0/1195009529 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa63c0957b0 con 0x7fa644101e40 2023-11-16T16:49:33.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:32 smithi141 ceph-mon[131184]: pgmap v347: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T16:49:33.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:32 smithi155 ceph-mon[131607]: pgmap v347: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T16:49:33.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.423+0000 7fa649dff700 1 -- 172.21.15.2:0/1195009529 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa644061cc0 con 0x7fa644101e40 2023-11-16T16:49:33.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.424+0000 7fa640ff9700 1 -- 172.21.15.2:0/1195009529 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa63c098dd0 con 0x7fa644101e40 2023-11-16T16:49:33.425 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T16:49:33.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.426+0000 7fa649dff700 1 -- 172.21.15.2:0/1195009529 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa62c0a1150 msgr2=0x7fa62c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:33.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.426+0000 7fa649dff700 1 --2- 172.21.15.2:0/1195009529 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa62c0a1150 0x7fa62c0a3600 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa63400b5c0 tx=0x7fa634005fb0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.426+0000 7fa649dff700 1 -- 172.21.15.2:0/1195009529 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa644101e40 msgr2=0x7fa64419d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:33.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.426+0000 7fa649dff700 1 --2- 172.21.15.2:0/1195009529 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa644101e40 0x7fa64419d640 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa63c007ee0 tx=0x7fa63c00d3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.426+0000 7fa649dff700 1 -- 172.21.15.2:0/1195009529 shutdown_connections 2023-11-16T16:49:33.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.426+0000 7fa649dff700 1 --2- 172.21.15.2:0/1195009529 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fa62c0a1150 0x7fa62c0a3600 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.426+0000 7fa649dff700 1 --2- 172.21.15.2:0/1195009529 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa644101e40 0x7fa64419d640 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.428 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.426+0000 7fa649dff700 1 --2- 172.21.15.2:0/1195009529 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa6441002c0 0x7fa644196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.426+0000 7fa649dff700 1 --2- 172.21.15.2:0/1195009529 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6441014b0 0x7fa6441965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.426+0000 7fa649dff700 1 -- 172.21.15.2:0/1195009529 >> 172.21.15.2:0/1195009529 conn(0x7fa6440fb8a0 msgr2=0x7fa644104990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:33.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.426+0000 7fa649dff700 1 -- 172.21.15.2:0/1195009529 shutdown_connections 2023-11-16T16:49:33.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.426+0000 7fa649dff700 1 -- 172.21.15.2:0/1195009529 wait complete. 2023-11-16T16:49:33.447 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:49:33.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.651+0000 7f598d9ff700 1 -- 172.21.15.2:0/951831973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5988073550 msgr2=0x7f59880739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:33.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.651+0000 7f598d9ff700 1 --2- 172.21.15.2:0/951831973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5988073550 0x7f59880739c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f597c009990 tx=0x7f597c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.651+0000 7f598d9ff700 1 -- 172.21.15.2:0/951831973 shutdown_connections 2023-11-16T16:49:33.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.651+0000 7f598d9ff700 1 --2- 172.21.15.2:0/951831973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5988074b30 0x7f5988072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.651+0000 7f598d9ff700 1 --2- 172.21.15.2:0/951831973 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f598810c860 0x7f598810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.651+0000 7f598d9ff700 1 --2- 172.21.15.2:0/951831973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5988073550 0x7f59880739c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.651+0000 7f598d9ff700 1 -- 172.21.15.2:0/951831973 >> 172.21.15.2:0/951831973 conn(0x7f59880fb870 msgr2=0x7f59880fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:33.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.651+0000 7f598d9ff700 1 -- 172.21.15.2:0/951831973 shutdown_connections 2023-11-16T16:49:33.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.651+0000 7f598d9ff700 1 -- 172.21.15.2:0/951831973 wait complete. 2023-11-16T16:49:33.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.652+0000 7f598d9ff700 1 Processor -- start 2023-11-16T16:49:33.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.652+0000 7f598d9ff700 1 -- start start 2023-11-16T16:49:33.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.652+0000 7f598d9ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5988073550 0x7f598819c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:33.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.652+0000 7f598d9ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5988074b30 0x7f598819cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:33.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.652+0000 7f598d9ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f598810c860 0x7f59881a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:33.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.652+0000 7f598d9ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59881a4170 con 0x7f5988073550 2023-11-16T16:49:33.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.652+0000 7f598d9ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59881a42b0 con 0x7f598810c860 2023-11-16T16:49:33.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.652+0000 7f598d9ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59881a4440 con 0x7f5988074b30 2023-11-16T16:49:33.656 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.653+0000 7f5986ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5988073550 0x7f598819c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:33.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.653+0000 7f5986ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5988073550 0x7f598819c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37944/0 (socket says 172.21.15.2:37944) 2023-11-16T16:49:33.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.653+0000 7f5986ffd700 1 -- 172.21.15.2:0/1263137919 learned_addr learned my addr 172.21.15.2:0/1263137919 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:33.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.653+0000 7f59877fe700 1 --2- 172.21.15.2:0/1263137919 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f598810c860 0x7f59881a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:33.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.653+0000 7f59867fc700 1 --2- 172.21.15.2:0/1263137919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5988074b30 0x7f598819cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:33.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.653+0000 7f5986ffd700 1 -- 172.21.15.2:0/1263137919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5988074b30 msgr2=0x7f598819cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:33.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.653+0000 7f5986ffd700 1 --2- 172.21.15.2:0/1263137919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5988074b30 0x7f598819cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.653+0000 7f5986ffd700 1 -- 172.21.15.2:0/1263137919 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f598810c860 msgr2=0x7f59881a3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:33.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.653+0000 7f5986ffd700 1 --2- 172.21.15.2:0/1263137919 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f598810c860 0x7f59881a3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.653+0000 7f5986ffd700 1 -- 172.21.15.2:0/1263137919 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597c0095f0 con 0x7f5988073550 2023-11-16T16:49:33.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.653+0000 7f59877fe700 1 --2- 172.21.15.2:0/1263137919 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f598810c860 0x7f59881a3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:33.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.653+0000 7f5986ffd700 1 --2- 172.21.15.2:0/1263137919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5988073550 0x7f598819c5f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f597800efc0 tx=0x7f597800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:33.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.654+0000 7f598c9fd700 1 -- 172.21.15.2:0/1263137919 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597800c9c0 con 0x7f5988073550 2023-11-16T16:49:33.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.654+0000 7f598d9ff700 1 -- 172.21.15.2:0/1263137919 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5988112270 con 0x7f5988073550 2023-11-16T16:49:33.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.654+0000 7f598c9fd700 1 -- 172.21.15.2:0/1263137919 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f597800f040 con 0x7f5988073550 2023-11-16T16:49:33.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.654+0000 7f598c9fd700 1 -- 172.21.15.2:0/1263137919 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5978014480 con 0x7f5988073550 2023-11-16T16:49:33.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.654+0000 7f598d9ff700 1 -- 172.21.15.2:0/1263137919 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59881120e0 con 0x7f5988073550 2023-11-16T16:49:33.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.655+0000 7f598d9ff700 1 -- 172.21.15.2:0/1263137919 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59881023b0 con 0x7f5988073550 2023-11-16T16:49:33.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.657+0000 7f598c9fd700 1 -- 172.21.15.2:0/1263137919 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f597800cb20 con 0x7f5988073550 2023-11-16T16:49:33.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.658+0000 7f598c9fd700 1 --2- 172.21.15.2:0/1263137919 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f59700a11d0 0x7f59700a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:33.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.658+0000 7f598c9fd700 1 -- 172.21.15.2:0/1263137919 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f59780ccee0 con 0x7f5988073550 2023-11-16T16:49:33.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.658+0000 7f59867fc700 1 --2- 172.21.15.2:0/1263137919 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f59700a11d0 0x7f59700a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:33.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.659+0000 7f59867fc700 1 --2- 172.21.15.2:0/1263137919 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f59700a11d0 0x7f59700a3680 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f597c00b2e0 tx=0x7f597c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:33.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.662+0000 7f598c9fd700 1 -- 172.21.15.2:0/1263137919 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5978096270 con 0x7f5988073550 2023-11-16T16:49:33.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.920+0000 7f598d9ff700 1 -- 172.21.15.2:0/1263137919 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f598802cfe0 con 0x7f5988073550 2023-11-16T16:49:33.923 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.921+0000 7f598c9fd700 1 -- 172.21.15.2:0/1263137919 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10811 (secure 0 0 0) 0x7f5978099890 con 0x7f5988073550 2023-11-16T16:49:33.923 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:49:33.923 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/1610211225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":1610211225},{"type":"v1","addr":"172.21.15.2:6837","nonce":1610211225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14538,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2583261818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2583261818},{"type":"v1","addr":"172.21.15.2:6839","nonce":2583261818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14550,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3170991923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3170991923},{"type":"v1","addr":"172.21.15.2:6841","nonce":3170991923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:26.349176+0000","modified":"2023-11-16T16:48:05.774072+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24544,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.141:6835/3516869188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3516869188},{"type":"v1","addr":"172.21.15.141:6835","nonce":3516869188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/2527313191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":2527313191},{"type":"v1","addr":"172.21.15.155:6835","nonce":2527313191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24523":{"gid":24523,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1607756751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1607756751},{"type":"v1","addr":"172.21.15.155:6837","nonce":1607756751}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.141:6837/2934891475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6836","nonce":2934891475},{"type":"v1","addr":"172.21.15.141:6837","nonce":2934891475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24544":{"gid":24544,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1158457009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1158457009},{"type":"v1","addr":"172.21.15.155:6839","nonce":1158457009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6839/3916358272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6838","nonce":3916358272},{"type":"v1","addr":"172.21.15.141:6839","nonce":3916358272}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:49:33.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.924+0000 7f598d9ff700 1 -- 172.21.15.2:0/1263137919 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f59700a11d0 msgr2=0x7f59700a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:33.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.924+0000 7f598d9ff700 1 --2- 172.21.15.2:0/1263137919 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f59700a11d0 0x7f59700a3680 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f597c00b2e0 tx=0x7f597c005fb0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.924+0000 7f598d9ff700 1 -- 172.21.15.2:0/1263137919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5988073550 msgr2=0x7f598819c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:33.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.924+0000 7f598d9ff700 1 --2- 172.21.15.2:0/1263137919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5988073550 0x7f598819c5f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f597800efc0 tx=0x7f597800c3a0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.924+0000 7f598d9ff700 1 -- 172.21.15.2:0/1263137919 shutdown_connections 2023-11-16T16:49:33.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.924+0000 7f598d9ff700 1 --2- 172.21.15.2:0/1263137919 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f59700a11d0 0x7f59700a3680 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.924+0000 7f598d9ff700 1 --2- 172.21.15.2:0/1263137919 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f598810c860 0x7f59881a3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.924+0000 7f598d9ff700 1 --2- 172.21.15.2:0/1263137919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5988073550 0x7f598819c5f0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.924+0000 7f598d9ff700 1 --2- 172.21.15.2:0/1263137919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5988074b30 0x7f598819cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:33.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.924+0000 7f598d9ff700 1 -- 172.21.15.2:0/1263137919 >> 172.21.15.2:0/1263137919 conn(0x7f59880fb870 msgr2=0x7f59880fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:33.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.924+0000 7f598d9ff700 1 -- 172.21.15.2:0/1263137919 shutdown_connections 2023-11-16T16:49:33.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:33.924+0000 7f598d9ff700 1 -- 172.21.15.2:0/1263137919 wait complete. 2023-11-16T16:49:33.927 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 26 2023-11-16T16:49:33.945 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:49:34.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.149+0000 7f2d147ff700 1 -- 172.21.15.2:0/2779424992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d0c10c860 msgr2=0x7f2d0c10ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:34.151 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.149+0000 7f2d147ff700 1 --2- 172.21.15.2:0/2779424992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d0c10c860 0x7f2d0c10ec40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400b0a0 tx=0x7f2d0400b3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.151 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.149+0000 7f2d147ff700 1 -- 172.21.15.2:0/2779424992 shutdown_connections 2023-11-16T16:49:34.151 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.149+0000 7f2d147ff700 1 --2- 172.21.15.2:0/2779424992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d0c10c860 0x7f2d0c10ec40 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.151 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.149+0000 7f2d147ff700 1 --2- 172.21.15.2:0/2779424992 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d0c074b30 0x7f2d0c072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.151 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.149+0000 7f2d147ff700 1 --2- 172.21.15.2:0/2779424992 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d0c073550 0x7f2d0c0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.151 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.149+0000 7f2d147ff700 1 -- 172.21.15.2:0/2779424992 >> 172.21.15.2:0/2779424992 conn(0x7f2d0c0fb870 msgr2=0x7f2d0c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:34.151 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.149+0000 7f2d147ff700 1 -- 172.21.15.2:0/2779424992 shutdown_connections 2023-11-16T16:49:34.151 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.149+0000 7f2d147ff700 1 -- 172.21.15.2:0/2779424992 wait complete. 2023-11-16T16:49:34.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.150+0000 7f2d147ff700 1 Processor -- start 2023-11-16T16:49:34.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.150+0000 7f2d147ff700 1 -- start start 2023-11-16T16:49:34.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.150+0000 7f2d147ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d0c073550 0x7f2d0c19c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:34.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.150+0000 7f2d147ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d0c074b30 0x7f2d0c19cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:34.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.150+0000 7f2d147ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d0c10c860 0x7f2d0c1a3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:34.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.150+0000 7f2d147ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0c1a4230 con 0x7f2d0c10c860 2023-11-16T16:49:34.153 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.150+0000 7f2d147ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0c1a4370 con 0x7f2d0c073550 2023-11-16T16:49:34.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.150+0000 7f2d147ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0c1a4500 con 0x7f2d0c074b30 2023-11-16T16:49:34.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.151+0000 7f2d12d9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d0c10c860 0x7f2d0c1a3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:34.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.151+0000 7f2d1259b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d0c073550 0x7f2d0c19c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:34.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.151+0000 7f2d12d9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d0c10c860 0x7f2d0c1a3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37960/0 (socket says 172.21.15.2:37960) 2023-11-16T16:49:34.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.151+0000 7f2d12d9c700 1 -- 172.21.15.2:0/1089638563 learned_addr learned my addr 172.21.15.2:0/1089638563 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:34.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.151+0000 7f2d11d9a700 1 --2- 172.21.15.2:0/1089638563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d0c074b30 0x7f2d0c19cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:34.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.151+0000 7f2d12d9c700 1 -- 172.21.15.2:0/1089638563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d0c074b30 msgr2=0x7f2d0c19cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:34.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.151+0000 7f2d12d9c700 1 --2- 172.21.15.2:0/1089638563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d0c074b30 0x7f2d0c19cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.151+0000 7f2d12d9c700 1 -- 172.21.15.2:0/1089638563 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d0c073550 msgr2=0x7f2d0c19c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:34.154 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.151+0000 7f2d12d9c700 1 --2- 172.21.15.2:0/1089638563 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d0c073550 0x7f2d0c19c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.155 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.151+0000 7f2d12d9c700 1 -- 172.21.15.2:0/1089638563 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d04009d00 con 0x7f2d0c10c860 2023-11-16T16:49:34.155 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.151+0000 7f2d12d9c700 1 --2- 172.21.15.2:0/1089638563 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d0c10c860 0x7f2d0c1a3c60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2d04009fd0 tx=0x7f2d0400bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:34.155 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.152+0000 7f2d037fe700 1 -- 172.21.15.2:0/1089638563 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0400e040 con 0x7f2d0c10c860 2023-11-16T16:49:34.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.152+0000 7f2d037fe700 1 -- 172.21.15.2:0/1089638563 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d04003e10 con 0x7f2d0c10c860 2023-11-16T16:49:34.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.152+0000 7f2d147ff700 1 -- 172.21.15.2:0/1089638563 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d0c112270 con 0x7f2d0c10c860 2023-11-16T16:49:34.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.152+0000 7f2d037fe700 1 -- 172.21.15.2:0/1089638563 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0401b750 con 0x7f2d0c10c860 2023-11-16T16:49:34.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.152+0000 7f2d147ff700 1 -- 172.21.15.2:0/1089638563 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d0c1120e0 con 0x7f2d0c10c860 2023-11-16T16:49:34.156 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.153+0000 7f2d147ff700 1 -- 172.21.15.2:0/1089638563 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d0c1023b0 con 0x7f2d0c10c860 2023-11-16T16:49:34.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:33 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1195009529' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:34.157 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.155+0000 7f2d037fe700 1 -- 172.21.15.2:0/1089638563 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f2d0401da30 con 0x7f2d0c10c860 2023-11-16T16:49:34.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.156+0000 7f2d037fe700 1 --2- 172.21.15.2:0/1089638563 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2cf80a11d0 0x7f2cf80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:34.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.156+0000 7f2d037fe700 1 -- 172.21.15.2:0/1089638563 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f2d040ce140 con 0x7f2d0c10c860 2023-11-16T16:49:34.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.156+0000 7f2d1259b700 1 --2- 172.21.15.2:0/1089638563 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2cf80a11d0 0x7f2cf80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:34.158 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.157+0000 7f2d1259b700 1 --2- 172.21.15.2:0/1089638563 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2cf80a11d0 0x7f2cf80a3680 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c074f00 tx=0x7f2d08009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:34.162 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.160+0000 7f2d037fe700 1 -- 172.21.15.2:0/1089638563 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2d04097560 con 0x7f2d0c10c860 2023-11-16T16:49:34.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:33 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1195009529' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:34.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:33 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1195009529' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:34.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.370+0000 7f2d147ff700 1 -- 172.21.15.2:0/1089638563 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d0c02cfe0 con 0x7f2d0c10c860 2023-11-16T16:49:34.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.372+0000 7f2d037fe700 1 -- 172.21.15.2:0/1089638563 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24567 (secure 0 0 0) 0x7f2d0409ab80 con 0x7f2d0c10c860 2023-11-16T16:49:34.375 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:49:34.375 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":93,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:48:45.491739+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:24.458452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:25.786005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:29.313181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:49:34.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.375+0000 7f2d147ff700 1 -- 172.21.15.2:0/1089638563 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2cf80a11d0 msgr2=0x7f2cf80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:34.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.375+0000 7f2d147ff700 1 --2- 172.21.15.2:0/1089638563 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2cf80a11d0 0x7f2cf80a3680 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c074f00 tx=0x7f2d08009260 comp rx=0 tx=0).stop 2023-11-16T16:49:34.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.375+0000 7f2d147ff700 1 -- 172.21.15.2:0/1089638563 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d0c10c860 msgr2=0x7f2d0c1a3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:34.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.375+0000 7f2d147ff700 1 --2- 172.21.15.2:0/1089638563 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d0c10c860 0x7f2d0c1a3c60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2d04009fd0 tx=0x7f2d0400bce0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.375+0000 7f2d147ff700 1 -- 172.21.15.2:0/1089638563 shutdown_connections 2023-11-16T16:49:34.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.375+0000 7f2d147ff700 1 --2- 172.21.15.2:0/1089638563 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2cf80a11d0 0x7f2cf80a3680 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.375+0000 7f2d147ff700 1 --2- 172.21.15.2:0/1089638563 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d0c10c860 0x7f2d0c1a3c60 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.375+0000 7f2d147ff700 1 --2- 172.21.15.2:0/1089638563 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2d0c073550 0x7f2d0c19c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.375+0000 7f2d147ff700 1 --2- 172.21.15.2:0/1089638563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d0c074b30 0x7f2d0c19cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.375+0000 7f2d147ff700 1 -- 172.21.15.2:0/1089638563 >> 172.21.15.2:0/1089638563 conn(0x7f2d0c0fb870 msgr2=0x7f2d0c0fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:34.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.375+0000 7f2d147ff700 1 -- 172.21.15.2:0/1089638563 shutdown_connections 2023-11-16T16:49:34.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.375+0000 7f2d147ff700 1 -- 172.21.15.2:0/1089638563 wait complete. 2023-11-16T16:49:34.398 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:04.306584+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T16:47:24.458452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T16:47:25.786005+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T16:47:29.313181+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T16:49:34.398 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:49:34.606 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.604+0000 7f170a1ff700 1 -- 172.21.15.2:0/156869703 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17041002c0 msgr2=0x7f17041006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:34.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.604+0000 7f170a1ff700 1 --2- 172.21.15.2:0/156869703 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17041002c0 0x7f17041006d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f16f4009990 tx=0x7f16f4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.604+0000 7f170a1ff700 1 -- 172.21.15.2:0/156869703 shutdown_connections 2023-11-16T16:49:34.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.604+0000 7f170a1ff700 1 --2- 172.21.15.2:0/156869703 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1704101e40 0x7f1704108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.604+0000 7f170a1ff700 1 --2- 172.21.15.2:0/156869703 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17041002c0 0x7f17041006d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.604+0000 7f170a1ff700 1 --2- 172.21.15.2:0/156869703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17041014b0 0x7f1704101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.604+0000 7f170a1ff700 1 -- 172.21.15.2:0/156869703 >> 172.21.15.2:0/156869703 conn(0x7f17040fb8a0 msgr2=0x7f17040fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:34.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.604+0000 7f170a1ff700 1 -- 172.21.15.2:0/156869703 shutdown_connections 2023-11-16T16:49:34.607 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.604+0000 7f170a1ff700 1 -- 172.21.15.2:0/156869703 wait complete. 2023-11-16T16:49:34.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.606+0000 7f170a1ff700 1 Processor -- start 2023-11-16T16:49:34.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.606+0000 7f170a1ff700 1 -- start start 2023-11-16T16:49:34.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.606+0000 7f170a1ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17041002c0 0x7f1704198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:34.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.606+0000 7f170a1ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17041014b0 0x7f17041987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:34.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.606+0000 7f170a1ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1704101e40 0x7f170419f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:34.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.606+0000 7f170a1ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170419fdf0 con 0x7f1704101e40 2023-11-16T16:49:34.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.607+0000 7f170a1ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170419ff30 con 0x7f17041002c0 2023-11-16T16:49:34.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.607+0000 7f170a1ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17041a00c0 con 0x7f17041014b0 2023-11-16T16:49:34.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.607+0000 7f17037fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17041002c0 0x7f1704198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:34.609 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.607+0000 7f17037fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17041002c0 0x7f1704198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:41114/0 (socket says 172.21.15.2:41114) 2023-11-16T16:49:34.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.607+0000 7f17037fe700 1 -- 172.21.15.2:0/1551232342 learned_addr learned my addr 172.21.15.2:0/1551232342 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:34.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.607+0000 7f1703fff700 1 --2- 172.21.15.2:0/1551232342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1704101e40 0x7f170419f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:34.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.607+0000 7f1702ffd700 1 --2- 172.21.15.2:0/1551232342 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17041014b0 0x7f17041987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:34.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.607+0000 7f17037fe700 1 -- 172.21.15.2:0/1551232342 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17041014b0 msgr2=0x7f17041987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:34.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.607+0000 7f17037fe700 1 --2- 172.21.15.2:0/1551232342 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17041014b0 0x7f17041987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.607+0000 7f17037fe700 1 -- 172.21.15.2:0/1551232342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1704101e40 msgr2=0x7f170419f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:34.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.607+0000 7f17037fe700 1 --2- 172.21.15.2:0/1551232342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1704101e40 0x7f170419f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.607+0000 7f17037fe700 1 -- 172.21.15.2:0/1551232342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f40095f0 con 0x7f17041002c0 2023-11-16T16:49:34.610 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.607+0000 7f17037fe700 1 --2- 172.21.15.2:0/1551232342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17041002c0 0x7f1704198270 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f16f400b620 tx=0x7f16f4005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:34.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.608+0000 7f1700ff9700 1 -- 172.21.15.2:0/1551232342 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f401d070 con 0x7f17041002c0 2023-11-16T16:49:34.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.608+0000 7f170a1ff700 1 -- 172.21.15.2:0/1551232342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f170410bc90 con 0x7f17041002c0 2023-11-16T16:49:34.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.608+0000 7f1700ff9700 1 -- 172.21.15.2:0/1551232342 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16f4021410 con 0x7f17041002c0 2023-11-16T16:49:34.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.608+0000 7f1700ff9700 1 -- 172.21.15.2:0/1551232342 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f4017650 con 0x7f17041002c0 2023-11-16T16:49:34.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.608+0000 7f170a1ff700 1 -- 172.21.15.2:0/1551232342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f170410bb00 con 0x7f17041002c0 2023-11-16T16:49:34.611 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.609+0000 7f170a1ff700 1 -- 172.21.15.2:0/1551232342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17041929f0 con 0x7f17041002c0 2023-11-16T16:49:34.612 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.611+0000 7f1700ff9700 1 -- 172.21.15.2:0/1551232342 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f16f4021580 con 0x7f17041002c0 2023-11-16T16:49:34.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.612+0000 7f1700ff9700 1 --2- 172.21.15.2:0/1551232342 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f16ec0a1150 0x7f16ec0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:34.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.612+0000 7f1700ff9700 1 -- 172.21.15.2:0/1551232342 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f16f40ce1d0 con 0x7f17041002c0 2023-11-16T16:49:34.614 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.612+0000 7f1702ffd700 1 --2- 172.21.15.2:0/1551232342 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f16ec0a1150 0x7f16ec0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:34.615 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.613+0000 7f1702ffd700 1 --2- 172.21.15.2:0/1551232342 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f16ec0a1150 0x7f16ec0a3600 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009bf0 tx=0x7f16f8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:34.617 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.616+0000 7f1700ff9700 1 -- 172.21.15.2:0/1551232342 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f16f40975f0 con 0x7f17041002c0 2023-11-16T16:49:34.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.828+0000 7f170a1ff700 1 -- 172.21.15.2:0/1551232342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f17040617a0 con 0x7f17041002c0 2023-11-16T16:49:34.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.829+0000 7f1700ff9700 1 -- 172.21.15.2:0/1551232342 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f16f409ac10 con 0x7f17041002c0 2023-11-16T16:49:34.831 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T16:49:34.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.831+0000 7f170a1ff700 1 -- 172.21.15.2:0/1551232342 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f16ec0a1150 msgr2=0x7f16ec0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:34.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.831+0000 7f170a1ff700 1 --2- 172.21.15.2:0/1551232342 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f16ec0a1150 0x7f16ec0a3600 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009bf0 tx=0x7f16f8009260 comp rx=0 tx=0).stop 2023-11-16T16:49:34.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.831+0000 7f170a1ff700 1 -- 172.21.15.2:0/1551232342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17041002c0 msgr2=0x7f1704198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:34.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.831+0000 7f170a1ff700 1 --2- 172.21.15.2:0/1551232342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17041002c0 0x7f1704198270 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f16f400b620 tx=0x7f16f4005740 comp rx=0 tx=0).stop 2023-11-16T16:49:34.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.831+0000 7f170a1ff700 1 -- 172.21.15.2:0/1551232342 shutdown_connections 2023-11-16T16:49:34.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.831+0000 7f170a1ff700 1 --2- 172.21.15.2:0/1551232342 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f16ec0a1150 0x7f16ec0a3600 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.831+0000 7f170a1ff700 1 --2- 172.21.15.2:0/1551232342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1704101e40 0x7f170419f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.831+0000 7f170a1ff700 1 --2- 172.21.15.2:0/1551232342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f17041002c0 0x7f1704198270 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.831+0000 7f170a1ff700 1 --2- 172.21.15.2:0/1551232342 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f17041014b0 0x7f17041987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:34.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.831+0000 7f170a1ff700 1 -- 172.21.15.2:0/1551232342 >> 172.21.15.2:0/1551232342 conn(0x7f17040fb8a0 msgr2=0x7f1704106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:34.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.832+0000 7f170a1ff700 1 -- 172.21.15.2:0/1551232342 shutdown_connections 2023-11-16T16:49:34.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:34.832+0000 7f170a1ff700 1 -- 172.21.15.2:0/1551232342 wait complete. 2023-11-16T16:49:34.852 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T16:49:35.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.059+0000 7f11b55ff700 1 -- 172.21.15.2:0/3342708947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11b010c890 msgr2=0x7f11b010ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:35.061 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.059+0000 7f11b55ff700 1 --2- 172.21.15.2:0/3342708947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11b010c890 0x7f11b010ec70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f11a8009f90 tx=0x7f11a800b330 comp rx=0 tx=0).stop 2023-11-16T16:49:35.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.059+0000 7f11b55ff700 1 -- 172.21.15.2:0/3342708947 shutdown_connections 2023-11-16T16:49:35.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.059+0000 7f11b55ff700 1 --2- 172.21.15.2:0/3342708947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11b010c890 0x7f11b010ec70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.059+0000 7f11b55ff700 1 --2- 172.21.15.2:0/3342708947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11b0074b30 0x7f11b0072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.059+0000 7f11b55ff700 1 --2- 172.21.15.2:0/3342708947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11b0073550 0x7f11b00739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.059+0000 7f11b55ff700 1 -- 172.21.15.2:0/3342708947 >> 172.21.15.2:0/3342708947 conn(0x7f11b00fb8a0 msgr2=0x7f11b00fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:35.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.059+0000 7f11b55ff700 1 -- 172.21.15.2:0/3342708947 shutdown_connections 2023-11-16T16:49:35.062 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.060+0000 7f11b55ff700 1 -- 172.21.15.2:0/3342708947 wait complete. 2023-11-16T16:49:35.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.061+0000 7f11b55ff700 1 Processor -- start 2023-11-16T16:49:35.064 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.061+0000 7f11b55ff700 1 -- start start 2023-11-16T16:49:35.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.061+0000 7f11b55ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11b0073550 0x7f11b019c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:35.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.061+0000 7f11b55ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11b0074b30 0x7f11b019ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:35.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.061+0000 7f11b55ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11b010c890 0x7f11b01a3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:35.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.061+0000 7f11b55ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b01a4310 con 0x7f11b010c890 2023-11-16T16:49:35.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.061+0000 7f11b55ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b01a4450 con 0x7f11b0073550 2023-11-16T16:49:35.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.061+0000 7f11b55ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b01a45e0 con 0x7f11b0074b30 2023-11-16T16:49:35.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.061+0000 7f11af7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11b010c890 0x7f11b01a3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:35.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.061+0000 7f11af7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11b010c890 0x7f11b01a3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43156/0 (socket says 172.21.15.2:43156) 2023-11-16T16:49:35.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.061+0000 7f11af7fe700 1 -- 172.21.15.2:0/3892249895 learned_addr learned my addr 172.21.15.2:0/3892249895 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:35.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.062+0000 7f11aeffd700 1 --2- 172.21.15.2:0/3892249895 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11b0073550 0x7f11b019c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:35.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.062+0000 7f11ae7fc700 1 --2- 172.21.15.2:0/3892249895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11b0074b30 0x7f11b019ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:35.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.062+0000 7f11af7fe700 1 -- 172.21.15.2:0/3892249895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11b0074b30 msgr2=0x7f11b019ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:35.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.062+0000 7f11af7fe700 1 --2- 172.21.15.2:0/3892249895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11b0074b30 0x7f11b019ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.062+0000 7f11af7fe700 1 -- 172.21.15.2:0/3892249895 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11b0073550 msgr2=0x7f11b019c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:35.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.062+0000 7f11af7fe700 1 --2- 172.21.15.2:0/3892249895 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11b0073550 0x7f11b019c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.062+0000 7f11af7fe700 1 -- 172.21.15.2:0/3892249895 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a8009c40 con 0x7f11b010c890 2023-11-16T16:49:35.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.062+0000 7f11aeffd700 1 --2- 172.21.15.2:0/3892249895 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11b0073550 0x7f11b019c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:49:35.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.062+0000 7f11af7fe700 1 --2- 172.21.15.2:0/3892249895 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11b010c890 0x7f11b01a3d40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f11a8009f60 tx=0x7f11a8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:35.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.062+0000 7f1197fff700 1 -- 172.21.15.2:0/3892249895 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a800e040 con 0x7f11b010c890 2023-11-16T16:49:35.066 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.062+0000 7f1197fff700 1 -- 172.21.15.2:0/3892249895 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11a800bbf0 con 0x7f11b010c890 2023-11-16T16:49:35.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.062+0000 7f11b55ff700 1 -- 172.21.15.2:0/3892249895 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b01122a0 con 0x7f11b010c890 2023-11-16T16:49:35.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.062+0000 7f1197fff700 1 -- 172.21.15.2:0/3892249895 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a80052a0 con 0x7f11b010c890 2023-11-16T16:49:35.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.062+0000 7f11b55ff700 1 -- 172.21.15.2:0/3892249895 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b0112110 con 0x7f11b010c890 2023-11-16T16:49:35.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.063+0000 7f11b55ff700 1 -- 172.21.15.2:0/3892249895 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b01023e0 con 0x7f11b010c890 2023-11-16T16:49:35.067 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.065+0000 7f1197fff700 1 -- 172.21.15.2:0/3892249895 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f11a801d470 con 0x7f11b010c890 2023-11-16T16:49:35.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.068+0000 7f1197fff700 1 --2- 172.21.15.2:0/3892249895 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f11980a11d0 0x7f11980a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:35.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.068+0000 7f1197fff700 1 -- 172.21.15.2:0/3892249895 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f11a80ce310 con 0x7f11b010c890 2023-11-16T16:49:35.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.068+0000 7f11aeffd700 1 --2- 172.21.15.2:0/3892249895 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f11980a11d0 0x7f11980a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:35.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.069+0000 7f11aeffd700 1 --2- 172.21.15.2:0/3892249895 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f11980a11d0 0x7f11980a3680 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f11b0074f00 tx=0x7f11a0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:35.074 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.072+0000 7f1197fff700 1 -- 172.21.15.2:0/3892249895 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f11a8097730 con 0x7f11b010c890 2023-11-16T16:49:35.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:34 smithi002 ceph-mon[128712]: pgmap v348: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:49:35.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:34 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1263137919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:49:35.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:34 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1089638563' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:35.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:34 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1551232342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:35.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:34 smithi141 ceph-mon[131184]: pgmap v348: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:49:35.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:34 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1263137919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:49:35.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:34 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1089638563' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:35.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:34 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1551232342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:35.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.287+0000 7f11b55ff700 1 -- 172.21.15.2:0/3892249895 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f11b0067b00 con 0x7f11b010c890 2023-11-16T16:49:35.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.288+0000 7f1197fff700 1 -- 172.21.15.2:0/3892249895 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f11a809ad50 con 0x7f11b010c890 2023-11-16T16:49:35.290 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T16:49:35.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.290+0000 7f11b55ff700 1 -- 172.21.15.2:0/3892249895 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f11980a11d0 msgr2=0x7f11980a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:35.292 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.290+0000 7f11b55ff700 1 --2- 172.21.15.2:0/3892249895 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f11980a11d0 0x7f11980a3680 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f11b0074f00 tx=0x7f11a0009260 comp rx=0 tx=0).stop 2023-11-16T16:49:35.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.290+0000 7f11b55ff700 1 -- 172.21.15.2:0/3892249895 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11b010c890 msgr2=0x7f11b01a3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:35.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.290+0000 7f11b55ff700 1 --2- 172.21.15.2:0/3892249895 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11b010c890 0x7f11b01a3d40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f11a8009f60 tx=0x7f11a8004d10 comp rx=0 tx=0).stop 2023-11-16T16:49:35.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.290+0000 7f11b55ff700 1 -- 172.21.15.2:0/3892249895 shutdown_connections 2023-11-16T16:49:35.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.290+0000 7f11b55ff700 1 --2- 172.21.15.2:0/3892249895 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f11980a11d0 0x7f11980a3680 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.290+0000 7f11b55ff700 1 --2- 172.21.15.2:0/3892249895 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11b010c890 0x7f11b01a3d40 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.290+0000 7f11b55ff700 1 --2- 172.21.15.2:0/3892249895 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f11b0073550 0x7f11b019c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.290+0000 7f11b55ff700 1 --2- 172.21.15.2:0/3892249895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11b0074b30 0x7f11b019ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.290+0000 7f11b55ff700 1 -- 172.21.15.2:0/3892249895 >> 172.21.15.2:0/3892249895 conn(0x7f11b00fb8a0 msgr2=0x7f11b00fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:35.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.290+0000 7f11b55ff700 1 -- 172.21.15.2:0/3892249895 shutdown_connections 2023-11-16T16:49:35.293 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.291+0000 7f11b55ff700 1 -- 172.21.15.2:0/3892249895 wait complete. 2023-11-16T16:49:35.311 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T16:49:35.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:34 smithi155 ceph-mon[131607]: pgmap v348: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:49:35.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:34 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1263137919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:49:35.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:34 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1089638563' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:35.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:34 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1551232342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T16:49:35.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.517+0000 7fcab03ff700 1 -- 172.21.15.2:0/4178328810 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa81002c0 msgr2=0x7fcaa81006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:35.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.517+0000 7fcab03ff700 1 --2- 172.21.15.2:0/4178328810 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa81002c0 0x7fcaa81006d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4009990 tx=0x7fcaa4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.517+0000 7fcab03ff700 1 -- 172.21.15.2:0/4178328810 shutdown_connections 2023-11-16T16:49:35.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.517+0000 7fcab03ff700 1 --2- 172.21.15.2:0/4178328810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcaa8101e40 0x7fcaa8108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.517+0000 7fcab03ff700 1 --2- 172.21.15.2:0/4178328810 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa81002c0 0x7fcaa81006d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.517+0000 7fcab03ff700 1 --2- 172.21.15.2:0/4178328810 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcaa81014b0 0x7fcaa8101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.517+0000 7fcab03ff700 1 -- 172.21.15.2:0/4178328810 >> 172.21.15.2:0/4178328810 conn(0x7fcaa80fb8a0 msgr2=0x7fcaa80fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:35.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.517+0000 7fcab03ff700 1 -- 172.21.15.2:0/4178328810 shutdown_connections 2023-11-16T16:49:35.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.517+0000 7fcab03ff700 1 -- 172.21.15.2:0/4178328810 wait complete. 2023-11-16T16:49:35.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.518+0000 7fcab03ff700 1 Processor -- start 2023-11-16T16:49:35.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.519+0000 7fcab03ff700 1 -- start start 2023-11-16T16:49:35.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.519+0000 7fcab03ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa81002c0 0x7fcaa80748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:35.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.519+0000 7fcab03ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcaa81014b0 0x7fcaa8072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:35.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.519+0000 7fcab03ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcaa8101e40 0x7fcaa8073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:35.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.519+0000 7fcab03ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa8073bc0 con 0x7fcaa81002c0 2023-11-16T16:49:35.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.519+0000 7fcab03ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa8073d30 con 0x7fcaa8101e40 2023-11-16T16:49:35.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.519+0000 7fcab03ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa81a00f0 con 0x7fcaa81014b0 2023-11-16T16:49:35.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.519+0000 7fcaae19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa81002c0 0x7fcaa80748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:35.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.519+0000 7fcaae19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa81002c0 0x7fcaa80748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43174/0 (socket says 172.21.15.2:43174) 2023-11-16T16:49:35.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.519+0000 7fcaae19b700 1 -- 172.21.15.2:0/3542437293 learned_addr learned my addr 172.21.15.2:0/3542437293 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:35.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.520+0000 7fcaad99a700 1 --2- 172.21.15.2:0/3542437293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcaa81014b0 0x7fcaa8072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:35.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.520+0000 7fcaae99c700 1 --2- 172.21.15.2:0/3542437293 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcaa8101e40 0x7fcaa8073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:35.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.520+0000 7fcaae19b700 1 -- 172.21.15.2:0/3542437293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcaa81014b0 msgr2=0x7fcaa8072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:35.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.520+0000 7fcaae19b700 1 --2- 172.21.15.2:0/3542437293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcaa81014b0 0x7fcaa8072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.520+0000 7fcaae19b700 1 -- 172.21.15.2:0/3542437293 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcaa8101e40 msgr2=0x7fcaa8073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:35.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.520+0000 7fcaae19b700 1 --2- 172.21.15.2:0/3542437293 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcaa8101e40 0x7fcaa8073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.520+0000 7fcaae19b700 1 -- 172.21.15.2:0/3542437293 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca98009520 con 0x7fcaa81002c0 2023-11-16T16:49:35.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.520+0000 7fcaae19b700 1 --2- 172.21.15.2:0/3542437293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa81002c0 0x7fcaa80748a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcaa400b620 tx=0x7fcaa4005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:35.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.520+0000 7fca9f7fe700 1 -- 172.21.15.2:0/3542437293 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa401d070 con 0x7fcaa81002c0 2023-11-16T16:49:35.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.520+0000 7fcab03ff700 1 -- 172.21.15.2:0/3542437293 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa40095f0 con 0x7fcaa81002c0 2023-11-16T16:49:35.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.520+0000 7fca9f7fe700 1 -- 172.21.15.2:0/3542437293 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcaa4021410 con 0x7fcaa81002c0 2023-11-16T16:49:35.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.521+0000 7fcab03ff700 1 -- 172.21.15.2:0/3542437293 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa810bc90 con 0x7fcaa81002c0 2023-11-16T16:49:35.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.521+0000 7fca9f7fe700 1 -- 172.21.15.2:0/3542437293 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa40175f0 con 0x7fcaa81002c0 2023-11-16T16:49:35.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.522+0000 7fcab03ff700 1 -- 172.21.15.2:0/3542437293 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaa8107850 con 0x7fcaa81002c0 2023-11-16T16:49:35.525 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.524+0000 7fca9f7fe700 1 -- 172.21.15.2:0/3542437293 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fcaa400fb10 con 0x7fcaa81002c0 2023-11-16T16:49:35.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.525+0000 7fca9f7fe700 1 --2- 172.21.15.2:0/3542437293 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fca940a1150 0x7fca940a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:35.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.525+0000 7fca9f7fe700 1 -- 172.21.15.2:0/3542437293 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fcaa40cde30 con 0x7fcaa81002c0 2023-11-16T16:49:35.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.525+0000 7fcaad99a700 1 --2- 172.21.15.2:0/3542437293 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fca940a1150 0x7fca940a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:35.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.526+0000 7fcaad99a700 1 --2- 172.21.15.2:0/3542437293 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fca940a1150 0x7fca940a3600 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fca98009df0 tx=0x7fca98009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:35.530 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.529+0000 7fca9f7fe700 1 -- 172.21.15.2:0/3542437293 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcaa405b730 con 0x7fcaa81002c0 2023-11-16T16:49:35.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.738+0000 7fcab03ff700 1 -- 172.21.15.2:0/3542437293 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcaa8067b00 con 0x7fcaa81002c0 2023-11-16T16:49:35.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.739+0000 7fca9f7fe700 1 -- 172.21.15.2:0/3542437293 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcaa4097290 con 0x7fcaa81002c0 2023-11-16T16:49:35.741 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2023-11-16T16:49:35.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.741+0000 7fcab03ff700 1 -- 172.21.15.2:0/3542437293 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fca940a1150 msgr2=0x7fca940a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:35.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.741+0000 7fcab03ff700 1 --2- 172.21.15.2:0/3542437293 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fca940a1150 0x7fca940a3600 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fca98009df0 tx=0x7fca98009260 comp rx=0 tx=0).stop 2023-11-16T16:49:35.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.741+0000 7fcab03ff700 1 -- 172.21.15.2:0/3542437293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa81002c0 msgr2=0x7fcaa80748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:35.743 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.741+0000 7fcab03ff700 1 --2- 172.21.15.2:0/3542437293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa81002c0 0x7fcaa80748a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcaa400b620 tx=0x7fcaa4005740 comp rx=0 tx=0).stop 2023-11-16T16:49:35.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.741+0000 7fcab03ff700 1 -- 172.21.15.2:0/3542437293 shutdown_connections 2023-11-16T16:49:35.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.741+0000 7fcab03ff700 1 --2- 172.21.15.2:0/3542437293 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fca940a1150 0x7fca940a3600 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.741+0000 7fcab03ff700 1 --2- 172.21.15.2:0/3542437293 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcaa8101e40 0x7fcaa8073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.741+0000 7fcab03ff700 1 --2- 172.21.15.2:0/3542437293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa81002c0 0x7fcaa80748a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.741+0000 7fcab03ff700 1 --2- 172.21.15.2:0/3542437293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcaa81014b0 0x7fcaa8072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.741+0000 7fcab03ff700 1 -- 172.21.15.2:0/3542437293 >> 172.21.15.2:0/3542437293 conn(0x7fcaa80fb8a0 msgr2=0x7fcaa8104990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:35.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.741+0000 7fcab03ff700 1 -- 172.21.15.2:0/3542437293 shutdown_connections 2023-11-16T16:49:35.744 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.741+0000 7fcab03ff700 1 -- 172.21.15.2:0/3542437293 wait complete. 2023-11-16T16:49:35.762 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T16:49:35.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.970+0000 7f2f785ff700 1 -- 172.21.15.2:0/1180155845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f701002d0 msgr2=0x7f2f701006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:35.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.970+0000 7f2f785ff700 1 --2- 172.21.15.2:0/1180155845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f701002d0 0x7f2f701006e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0098f0 tx=0x7f2f6c009c00 comp rx=0 tx=0).stop 2023-11-16T16:49:35.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.970+0000 7f2f785ff700 1 -- 172.21.15.2:0/1180155845 shutdown_connections 2023-11-16T16:49:35.972 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.970+0000 7f2f785ff700 1 --2- 172.21.15.2:0/1180155845 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f70101e50 0x7f2f70108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.970+0000 7f2f785ff700 1 --2- 172.21.15.2:0/1180155845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f701002d0 0x7f2f701006e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.970+0000 7f2f785ff700 1 --2- 172.21.15.2:0/1180155845 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f701014c0 0x7f2f70101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.970+0000 7f2f785ff700 1 -- 172.21.15.2:0/1180155845 >> 172.21.15.2:0/1180155845 conn(0x7f2f700fb8b0 msgr2=0x7f2f700fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:35.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.970+0000 7f2f785ff700 1 -- 172.21.15.2:0/1180155845 shutdown_connections 2023-11-16T16:49:35.973 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.970+0000 7f2f785ff700 1 -- 172.21.15.2:0/1180155845 wait complete. 2023-11-16T16:49:35.975 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.971+0000 7f2f785ff700 1 Processor -- start 2023-11-16T16:49:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.972+0000 7f2f785ff700 1 -- start start 2023-11-16T16:49:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.972+0000 7f2f785ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f701002d0 0x7f2f70196130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.972+0000 7f2f785ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f701014c0 0x7f2f70196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.972+0000 7f2f785ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f70101e50 0x7f2f7019d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.972+0000 7f2f785ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7019dcb0 con 0x7f2f70101e50 2023-11-16T16:49:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.972+0000 7f2f785ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7019ddf0 con 0x7f2f701014c0 2023-11-16T16:49:35.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.972+0000 7f2f785ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7019df80 con 0x7f2f701002d0 2023-11-16T16:49:35.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.973+0000 7f2f76b9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f70101e50 0x7f2f7019d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:35.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.973+0000 7f2f7639b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f701002d0 0x7f2f70196130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:35.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.973+0000 7f2f7639b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f701002d0 0x7f2f70196130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:33156/0 (socket says 172.21.15.2:33156) 2023-11-16T16:49:35.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.973+0000 7f2f76b9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f70101e50 0x7f2f7019d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43186/0 (socket says 172.21.15.2:43186) 2023-11-16T16:49:35.977 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.973+0000 7f2f75b9a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f701014c0 0x7f2f70196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:35.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.973+0000 7f2f7639b700 1 -- 172.21.15.2:0/4177979842 learned_addr learned my addr 172.21.15.2:0/4177979842 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:35.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.973+0000 7f2f7639b700 1 -- 172.21.15.2:0/4177979842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f701014c0 msgr2=0x7f2f70196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:35.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.973+0000 7f2f7639b700 1 --2- 172.21.15.2:0/4177979842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f701014c0 0x7f2f70196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.973+0000 7f2f7639b700 1 -- 172.21.15.2:0/4177979842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f70101e50 msgr2=0x7f2f7019d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:35.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.973+0000 7f2f7639b700 1 --2- 172.21.15.2:0/4177979842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f70101e50 0x7f2f7019d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:35.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.973+0000 7f2f7639b700 1 -- 172.21.15.2:0/4177979842 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f600095f0 con 0x7f2f701002d0 2023-11-16T16:49:35.978 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.973+0000 7f2f76b9c700 1 --2- 172.21.15.2:0/4177979842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f70101e50 0x7f2f7019d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:35.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.973+0000 7f2f7639b700 1 --2- 172.21.15.2:0/4177979842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f701002d0 0x7f2f70196130 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c005e90 tx=0x7f2f6c00b730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:35.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.974+0000 7f2f677fe700 1 -- 172.21.15.2:0/4177979842 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c01c070 con 0x7f2f701002d0 2023-11-16T16:49:35.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.974+0000 7f2f785ff700 1 -- 172.21.15.2:0/4177979842 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6c009520 con 0x7f2f701002d0 2023-11-16T16:49:35.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.974+0000 7f2f785ff700 1 -- 172.21.15.2:0/4177979842 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f7010bca0 con 0x7f2f701002d0 2023-11-16T16:49:35.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.974+0000 7f2f677fe700 1 -- 172.21.15.2:0/4177979842 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2f6c00f7f0 con 0x7f2f701002d0 2023-11-16T16:49:35.979 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.974+0000 7f2f677fe700 1 -- 172.21.15.2:0/4177979842 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c003970 con 0x7f2f701002d0 2023-11-16T16:49:35.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.975+0000 7f2f785ff700 1 -- 172.21.15.2:0/4177979842 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f54005310 con 0x7f2f701002d0 2023-11-16T16:49:35.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.977+0000 7f2f677fe700 1 -- 172.21.15.2:0/4177979842 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f2f6c020030 con 0x7f2f701002d0 2023-11-16T16:49:35.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.978+0000 7f2f677fe700 1 --2- 172.21.15.2:0/4177979842 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2f5c0a1150 0x7f2f5c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:35.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.978+0000 7f2f677fe700 1 -- 172.21.15.2:0/4177979842 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f2f6c0ce500 con 0x7f2f701002d0 2023-11-16T16:49:35.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.978+0000 7f2f75b9a700 1 --2- 172.21.15.2:0/4177979842 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2f5c0a1150 0x7f2f5c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:35.981 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.979+0000 7f2f75b9a700 1 --2- 172.21.15.2:0/4177979842 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2f5c0a1150 0x7f2f5c0a3600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2f60005fd0 tx=0x7f2f60009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:35.982 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:35.980+0000 7f2f677fe700 1 -- 172.21.15.2:0/4177979842 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2f6c097920 con 0x7f2f701002d0 2023-11-16T16:49:36.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:35 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3892249895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:36.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:35 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3542437293' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:36.192 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.191+0000 7f2f785ff700 1 -- 172.21.15.2:0/4177979842 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f2f54005f60 con 0x7f2f701002d0 2023-11-16T16:49:36.193 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.192+0000 7f2f677fe700 1 -- 172.21.15.2:0/4177979842 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2f6c00e3e0 con 0x7f2f701002d0 2023-11-16T16:49:36.193 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T16:49:36.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.194+0000 7f2f785ff700 1 -- 172.21.15.2:0/4177979842 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2f5c0a1150 msgr2=0x7f2f5c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.194+0000 7f2f785ff700 1 --2- 172.21.15.2:0/4177979842 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2f5c0a1150 0x7f2f5c0a3600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2f60005fd0 tx=0x7f2f60009310 comp rx=0 tx=0).stop 2023-11-16T16:49:36.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.194+0000 7f2f785ff700 1 -- 172.21.15.2:0/4177979842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f701002d0 msgr2=0x7f2f70196130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.196 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.194+0000 7f2f785ff700 1 --2- 172.21.15.2:0/4177979842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f701002d0 0x7f2f70196130 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c005e90 tx=0x7f2f6c00b730 comp rx=0 tx=0).stop 2023-11-16T16:49:36.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.194+0000 7f2f785ff700 1 -- 172.21.15.2:0/4177979842 shutdown_connections 2023-11-16T16:49:36.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.194+0000 7f2f785ff700 1 --2- 172.21.15.2:0/4177979842 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f2f5c0a1150 0x7f2f5c0a3600 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.194+0000 7f2f785ff700 1 --2- 172.21.15.2:0/4177979842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f701002d0 0x7f2f70196130 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.194+0000 7f2f785ff700 1 --2- 172.21.15.2:0/4177979842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f70101e50 0x7f2f7019d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.194+0000 7f2f785ff700 1 --2- 172.21.15.2:0/4177979842 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2f701014c0 0x7f2f70196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.194+0000 7f2f785ff700 1 -- 172.21.15.2:0/4177979842 >> 172.21.15.2:0/4177979842 conn(0x7f2f700fb8b0 msgr2=0x7f2f700fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:36.197 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.194+0000 7f2f785ff700 1 -- 172.21.15.2:0/4177979842 shutdown_connections 2023-11-16T16:49:36.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.194+0000 7f2f785ff700 1 -- 172.21.15.2:0/4177979842 wait complete. 2023-11-16T16:49:36.215 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T16:49:36.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:35 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3892249895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:36.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:35 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3542437293' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:36.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:35 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3892249895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T16:49:36.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:35 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3542437293' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T16:49:36.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.417+0000 7f4585bff700 1 -- 172.21.15.2:0/219864613 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4580100290 msgr2=0x7f45801006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.417+0000 7f4585bff700 1 --2- 172.21.15.2:0/219864613 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4580100290 0x7f45801006a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4570009990 tx=0x7f4570009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.417+0000 7f4585bff700 1 -- 172.21.15.2:0/219864613 shutdown_connections 2023-11-16T16:49:36.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.417+0000 7f4585bff700 1 --2- 172.21.15.2:0/219864613 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4580101e10 0x7f45801086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.419 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.417+0000 7f4585bff700 1 --2- 172.21.15.2:0/219864613 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4580100290 0x7f45801006a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.417+0000 7f4585bff700 1 --2- 172.21.15.2:0/219864613 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4580101480 0x7f45801018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.417+0000 7f4585bff700 1 -- 172.21.15.2:0/219864613 >> 172.21.15.2:0/219864613 conn(0x7f45800fb870 msgr2=0x7f45800fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:36.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.417+0000 7f4585bff700 1 -- 172.21.15.2:0/219864613 shutdown_connections 2023-11-16T16:49:36.420 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.417+0000 7f4585bff700 1 -- 172.21.15.2:0/219864613 wait complete. 2023-11-16T16:49:36.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.418+0000 7f4585bff700 1 Processor -- start 2023-11-16T16:49:36.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.418+0000 7f4585bff700 1 -- start start 2023-11-16T16:49:36.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.419+0000 7f4585bff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4580100290 0x7f4580195fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.422 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.419+0000 7f4585bff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4580101480 0x7f45801964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.419+0000 7f4585bff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4580101e10 0x7f458019d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.419+0000 7f4585bff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458019db20 con 0x7f4580101e10 2023-11-16T16:49:36.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.419+0000 7f4585bff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458019dc60 con 0x7f4580101480 2023-11-16T16:49:36.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.419+0000 7f4585bff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458019ddf0 con 0x7f4580100290 2023-11-16T16:49:36.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.419+0000 7f457ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4580101e10 0x7f458019d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.419+0000 7f457ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4580101e10 0x7f458019d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43212/0 (socket says 172.21.15.2:43212) 2023-11-16T16:49:36.423 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.419+0000 7f457ffff700 1 -- 172.21.15.2:0/4128069396 learned_addr learned my addr 172.21.15.2:0/4128069396 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:36.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.419+0000 7f4577fff700 1 --2- 172.21.15.2:0/4128069396 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4580101480 0x7f45801964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.419+0000 7f457f7fe700 1 --2- 172.21.15.2:0/4128069396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4580100290 0x7f4580195fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.419+0000 7f457ffff700 1 -- 172.21.15.2:0/4128069396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4580100290 msgr2=0x7f4580195fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.419+0000 7f457ffff700 1 --2- 172.21.15.2:0/4128069396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4580100290 0x7f4580195fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.419+0000 7f457ffff700 1 -- 172.21.15.2:0/4128069396 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4580101480 msgr2=0x7f45801964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.419+0000 7f457ffff700 1 --2- 172.21.15.2:0/4128069396 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4580101480 0x7f45801964e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.424 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.419+0000 7f457ffff700 1 -- 172.21.15.2:0/4128069396 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45700095f0 con 0x7f4580101e10 2023-11-16T16:49:36.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.420+0000 7f457ffff700 1 --2- 172.21.15.2:0/4128069396 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4580101e10 0x7f458019d550 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f457800ef30 tx=0x7f457800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:36.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.420+0000 7f457d7fa700 1 -- 172.21.15.2:0/4128069396 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578018070 con 0x7f4580101e10 2023-11-16T16:49:36.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.420+0000 7f4585bff700 1 -- 172.21.15.2:0/4128069396 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458010bc40 con 0x7f4580101e10 2023-11-16T16:49:36.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.420+0000 7f457d7fa700 1 -- 172.21.15.2:0/4128069396 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f457800f040 con 0x7f4580101e10 2023-11-16T16:49:36.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.420+0000 7f457d7fa700 1 -- 172.21.15.2:0/4128069396 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457800cdb0 con 0x7f4580101e10 2023-11-16T16:49:36.425 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.420+0000 7f4585bff700 1 -- 172.21.15.2:0/4128069396 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f458010bab0 con 0x7f4580101e10 2023-11-16T16:49:36.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.421+0000 7f4585bff700 1 -- 172.21.15.2:0/4128069396 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4580107210 con 0x7f4580101e10 2023-11-16T16:49:36.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.423+0000 7f457d7fa700 1 -- 172.21.15.2:0/4128069396 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f4578003a00 con 0x7f4580101e10 2023-11-16T16:49:36.426 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.424+0000 7f457d7fa700 1 --2- 172.21.15.2:0/4128069396 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f456c0a1150 0x7f456c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.424+0000 7f457d7fa700 1 -- 172.21.15.2:0/4128069396 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f45780cc880 con 0x7f4580101e10 2023-11-16T16:49:36.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.425+0000 7f457f7fe700 1 --2- 172.21.15.2:0/4128069396 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f456c0a1150 0x7f456c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.427 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.425+0000 7f457f7fe700 1 --2- 172.21.15.2:0/4128069396 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f456c0a1150 0x7f456c0a3600 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f457000b2e0 tx=0x7f4570005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:36.429 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.428+0000 7f457d7fa700 1 -- 172.21.15.2:0/4128069396 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4578095ca0 con 0x7f4580101e10 2023-11-16T16:49:36.681 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.680+0000 7f4585bff700 1 -- 172.21.15.2:0/4128069396 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f458002cfe0 con 0x7f4580101e10 2023-11-16T16:49:36.682 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.681+0000 7f457d7fa700 1 -- 172.21.15.2:0/4128069396 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10811 (secure 0 0 0) 0x7f45780992c0 con 0x7f4580101e10 2023-11-16T16:49:36.682 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:49:36.683 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/1610211225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":1610211225},{"type":"v1","addr":"172.21.15.2:6837","nonce":1610211225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14538,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2583261818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2583261818},{"type":"v1","addr":"172.21.15.2:6839","nonce":2583261818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14550,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3170991923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3170991923},{"type":"v1","addr":"172.21.15.2:6841","nonce":3170991923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:26.349176+0000","modified":"2023-11-16T16:48:05.774072+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24544,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.141:6835/3516869188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3516869188},{"type":"v1","addr":"172.21.15.141:6835","nonce":3516869188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/2527313191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":2527313191},{"type":"v1","addr":"172.21.15.155:6835","nonce":2527313191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24523":{"gid":24523,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1607756751","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1607756751},{"type":"v1","addr":"172.21.15.155:6837","nonce":1607756751}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.141:6837/2934891475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6836","nonce":2934891475},{"type":"v1","addr":"172.21.15.141:6837","nonce":2934891475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24544":{"gid":24544,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1158457009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1158457009},{"type":"v1","addr":"172.21.15.155:6839","nonce":1158457009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6839/3916358272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6838","nonce":3916358272},{"type":"v1","addr":"172.21.15.141:6839","nonce":3916358272}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T16:49:36.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.683+0000 7f4585bff700 1 -- 172.21.15.2:0/4128069396 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f456c0a1150 msgr2=0x7f456c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.683+0000 7f4585bff700 1 --2- 172.21.15.2:0/4128069396 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f456c0a1150 0x7f456c0a3600 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f457000b2e0 tx=0x7f4570005fb0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.683+0000 7f4585bff700 1 -- 172.21.15.2:0/4128069396 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4580101e10 msgr2=0x7f458019d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.683+0000 7f4585bff700 1 --2- 172.21.15.2:0/4128069396 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4580101e10 0x7f458019d550 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f457800ef30 tx=0x7f457800c3a0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.685 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.683+0000 7f4585bff700 1 -- 172.21.15.2:0/4128069396 shutdown_connections 2023-11-16T16:49:36.686 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.683+0000 7f4585bff700 1 --2- 172.21.15.2:0/4128069396 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f456c0a1150 0x7f456c0a3600 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.686 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.683+0000 7f4585bff700 1 --2- 172.21.15.2:0/4128069396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4580100290 0x7f4580195fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.686 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.683+0000 7f4585bff700 1 --2- 172.21.15.2:0/4128069396 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4580101e10 0x7f458019d550 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.686 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.683+0000 7f4585bff700 1 --2- 172.21.15.2:0/4128069396 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4580101480 0x7f45801964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.686 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.683+0000 7f4585bff700 1 -- 172.21.15.2:0/4128069396 >> 172.21.15.2:0/4128069396 conn(0x7f45800fb870 msgr2=0x7f45800fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:36.686 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.683+0000 7f4585bff700 1 -- 172.21.15.2:0/4128069396 shutdown_connections 2023-11-16T16:49:36.687 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.683+0000 7f4585bff700 1 -- 172.21.15.2:0/4128069396 wait complete. 2023-11-16T16:49:36.687 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 26 2023-11-16T16:49:36.704 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:49:36.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.908+0000 7f62d91ff700 1 -- 172.21.15.2:0/2999414129 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d410c860 msgr2=0x7f62d410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.908+0000 7f62d91ff700 1 --2- 172.21.15.2:0/2999414129 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d410c860 0x7f62d410ec40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00b0a0 tx=0x7f62cc00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.908+0000 7f62d91ff700 1 -- 172.21.15.2:0/2999414129 shutdown_connections 2023-11-16T16:49:36.910 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.908+0000 7f62d91ff700 1 --2- 172.21.15.2:0/2999414129 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62d4074b30 0x7f62d4072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.908+0000 7f62d91ff700 1 --2- 172.21.15.2:0/2999414129 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d410c860 0x7f62d410ec40 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.908+0000 7f62d91ff700 1 --2- 172.21.15.2:0/2999414129 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f62d4073550 0x7f62d40739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.908+0000 7f62d91ff700 1 -- 172.21.15.2:0/2999414129 >> 172.21.15.2:0/2999414129 conn(0x7f62d40fb870 msgr2=0x7f62d40fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:36.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.908+0000 7f62d91ff700 1 -- 172.21.15.2:0/2999414129 shutdown_connections 2023-11-16T16:49:36.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.908+0000 7f62d91ff700 1 -- 172.21.15.2:0/2999414129 wait complete. 2023-11-16T16:49:36.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.909+0000 7f62d91ff700 1 Processor -- start 2023-11-16T16:49:36.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.909+0000 7f62d91ff700 1 -- start start 2023-11-16T16:49:36.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.909+0000 7f62d91ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f62d4073550 0x7f62d419c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.910+0000 7f62d91ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d4074b30 0x7f62d419cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.910+0000 7f62d91ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62d410c860 0x7f62d41a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.910+0000 7f62d91ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d41a4220 con 0x7f62d4074b30 2023-11-16T16:49:36.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.910+0000 7f62d91ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d41a4360 con 0x7f62d4073550 2023-11-16T16:49:36.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.910+0000 7f62d91ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d41a44f0 con 0x7f62d410c860 2023-11-16T16:49:36.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.910+0000 7f62d259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d4074b30 0x7f62d419cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.910+0000 7f62d2d9d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f62d4073550 0x7f62d419c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.915 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.910+0000 7f62d259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d4074b30 0x7f62d419cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43228/0 (socket says 172.21.15.2:43228) 2023-11-16T16:49:36.915 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.910+0000 7f62d259c700 1 -- 172.21.15.2:0/1634032009 learned_addr learned my addr 172.21.15.2:0/1634032009 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:36.915 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.910+0000 7f62d2d9d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f62d4073550 0x7f62d419c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:41212/0 (socket says 172.21.15.2:41212) 2023-11-16T16:49:36.915 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.910+0000 7f62d359e700 1 --2- 172.21.15.2:0/1634032009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62d410c860 0x7f62d41a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.915 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.910+0000 7f62d259c700 1 -- 172.21.15.2:0/1634032009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62d410c860 msgr2=0x7f62d41a3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.915 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.910+0000 7f62d259c700 1 --2- 172.21.15.2:0/1634032009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62d410c860 0x7f62d41a3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.910+0000 7f62d259c700 1 -- 172.21.15.2:0/1634032009 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f62d4073550 msgr2=0x7f62d419c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:36.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.910+0000 7f62d259c700 1 --2- 172.21.15.2:0/1634032009 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f62d4073550 0x7f62d419c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:36.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.910+0000 7f62d259c700 1 -- 172.21.15.2:0/1634032009 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62cc009d00 con 0x7f62d4074b30 2023-11-16T16:49:36.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.911+0000 7f62d259c700 1 --2- 172.21.15.2:0/1634032009 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d4074b30 0x7f62d419cbe0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f62c800e970 tx=0x7f62c800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:36.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.911+0000 7f62bbfff700 1 -- 172.21.15.2:0/1634032009 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c8009790 con 0x7f62d4074b30 2023-11-16T16:49:36.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.911+0000 7f62bbfff700 1 -- 172.21.15.2:0/1634032009 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62c800c7e0 con 0x7f62d4074b30 2023-11-16T16:49:36.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.911+0000 7f62d91ff700 1 -- 172.21.15.2:0/1634032009 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d4112270 con 0x7f62d4074b30 2023-11-16T16:49:36.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.911+0000 7f62bbfff700 1 -- 172.21.15.2:0/1634032009 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c8003c70 con 0x7f62d4074b30 2023-11-16T16:49:36.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.911+0000 7f62d91ff700 1 -- 172.21.15.2:0/1634032009 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d41120e0 con 0x7f62d4074b30 2023-11-16T16:49:36.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.912+0000 7f62d91ff700 1 -- 172.21.15.2:0/1634032009 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d41023b0 con 0x7f62d4074b30 2023-11-16T16:49:36.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.914+0000 7f62bbfff700 1 -- 172.21.15.2:0/1634032009 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f62c800cc50 con 0x7f62d4074b30 2023-11-16T16:49:36.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.915+0000 7f62bbfff700 1 --2- 172.21.15.2:0/1634032009 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62bc0a1150 0x7f62bc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:36.918 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.915+0000 7f62bbfff700 1 -- 172.21.15.2:0/1634032009 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f62c8014070 con 0x7f62d4074b30 2023-11-16T16:49:36.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.915+0000 7f62d2d9d700 1 --2- 172.21.15.2:0/1634032009 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62bc0a1150 0x7f62bc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:36.919 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.916+0000 7f62d2d9d700 1 --2- 172.21.15.2:0/1634032009 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62bc0a1150 0x7f62bc0a3600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f62d4074f00 tx=0x7f62c4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:36.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:36.919+0000 7f62bbfff700 1 -- 172.21.15.2:0/1634032009 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f62c80d3080 con 0x7f62d4074b30 2023-11-16T16:49:37.131 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.129+0000 7f62d91ff700 1 -- 172.21.15.2:0/1634032009 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62d402cfe0 con 0x7f62d4074b30 2023-11-16T16:49:37.133 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.131+0000 7f62bbfff700 1 -- 172.21.15.2:0/1634032009 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24567 (secure 0 0 0) 0x7f62c8096210 con 0x7f62d4074b30 2023-11-16T16:49:37.133 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:49:37.134 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":93,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:48:45.491739+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:24.458452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:25.786005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:29.313181+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:49:37.135 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.133+0000 7f62d91ff700 1 -- 172.21.15.2:0/1634032009 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62bc0a1150 msgr2=0x7f62bc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.136 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.133+0000 7f62d91ff700 1 --2- 172.21.15.2:0/1634032009 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62bc0a1150 0x7f62bc0a3600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f62d4074f00 tx=0x7f62c4009260 comp rx=0 tx=0).stop 2023-11-16T16:49:37.136 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.133+0000 7f62d91ff700 1 -- 172.21.15.2:0/1634032009 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d4074b30 msgr2=0x7f62d419cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.136 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.133+0000 7f62d91ff700 1 --2- 172.21.15.2:0/1634032009 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d4074b30 0x7f62d419cbe0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f62c800e970 tx=0x7f62c800ec80 comp rx=0 tx=0).stop 2023-11-16T16:49:37.136 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.133+0000 7f62d91ff700 1 -- 172.21.15.2:0/1634032009 shutdown_connections 2023-11-16T16:49:37.136 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.133+0000 7f62d91ff700 1 --2- 172.21.15.2:0/1634032009 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f62bc0a1150 0x7f62bc0a3600 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.137 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.133+0000 7f62d91ff700 1 --2- 172.21.15.2:0/1634032009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62d410c860 0x7f62d41a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.137 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.133+0000 7f62d91ff700 1 --2- 172.21.15.2:0/1634032009 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f62d4073550 0x7f62d419c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.137 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.133+0000 7f62d91ff700 1 --2- 172.21.15.2:0/1634032009 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d4074b30 0x7f62d419cbe0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.137 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.133+0000 7f62d91ff700 1 -- 172.21.15.2:0/1634032009 >> 172.21.15.2:0/1634032009 conn(0x7f62d40fb870 msgr2=0x7f62d40fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.137 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.134+0000 7f62d91ff700 1 -- 172.21.15.2:0/1634032009 shutdown_connections 2023-11-16T16:49:37.137 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.134+0000 7f62d91ff700 1 -- 172.21.15.2:0/1634032009 wait complete. 2023-11-16T16:49:37.152 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:36 smithi002 ceph-mon[128712]: pgmap v349: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T16:49:37.153 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:36 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/4177979842' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:37.153 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:36 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/4128069396' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:49:37.156 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-11-16T16:47:26.349176+0000', 'modified': '2023-11-16T16:48:05.774072+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24590, 'mds_1': 24544, 'mds_2': 24542}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24506': {'gid': 24506, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.141:6835/3516869188', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6834', 'nonce': 3516869188}, {'type': 'v1', 'addr': '172.21.15.141:6835', 'nonce': 3516869188}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24511': {'gid': 24511, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.155:6835/2527313191', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6834', 'nonce': 2527313191}, {'type': 'v1', 'addr': '172.21.15.155:6835', 'nonce': 2527313191}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24523': {'gid': 24523, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.155:6837/1607756751', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6836', 'nonce': 1607756751}, {'type': 'v1', 'addr': '172.21.15.155:6837', 'nonce': 1607756751}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24542': {'gid': 24542, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.141:6837/2934891475', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6836', 'nonce': 2934891475}, {'type': 'v1', 'addr': '172.21.15.141:6837', 'nonce': 2934891475}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24544': {'gid': 24544, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.155:6839/1158457009', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6838', 'nonce': 1158457009}, {'type': 'v1', 'addr': '172.21.15.155:6839', 'nonce': 1158457009}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24590': {'gid': 24590, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.141:6839/3916358272', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6838', 'nonce': 3916358272}, {'type': 'v1', 'addr': '172.21.15.141:6839', 'nonce': 3916358272}]}, 'join_fscid': -1, 'export_targets': [1, 2], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-16T16:49:37.156 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-11-16T16:49:37.156 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-11-16T16:49:37.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:36 smithi141 ceph-mon[131184]: pgmap v349: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T16:49:37.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:36 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/4177979842' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:37.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:36 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/4128069396' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:49:37.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.360+0000 7f28bf7ff700 1 -- 172.21.15.2:0/435928104 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28b8101db0 msgr2=0x7f28b8108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.360+0000 7f28bf7ff700 1 --2- 172.21.15.2:0/435928104 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28b8101db0 0x7f28b8108670 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f28b000b0a0 tx=0x7f28b000b3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.360+0000 7f28bf7ff700 1 -- 172.21.15.2:0/435928104 shutdown_connections 2023-11-16T16:49:37.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.360+0000 7f28bf7ff700 1 --2- 172.21.15.2:0/435928104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28b8100230 0x7f28b8100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.360+0000 7f28bf7ff700 1 --2- 172.21.15.2:0/435928104 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28b8101db0 0x7f28b8108670 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.360+0000 7f28bf7ff700 1 --2- 172.21.15.2:0/435928104 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28b8101420 0x7f28b8101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.362 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.360+0000 7f28bf7ff700 1 -- 172.21.15.2:0/435928104 >> 172.21.15.2:0/435928104 conn(0x7f28b80fb810 msgr2=0x7f28b80fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.360+0000 7f28bf7ff700 1 -- 172.21.15.2:0/435928104 shutdown_connections 2023-11-16T16:49:37.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.360+0000 7f28bf7ff700 1 -- 172.21.15.2:0/435928104 wait complete. 2023-11-16T16:49:37.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.361+0000 7f28bf7ff700 1 Processor -- start 2023-11-16T16:49:37.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.361+0000 7f28bf7ff700 1 -- start start 2023-11-16T16:49:37.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.362+0000 7f28bf7ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28b8100230 0x7f28b81960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.362+0000 7f28bf7ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28b8101420 0x7f28b81965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.362+0000 7f28bf7ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28b8101db0 0x7f28b819d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.362+0000 7f28bf7ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b819dc20 con 0x7f28b8101420 2023-11-16T16:49:37.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.362+0000 7f28bf7ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b819dd60 con 0x7f28b8101db0 2023-11-16T16:49:37.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.362+0000 7f28bf7ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b819def0 con 0x7f28b8100230 2023-11-16T16:49:37.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.362+0000 7f28bcd9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28b8101420 0x7f28b81965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.362+0000 7f28bd59b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28b8100230 0x7f28b81960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.362+0000 7f28bcd9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28b8101420 0x7f28b81965e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43236/0 (socket says 172.21.15.2:43236) 2023-11-16T16:49:37.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.362+0000 7f28bcd9a700 1 -- 172.21.15.2:0/4076601376 learned_addr learned my addr 172.21.15.2:0/4076601376 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:37.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.362+0000 7f28bdd9c700 1 --2- 172.21.15.2:0/4076601376 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28b8101db0 0x7f28b819d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.363+0000 7f28bcd9a700 1 -- 172.21.15.2:0/4076601376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28b8100230 msgr2=0x7f28b81960a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.363+0000 7f28bcd9a700 1 --2- 172.21.15.2:0/4076601376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28b8100230 0x7f28b81960a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.363+0000 7f28bcd9a700 1 -- 172.21.15.2:0/4076601376 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28b8101db0 msgr2=0x7f28b819d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.363+0000 7f28bcd9a700 1 --2- 172.21.15.2:0/4076601376 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28b8101db0 0x7f28b819d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.363+0000 7f28bcd9a700 1 -- 172.21.15.2:0/4076601376 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b0009d00 con 0x7f28b8101420 2023-11-16T16:49:37.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.363+0000 7f28bd59b700 1 --2- 172.21.15.2:0/4076601376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28b8100230 0x7f28b81960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:37.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.363+0000 7f28bdd9c700 1 --2- 172.21.15.2:0/4076601376 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28b8101db0 0x7f28b819d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:37.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.363+0000 7f28bcd9a700 1 --2- 172.21.15.2:0/4076601376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28b8101420 0x7f28b81965e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f28a800e9a0 tx=0x7f28a800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.363+0000 7f28ae7fc700 1 -- 172.21.15.2:0/4076601376 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a8009790 con 0x7f28b8101420 2023-11-16T16:49:37.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.363+0000 7f28ae7fc700 1 -- 172.21.15.2:0/4076601376 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28a8010430 con 0x7f28b8101420 2023-11-16T16:49:37.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.363+0000 7f28bf7ff700 1 -- 172.21.15.2:0/4076601376 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b810bc00 con 0x7f28b8101420 2023-11-16T16:49:37.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.363+0000 7f28ae7fc700 1 -- 172.21.15.2:0/4076601376 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a8018680 con 0x7f28b8101420 2023-11-16T16:49:37.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.363+0000 7f28bf7ff700 1 -- 172.21.15.2:0/4076601376 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b810ba70 con 0x7f28b8101420 2023-11-16T16:49:37.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:36 smithi155 ceph-mon[131607]: pgmap v349: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T16:49:37.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:36 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/4177979842' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T16:49:37.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:36 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/4128069396' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T16:49:37.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.366+0000 7f28ae7fc700 1 -- 172.21.15.2:0/4076601376 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f28a80105a0 con 0x7f28b8101420 2023-11-16T16:49:37.370 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.368+0000 7f28d313db80 1 Processor -- start 2023-11-16T16:49:37.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.369+0000 7f28d313db80 1 -- start start 2023-11-16T16:49:37.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.369+0000 7f28ae7fc700 1 --2- 172.21.15.2:0/4076601376 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f28a40a11d0 0x7f28a40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.369+0000 7f28d313db80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b4000 0x5654c59b4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.369+0000 7f28d313db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 0x5654c59b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.369+0000 7f28d313db80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b54c0 0x5654c59b5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.369+0000 7f28d313db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c58b5010 con 0x5654c59b4950 2023-11-16T16:49:37.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.369+0000 7f28d313db80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c5907850 con 0x5654c59b4000 2023-11-16T16:49:37.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.369+0000 7f28bdd9c700 1 --2- 172.21.15.2:0/4076601376 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f28a40a11d0 0x7f28a40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.369+0000 7f28d313db80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c5900ce0 con 0x5654c59b54c0 2023-11-16T16:49:37.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.369+0000 7f28ae7fc700 1 -- 172.21.15.2:0/4076601376 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f28a8014070 con 0x7f28b8101420 2023-11-16T16:49:37.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.369+0000 7f28bcd9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 0x5654c59b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.369+0000 7f28bd59b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b4000 0x5654c59b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.369+0000 7f28bdd9c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b54c0 0x5654c59b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.369+0000 7f28bcd9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 0x5654c59b4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43250/0 (socket says 172.21.15.2:43250) 2023-11-16T16:49:37.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.369+0000 7f28bcd9a700 1 -- 172.21.15.2:0/3963247457 learned_addr learned my addr 172.21.15.2:0/3963247457 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:37.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.370+0000 7f28bcd9a700 1 -- 172.21.15.2:0/3963247457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b54c0 msgr2=0x5654c59b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.370+0000 7f28bcd9a700 1 --2- 172.21.15.2:0/3963247457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b54c0 0x5654c59b5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.370+0000 7f28bcd9a700 1 -- 172.21.15.2:0/3963247457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b4000 msgr2=0x5654c59b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.370+0000 7f28bcd9a700 1 --2- 172.21.15.2:0/3963247457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b4000 0x5654c59b4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.370+0000 7f28bcd9a700 1 -- 172.21.15.2:0/3963247457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654c57825e0 con 0x5654c59b4950 2023-11-16T16:49:37.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.370+0000 7f28bdd9c700 1 --2- 172.21.15.2:0/4076601376 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f28a40a11d0 0x7f28a40a3680 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f28b000b780 tx=0x7f28b0007d70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.370+0000 7f28bcd9a700 1 --2- 172.21.15.2:0/3963247457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 0x5654c59b4ee0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f28a80a1db0 tx=0x7f28a8099ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b565d5698bceb08 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.370+0000 7f28a37fe700 1 -- 172.21.15.2:0/3963247457 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a80a22e0 con 0x5654c59b4950 2023-11-16T16:49:37.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.370+0000 7f28a37fe700 1 -- 172.21.15.2:0/3963247457 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28a80ca690 con 0x5654c59b4950 2023-11-16T16:49:37.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.370+0000 7f28a37fe700 1 -- 172.21.15.2:0/3963247457 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a80a1e50 con 0x5654c59b4950 2023-11-16T16:49:37.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.371+0000 7f28d313db80 1 -- 172.21.15.2:0/3963247457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 msgr2=0x5654c59b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.371+0000 7f28d313db80 1 --2- 172.21.15.2:0/3963247457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 0x5654c59b4ee0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f28a80a1db0 tx=0x7f28a8099ae0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.371+0000 7f28d313db80 1 -- 172.21.15.2:0/3963247457 shutdown_connections 2023-11-16T16:49:37.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.371+0000 7f28d313db80 1 --2- 172.21.15.2:0/3963247457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b54c0 0x5654c59b5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.371+0000 7f28d313db80 1 --2- 172.21.15.2:0/3963247457 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b4000 0x5654c59b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.374 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.371+0000 7f28d313db80 1 --2- 172.21.15.2:0/3963247457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 0x5654c59b4ee0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.371+0000 7f28d313db80 1 -- 172.21.15.2:0/3963247457 >> 172.21.15.2:0/3963247457 conn(0x5654c59a5310 msgr2=0x5654c59aa620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.371+0000 7f28d313db80 1 -- 172.21.15.2:0/3963247457 shutdown_connections 2023-11-16T16:49:37.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.371+0000 7f28d313db80 1 -- 172.21.15.2:0/3963247457 wait complete. 2023-11-16T16:49:37.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.372+0000 7f28d313db80 1 -- start start 2023-11-16T16:49:37.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.372+0000 7f28d313db80 1 Processor -- start 2023-11-16T16:49:37.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.372+0000 7f28d313db80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b4000 0x5654c59bda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.372+0000 7f28d313db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 0x5654c59bdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.372+0000 7f28d313db80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b54c0 0x5654c59c4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.372+0000 7f28d313db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c58a2890 con 0x5654c59b4950 2023-11-16T16:49:37.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.372+0000 7f28d313db80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c58a2530 con 0x5654c59b4000 2023-11-16T16:49:37.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.372+0000 7f28d313db80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c57848d0 con 0x5654c59b54c0 2023-11-16T16:49:37.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.372+0000 7f28d313db80 10 client.0 _ll_register_callbacks cb 0x5654c58c0730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:49:37.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.373+0000 7f28bcd9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 0x5654c59bdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.373+0000 7f28bcd9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 0x5654c59bdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43260/0 (socket says 172.21.15.2:43260) 2023-11-16T16:49:37.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.373+0000 7f28bcd9a700 1 -- 172.21.15.2:0/3983982392 learned_addr learned my addr 172.21.15.2:0/3983982392 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:37.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.373+0000 7f28bd59b700 1 --2- 172.21.15.2:0/3983982392 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b4000 0x5654c59bda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.373+0000 7f28bdd9c700 1 --2- 172.21.15.2:0/3983982392 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b54c0 0x5654c59c4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.373+0000 7f28bcd9a700 1 -- 172.21.15.2:0/3983982392 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b54c0 msgr2=0x5654c59c4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.373+0000 7f28bcd9a700 1 --2- 172.21.15.2:0/3983982392 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b54c0 0x5654c59c4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.373+0000 7f28bcd9a700 1 -- 172.21.15.2:0/3983982392 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b4000 msgr2=0x5654c59bda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.373+0000 7f28bcd9a700 1 --2- 172.21.15.2:0/3983982392 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b4000 0x5654c59bda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.373+0000 7f28bcd9a700 1 -- 172.21.15.2:0/3983982392 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b810ba70 con 0x5654c59b4950 2023-11-16T16:49:37.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.373+0000 7f28bcd9a700 1 --2- 172.21.15.2:0/3983982392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 0x5654c59bdfb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f28a80af040 tx=0x7f28a80a2080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f1a35a9670bae3d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.373+0000 7f28a0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:49:37.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.373+0000 7f28a0ff9700 1 -- 172.21.15.2:0/3983982392 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a80a22e0 con 0x5654c59b4950 2023-11-16T16:49:37.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.373+0000 7f28a0ff9700 1 -- 172.21.15.2:0/3983982392 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f28a80adda0 con 0x5654c59b4950 2023-11-16T16:49:37.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.373+0000 7f28d313db80 1 -- 172.21.15.2:0/3983982392 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5654c59a2ad0 con 0x5654c59b4950 2023-11-16T16:49:37.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.374+0000 7f28a0ff9700 1 -- 172.21.15.2:0/3983982392 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28a80ca250 con 0x5654c59b4950 2023-11-16T16:49:37.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.374+0000 7f28d313db80 10 client.15300 fetch_fsmap learned FSMap version 26 2023-11-16T16:49:37.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.374+0000 7f28d313db80 1 -- 172.21.15.2:0/3983982392 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5654c58c7140 con 0x5654c59b4950 2023-11-16T16:49:37.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.374+0000 7f28a0ff9700 1 -- 172.21.15.2:0/3983982392 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5654c59a2ad0 con 0x5654c59b4950 2023-11-16T16:49:37.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.374+0000 7f28a0ff9700 1 -- 172.21.15.2:0/3983982392 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f28a80831e0 con 0x5654c59b4950 2023-11-16T16:49:37.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.374+0000 7f28d313db80 10 client.15300 fetch_fsmap finished waiting for FSMap version 26 2023-11-16T16:49:37.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.374+0000 7f28d313db80 10 client.15300 resolve_mds: resolved name 'h' to daemon mds.h 2023-11-16T16:49:37.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.374+0000 7f28d313db80 20 client.15300 populate_metadata read hostname 'smithi002' 2023-11-16T16:49:37.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.374+0000 7f28d313db80 1 --2- 172.21.15.2:0/3983982392 >> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] conn(0x5654c59c8900 0x5654c59c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.374+0000 7f28d313db80 4 client.15300 mds_command: new command op to 24542 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:49:37.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.374+0000 7f28d313db80 1 -- 172.21.15.2:0/3983982392 --> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5654c58c3050 con 0x5654c59c8900 2023-11-16T16:49:37.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.375+0000 7f28bd59b700 1 --2- 172.21.15.2:0/3983982392 >> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] conn(0x5654c59c8900 0x5654c59c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.375+0000 7f28bd59b700 1 --2- 172.21.15.2:0/3983982392 >> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] conn(0x5654c59c8900 0x5654c59c7f20 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ecf49b112a124b1b server_cookie=98f16f5d41c971f4 in_seq=0 out_seq=0 2023-11-16T16:49:37.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.375+0000 7f28a0ff9700 10 client.15300 ms_handle_connect on v2:172.21.15.141:6836/2934891475 2023-11-16T16:49:37.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.376+0000 7f28a0ff9700 1 -- 172.21.15.2:0/3983982392 <== mds.2 v2:172.21.15.141:6836/2934891475 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5654c58c3050 con 0x5654c59c8900 2023-11-16T16:49:37.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.376+0000 7f28a0ff9700 10 client.15300 handle_command_reply: tid=0 2023-11-16T16:49:37.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.376+0000 7f28d313db80 1 client.15300 shutdown 2023-11-16T16:49:37.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.376+0000 7f28d313db80 1 -- 172.21.15.2:0/3983982392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 msgr2=0x5654c59bdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.376+0000 7f28d313db80 1 --2- 172.21.15.2:0/3983982392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 0x5654c59bdfb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f28a80af040 tx=0x7f28a80a2080 comp rx=0 tx=0).stop 2023-11-16T16:49:37.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.376+0000 7f28d313db80 1 -- 172.21.15.2:0/3983982392 shutdown_connections 2023-11-16T16:49:37.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.376+0000 7f28d313db80 1 --2- 172.21.15.2:0/3983982392 >> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] conn(0x5654c59c8900 0x5654c59c7f20 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.376+0000 7f28d313db80 1 --2- 172.21.15.2:0/3983982392 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b54c0 0x5654c59c4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.376+0000 7f28d313db80 1 --2- 172.21.15.2:0/3983982392 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b4000 0x5654c59bda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.376+0000 7f28a0ff9700 0 client.15300 ms_handle_reset on v2:172.21.15.141:6836/2934891475 2023-11-16T16:49:37.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.376+0000 7f28d313db80 1 --2- 172.21.15.2:0/3983982392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 0x5654c59bdfb0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.376+0000 7f28d313db80 1 -- 172.21.15.2:0/3983982392 >> 172.21.15.2:0/3983982392 conn(0x5654c59a5310 msgr2=0x5654c59a9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.376+0000 7f28d313db80 1 -- 172.21.15.2:0/3983982392 shutdown_connections 2023-11-16T16:49:37.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.376+0000 7f28d313db80 1 -- 172.21.15.2:0/3983982392 wait complete. 2023-11-16T16:49:37.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.377+0000 7f28d313db80 20 client.15300 trim_cache size 0 max 16384 2023-11-16T16:49:37.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.395+0000 7f28d313db80 1 Processor -- start 2023-11-16T16:49:37.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.395+0000 7f28d313db80 1 -- start start 2023-11-16T16:49:37.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.396+0000 7f28d313db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4000 0x5654c59c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.396+0000 7f28d313db80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b4950 0x5654c59c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.396+0000 7f28d313db80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b54c0 0x5654c59bc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.396+0000 7f28d313db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c59b5db0 con 0x5654c59b4000 2023-11-16T16:49:37.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.396+0000 7f28d313db80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c58aa730 con 0x5654c59b54c0 2023-11-16T16:49:37.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.396+0000 7f28d313db80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c589ce10 con 0x5654c59b4950 2023-11-16T16:49:37.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.396+0000 7f28bd59b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4000 0x5654c59c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.396+0000 7f28bd59b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4000 0x5654c59c7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43274/0 (socket says 172.21.15.2:43274) 2023-11-16T16:49:37.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.396+0000 7f28bd59b700 1 -- 172.21.15.2:0/3419694486 learned_addr learned my addr 172.21.15.2:0/3419694486 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:37.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.396+0000 7f28bdd9c700 1 --2- 172.21.15.2:0/3419694486 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b54c0 0x5654c59bc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.396+0000 7f28bcd9a700 1 --2- 172.21.15.2:0/3419694486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b4950 0x5654c59c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.396+0000 7f28bd59b700 1 -- 172.21.15.2:0/3419694486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b4950 msgr2=0x5654c59c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.396+0000 7f28bd59b700 1 --2- 172.21.15.2:0/3419694486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b4950 0x5654c59c4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.396+0000 7f28bd59b700 1 -- 172.21.15.2:0/3419694486 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b54c0 msgr2=0x5654c59bc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.396+0000 7f28bd59b700 1 --2- 172.21.15.2:0/3419694486 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b54c0 0x5654c59bc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.396+0000 7f28bd59b700 1 -- 172.21.15.2:0/3419694486 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654c58c6e90 con 0x5654c59b4000 2023-11-16T16:49:37.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.396+0000 7f28bdd9c700 1 --2- 172.21.15.2:0/3419694486 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b54c0 0x5654c59bc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:49:37.399 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.397+0000 7f28bd59b700 1 --2- 172.21.15.2:0/3419694486 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4000 0x5654c59c7f20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x5654c59c9890 tx=0x7f28b4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=779384f80db11a96 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.397+0000 7f28a3fff700 1 -- 172.21.15.2:0/3419694486 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b4017040 con 0x5654c59b4000 2023-11-16T16:49:37.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.397+0000 7f28a3fff700 1 -- 172.21.15.2:0/3419694486 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28b4018040 con 0x5654c59b4000 2023-11-16T16:49:37.400 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.397+0000 7f28a3fff700 1 -- 172.21.15.2:0/3419694486 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b4004cf0 con 0x5654c59b4000 2023-11-16T16:49:37.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.397+0000 7f28d313db80 1 -- 172.21.15.2:0/3419694486 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4000 msgr2=0x5654c59c7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.397+0000 7f28d313db80 1 --2- 172.21.15.2:0/3419694486 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4000 0x5654c59c7f20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x5654c59c9890 tx=0x7f28b4009260 comp rx=0 tx=0).stop 2023-11-16T16:49:37.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.397+0000 7f28d313db80 1 -- 172.21.15.2:0/3419694486 shutdown_connections 2023-11-16T16:49:37.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.397+0000 7f28d313db80 1 --2- 172.21.15.2:0/3419694486 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b54c0 0x5654c59bc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.397+0000 7f28d313db80 1 --2- 172.21.15.2:0/3419694486 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4000 0x5654c59c7f20 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.397+0000 7f28d313db80 1 --2- 172.21.15.2:0/3419694486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b4950 0x5654c59c4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.397+0000 7f28d313db80 1 -- 172.21.15.2:0/3419694486 >> 172.21.15.2:0/3419694486 conn(0x5654c59a5310 msgr2=0x5654c59d1690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.398+0000 7f28d313db80 1 -- 172.21.15.2:0/3419694486 shutdown_connections 2023-11-16T16:49:37.403 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.398+0000 7f28d313db80 1 -- 172.21.15.2:0/3419694486 wait complete. 2023-11-16T16:49:37.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.398+0000 7f28d313db80 1 -- start start 2023-11-16T16:49:37.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.398+0000 7f28d313db80 1 Processor -- start 2023-11-16T16:49:37.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.398+0000 7f28d313db80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b4000 0x5654c59c1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.398+0000 7f28d313db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 0x5654c59c2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.399+0000 7f28d313db80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b54c0 0x5654c5a1c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.399+0000 7f28d313db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c59c5070 con 0x5654c59b4950 2023-11-16T16:49:37.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.399+0000 7f28d313db80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c59c51e0 con 0x5654c59b54c0 2023-11-16T16:49:37.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.399+0000 7f28d313db80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654c59c53a0 con 0x5654c59b4000 2023-11-16T16:49:37.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.399+0000 7f28d313db80 10 client.0 _ll_register_callbacks cb 0x7f28a80997f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:49:37.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.399+0000 7f28bcd9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 0x5654c59c2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.399+0000 7f28bcd9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 0x5654c59c2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43282/0 (socket says 172.21.15.2:43282) 2023-11-16T16:49:37.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.399+0000 7f28bcd9a700 1 -- 172.21.15.2:0/3464688468 learned_addr learned my addr 172.21.15.2:0/3464688468 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:37.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.399+0000 7f28bd59b700 1 --2- 172.21.15.2:0/3464688468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b4000 0x5654c59c1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.404 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.399+0000 7f28bdd9c700 1 --2- 172.21.15.2:0/3464688468 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b54c0 0x5654c5a1c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.399+0000 7f28bcd9a700 1 -- 172.21.15.2:0/3464688468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b4000 msgr2=0x5654c59c1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.399+0000 7f28bcd9a700 1 --2- 172.21.15.2:0/3464688468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b4000 0x5654c59c1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.399+0000 7f28bcd9a700 1 -- 172.21.15.2:0/3464688468 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b54c0 msgr2=0x5654c5a1c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.399+0000 7f28bcd9a700 1 --2- 172.21.15.2:0/3464688468 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b54c0 0x5654c5a1c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.399+0000 7f28bcd9a700 1 -- 172.21.15.2:0/3464688468 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b4012170 con 0x5654c59b4950 2023-11-16T16:49:37.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.399+0000 7f28bdd9c700 1 --2- 172.21.15.2:0/3464688468 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b54c0 0x5654c5a1c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:49:37.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.399+0000 7f28bcd9a700 1 --2- 172.21.15.2:0/3464688468 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 0x5654c59c2050 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f28a8096390 tx=0x7f28a8098440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be50b56c502dc745 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.399+0000 7f28a0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:49:37.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.400+0000 7f28a0ff9700 1 -- 172.21.15.2:0/3464688468 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a8098e90 con 0x5654c59b4950 2023-11-16T16:49:37.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.400+0000 7f28a0ff9700 1 -- 172.21.15.2:0/3464688468 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f28a8010940 con 0x5654c59b4950 2023-11-16T16:49:37.405 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.400+0000 7f28d313db80 1 -- 172.21.15.2:0/3464688468 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f28a80ca690 con 0x5654c59b4950 2023-11-16T16:49:37.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.400+0000 7f28a0ff9700 1 -- 172.21.15.2:0/3464688468 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28a80a22e0 con 0x5654c59b4950 2023-11-16T16:49:37.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.400+0000 7f28d313db80 10 client.15306 fetch_fsmap learned FSMap version 26 2023-11-16T16:49:37.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.400+0000 7f28d313db80 1 -- 172.21.15.2:0/3464688468 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5654c5717c10 con 0x5654c59b4950 2023-11-16T16:49:37.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.400+0000 7f28a0ff9700 1 -- 172.21.15.2:0/3464688468 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28a80ca690 con 0x5654c59b4950 2023-11-16T16:49:37.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.400+0000 7f28a0ff9700 1 -- 172.21.15.2:0/3464688468 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f28a8083140 con 0x5654c59b4950 2023-11-16T16:49:37.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.400+0000 7f28d313db80 10 client.15306 fetch_fsmap finished waiting for FSMap version 26 2023-11-16T16:49:37.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.400+0000 7f28d313db80 10 client.15306 resolve_mds: resolved name 'h' to daemon mds.h 2023-11-16T16:49:37.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.400+0000 7f28d313db80 20 client.15306 populate_metadata read hostname 'smithi002' 2023-11-16T16:49:37.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.400+0000 7f28d313db80 1 --2- 172.21.15.2:0/3464688468 >> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] conn(0x5654c5a21660 0x5654c5a1df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.400+0000 7f28d313db80 4 client.15306 mds_command: new command op to 24542 tid=0[{"prefix": "status"}] 2023-11-16T16:49:37.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.400+0000 7f28d313db80 1 -- 172.21.15.2:0/3464688468 --> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5654c58cf610 con 0x5654c5a21660 2023-11-16T16:49:37.406 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.401+0000 7f28bd59b700 1 --2- 172.21.15.2:0/3464688468 >> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] conn(0x5654c5a21660 0x5654c5a1df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.401+0000 7f28bd59b700 1 --2- 172.21.15.2:0/3464688468 >> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] conn(0x5654c5a21660 0x5654c5a1df50 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e88ae1104953ff4a server_cookie=cd6420711b8e96d5 in_seq=0 out_seq=0 2023-11-16T16:49:37.407 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.401+0000 7f28a0ff9700 10 client.15306 ms_handle_connect on v2:172.21.15.141:6836/2934891475 2023-11-16T16:49:37.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.402+0000 7f28a0ff9700 1 -- 172.21.15.2:0/3464688468 <== mds.2 v2:172.21.15.141:6836/2934891475 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x5654c58cf610 con 0x5654c5a21660 2023-11-16T16:49:37.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.402+0000 7f28a0ff9700 10 client.15306 handle_command_reply: tid=0 2023-11-16T16:49:37.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.402+0000 7f28d313db80 1 client.15306 shutdown 2023-11-16T16:49:37.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.402+0000 7f28d313db80 1 -- 172.21.15.2:0/3464688468 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 msgr2=0x5654c59c2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.402+0000 7f28d313db80 1 --2- 172.21.15.2:0/3464688468 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 0x5654c59c2050 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f28a8096390 tx=0x7f28a8098440 comp rx=0 tx=0).stop 2023-11-16T16:49:37.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.402+0000 7f28d313db80 1 -- 172.21.15.2:0/3464688468 shutdown_connections 2023-11-16T16:49:37.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.402+0000 7f28d313db80 1 --2- 172.21.15.2:0/3464688468 >> [v2:172.21.15.141:6836/2934891475,v1:172.21.15.141:6837/2934891475] conn(0x5654c5a21660 0x5654c5a1df50 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.402+0000 7f28d313db80 1 --2- 172.21.15.2:0/3464688468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654c59b4000 0x5654c59c1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.402+0000 7f28d313db80 1 --2- 172.21.15.2:0/3464688468 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654c59b54c0 0x5654c5a1c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.402+0000 7f28d313db80 1 --2- 172.21.15.2:0/3464688468 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654c59b4950 0x5654c59c2050 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.408 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.402+0000 7f28a0ff9700 0 client.15306 ms_handle_reset on v2:172.21.15.141:6836/2934891475 2023-11-16T16:49:37.409 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.402+0000 7f28d313db80 1 -- 172.21.15.2:0/3464688468 >> 172.21.15.2:0/3464688468 conn(0x5654c59a5310 msgr2=0x5654c59c7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.409 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.402+0000 7f28d313db80 1 -- 172.21.15.2:0/3464688468 shutdown_connections 2023-11-16T16:49:37.409 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.402+0000 7f28d313db80 1 -- 172.21.15.2:0/3464688468 wait complete. 2023-11-16T16:49:37.409 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.402+0000 7f28d313db80 20 client.15306 trim_cache size 0 max 16384 2023-11-16T16:49:37.409 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T16:49:37.409 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "875fcd24-849e-11ee-8dbe-212e2dc638e7", 2023-11-16T16:49:37.409 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 2, 2023-11-16T16:49:37.409 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24542, 2023-11-16T16:49:37.409 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2023-11-16T16:49:37.409 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2023-11-16T16:49:37.410 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2023-11-16T16:49:37.410 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 105.505782043, 2023-11-16T16:49:37.410 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 26, 2023-11-16T16:49:37.410 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 93, 2023-11-16T16:49:37.410 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 91, 2023-11-16T16:49:37.410 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 147.966464011 2023-11-16T16:49:37.410 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T16:49:37.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.403+0000 7f28bf7ff700 1 -- 172.21.15.2:0/4076601376 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f28a40a11d0 msgr2=0x7f28a40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.403+0000 7f28bf7ff700 1 --2- 172.21.15.2:0/4076601376 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f28a40a11d0 0x7f28a40a3680 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f28b000b780 tx=0x7f28b0007d70 comp rx=0 tx=0).stop 2023-11-16T16:49:37.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.403+0000 7f28bf7ff700 1 -- 172.21.15.2:0/4076601376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28b8101420 msgr2=0x7f28b81965e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.403+0000 7f28bf7ff700 1 --2- 172.21.15.2:0/4076601376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28b8101420 0x7f28b81965e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f28a800e9a0 tx=0x7f28a800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.403+0000 7f28bf7ff700 1 -- 172.21.15.2:0/4076601376 shutdown_connections 2023-11-16T16:49:37.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.403+0000 7f28bf7ff700 1 --2- 172.21.15.2:0/4076601376 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f28a40a11d0 0x7f28a40a3680 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.403+0000 7f28bf7ff700 1 --2- 172.21.15.2:0/4076601376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28b8100230 0x7f28b81960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.403+0000 7f28bf7ff700 1 --2- 172.21.15.2:0/4076601376 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f28b8101db0 0x7f28b819d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.403+0000 7f28bf7ff700 1 --2- 172.21.15.2:0/4076601376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f28b8101420 0x7f28b81965e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.403+0000 7f28bf7ff700 1 -- 172.21.15.2:0/4076601376 >> 172.21.15.2:0/4076601376 conn(0x7f28b80fb810 msgr2=0x7f28b80fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.411 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.403+0000 7f28bf7ff700 1 -- 172.21.15.2:0/4076601376 shutdown_connections 2023-11-16T16:49:37.412 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.403+0000 7f28bf7ff700 1 -- 172.21.15.2:0/4076601376 wait complete. 2023-11-16T16:49:37.423 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-11-16T16:49:37.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.628+0000 7fd03d3ff700 1 -- 172.21.15.2:0/3184117411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd038074ab0 msgr2=0x7fd038072f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.628+0000 7fd03d3ff700 1 --2- 172.21.15.2:0/3184117411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd038074ab0 0x7fd038072f00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd028009990 tx=0x7fd028009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.628+0000 7fd03d3ff700 1 -- 172.21.15.2:0/3184117411 shutdown_connections 2023-11-16T16:49:37.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.628+0000 7fd03d3ff700 1 --2- 172.21.15.2:0/3184117411 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd038108120 0x7fd03810a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.628+0000 7fd03d3ff700 1 --2- 172.21.15.2:0/3184117411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd038074ab0 0x7fd038072f00 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.628+0000 7fd03d3ff700 1 --2- 172.21.15.2:0/3184117411 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0380734d0 0x7fd038073940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.628+0000 7fd03d3ff700 1 -- 172.21.15.2:0/3184117411 >> 172.21.15.2:0/3184117411 conn(0x7fd0380fb8b0 msgr2=0x7fd0380fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.628+0000 7fd03d3ff700 1 -- 172.21.15.2:0/3184117411 shutdown_connections 2023-11-16T16:49:37.630 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.628+0000 7fd03d3ff700 1 -- 172.21.15.2:0/3184117411 wait complete. 2023-11-16T16:49:37.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.629+0000 7fd03d3ff700 1 Processor -- start 2023-11-16T16:49:37.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.629+0000 7fd03d3ff700 1 -- start start 2023-11-16T16:49:37.632 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.629+0000 7fd03d3ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0380734d0 0x7fd0381981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.629+0000 7fd03d3ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd038074ab0 0x7fd038198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.629+0000 7fd03d3ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd038108120 0x7fd03819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.630+0000 7fd03d3ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03819fd50 con 0x7fd0380734d0 2023-11-16T16:49:37.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.630+0000 7fd03d3ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03819fe90 con 0x7fd038108120 2023-11-16T16:49:37.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.630+0000 7fd03d3ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0381a0020 con 0x7fd038074ab0 2023-11-16T16:49:37.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.630+0000 7fd036ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0380734d0 0x7fd0381981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.630+0000 7fd036ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0380734d0 0x7fd0381981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43312/0 (socket says 172.21.15.2:43312) 2023-11-16T16:49:37.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.630+0000 7fd036ffd700 1 -- 172.21.15.2:0/4152828567 learned_addr learned my addr 172.21.15.2:0/4152828567 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:37.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.630+0000 7fd0377fe700 1 --2- 172.21.15.2:0/4152828567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd038108120 0x7fd03819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.630+0000 7fd0367fc700 1 --2- 172.21.15.2:0/4152828567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd038074ab0 0x7fd038198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.630+0000 7fd036ffd700 1 -- 172.21.15.2:0/4152828567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd038074ab0 msgr2=0x7fd038198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.630+0000 7fd036ffd700 1 --2- 172.21.15.2:0/4152828567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd038074ab0 0x7fd038198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.633 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.630+0000 7fd036ffd700 1 -- 172.21.15.2:0/4152828567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd038108120 msgr2=0x7fd03819f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.630+0000 7fd036ffd700 1 --2- 172.21.15.2:0/4152828567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd038108120 0x7fd03819f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.630+0000 7fd036ffd700 1 -- 172.21.15.2:0/4152828567 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0280095f0 con 0x7fd0380734d0 2023-11-16T16:49:37.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.630+0000 7fd036ffd700 1 --2- 172.21.15.2:0/4152828567 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0380734d0 0x7fd0381981d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd028005910 tx=0x7fd02800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.631+0000 7fd01ffff700 1 -- 172.21.15.2:0/4152828567 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02801d070 con 0x7fd0380734d0 2023-11-16T16:49:37.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.631+0000 7fd01ffff700 1 -- 172.21.15.2:0/4152828567 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd02800fb80 con 0x7fd0380734d0 2023-11-16T16:49:37.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.631+0000 7fd03d3ff700 1 -- 172.21.15.2:0/4152828567 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03810de10 con 0x7fd0380734d0 2023-11-16T16:49:37.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.631+0000 7fd01ffff700 1 -- 172.21.15.2:0/4152828567 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd028017650 con 0x7fd0380734d0 2023-11-16T16:49:37.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.631+0000 7fd03d3ff700 1 -- 172.21.15.2:0/4152828567 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd03810dc80 con 0x7fd0380734d0 2023-11-16T16:49:37.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.634+0000 7fd01ffff700 1 -- 172.21.15.2:0/4152828567 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7fd02800fcf0 con 0x7fd0380734d0 2023-11-16T16:49:37.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.635+0000 7fd050dc0b80 1 Processor -- start 2023-11-16T16:49:37.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.635+0000 7fd01ffff700 1 --2- 172.21.15.2:0/4152828567 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd0200a1150 0x7fd0200a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.635+0000 7fd050dc0b80 1 -- start start 2023-11-16T16:49:37.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.635+0000 7fd01ffff700 1 -- 172.21.15.2:0/4152828567 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fd0280ce250 con 0x7fd0380734d0 2023-11-16T16:49:37.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.636+0000 7fd050dc0b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df328bb0 0x5568df328fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.636+0000 7fd050dc0b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df329500 0x5568df329ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.636+0000 7fd050dc0b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df32a0c0 0x5568df32a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.636+0000 7fd050dc0b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568df22e010 con 0x5568df329500 2023-11-16T16:49:37.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.636+0000 7fd050dc0b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568df280850 con 0x5568df328bb0 2023-11-16T16:49:37.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.636+0000 7fd050dc0b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568df279ce0 con 0x5568df32a0c0 2023-11-16T16:49:37.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.636+0000 7fd0377fe700 1 --2- 172.21.15.2:0/4152828567 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd0200a1150 0x7fd0200a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.636+0000 7fd0367fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df329500 0x5568df329ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.636+0000 7fd0367fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df329500 0x5568df329ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43314/0 (socket says 172.21.15.2:43314) 2023-11-16T16:49:37.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.636+0000 7fd0367fc700 1 -- 172.21.15.2:0/3701062339 learned_addr learned my addr 172.21.15.2:0/3701062339 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:37.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.636+0000 7fd036ffd700 1 --2- 172.21.15.2:0/3701062339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df328bb0 0x5568df328fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.636+0000 7fd0377fe700 1 --2- 172.21.15.2:0/3701062339 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df32a0c0 0x5568df32a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.636+0000 7fd0367fc700 1 -- 172.21.15.2:0/3701062339 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df32a0c0 msgr2=0x5568df32a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.636+0000 7fd0367fc700 1 --2- 172.21.15.2:0/3701062339 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df32a0c0 0x5568df32a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.636+0000 7fd0367fc700 1 -- 172.21.15.2:0/3701062339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df328bb0 msgr2=0x5568df328fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.636+0000 7fd0367fc700 1 --2- 172.21.15.2:0/3701062339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df328bb0 0x5568df328fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.636+0000 7fd0367fc700 1 -- 172.21.15.2:0/3701062339 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568df0fb5e0 con 0x5568df329500 2023-11-16T16:49:37.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.636+0000 7fd0377fe700 1 --2- 172.21.15.2:0/3701062339 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df32a0c0 0x5568df32a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:49:37.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.637+0000 7fd0367fc700 1 --2- 172.21.15.2:0/3701062339 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df329500 0x5568df329ae0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd02c00bd70 tx=0x7fd02c00f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3a028b241229513 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.637+0000 7fd0377fe700 1 --2- 172.21.15.2:0/4152828567 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd0200a1150 0x7fd0200a3600 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd030009a50 tx=0x7fd030009e10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.637+0000 7fd01d7fa700 1 -- 172.21.15.2:0/3701062339 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c00fda0 con 0x5568df329500 2023-11-16T16:49:37.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.637+0000 7fd01d7fa700 1 -- 172.21.15.2:0/3701062339 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd02c01a920 con 0x5568df329500 2023-11-16T16:49:37.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.637+0000 7fd01d7fa700 1 -- 172.21.15.2:0/3701062339 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c011780 con 0x5568df329500 2023-11-16T16:49:37.643 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.638+0000 7fd050dc0b80 1 -- 172.21.15.2:0/3701062339 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df329500 msgr2=0x5568df329ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.643 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.638+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/3701062339 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df329500 0x5568df329ae0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd02c00bd70 tx=0x7fd02c00f640 comp rx=0 tx=0).stop 2023-11-16T16:49:37.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.638+0000 7fd050dc0b80 1 -- 172.21.15.2:0/3701062339 shutdown_connections 2023-11-16T16:49:37.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.638+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/3701062339 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df32a0c0 0x5568df32a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.638+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/3701062339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df328bb0 0x5568df328fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.638+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/3701062339 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df329500 0x5568df329ae0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.638+0000 7fd050dc0b80 1 -- 172.21.15.2:0/3701062339 >> 172.21.15.2:0/3701062339 conn(0x5568df31e310 msgr2=0x5568df323620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.638+0000 7fd050dc0b80 1 -- 172.21.15.2:0/3701062339 shutdown_connections 2023-11-16T16:49:37.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.638+0000 7fd050dc0b80 1 -- 172.21.15.2:0/3701062339 wait complete. 2023-11-16T16:49:37.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.638+0000 7fd050dc0b80 1 -- start start 2023-11-16T16:49:37.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.639+0000 7fd050dc0b80 1 Processor -- start 2023-11-16T16:49:37.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.639+0000 7fd050dc0b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 0x5568df332570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.639+0000 7fd050dc0b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df329500 0x5568df332ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.639+0000 7fd050dc0b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df32a0c0 0x5568df339af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.639+0000 7fd050dc0b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568df21b890 con 0x5568df328bb0 2023-11-16T16:49:37.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.639+0000 7fd050dc0b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568df21b530 con 0x5568df32a0c0 2023-11-16T16:49:37.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.639+0000 7fd050dc0b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568df0fd8d0 con 0x5568df329500 2023-11-16T16:49:37.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.639+0000 7fd050dc0b80 10 client.0 _ll_register_callbacks cb 0x5568df239730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:49:37.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.639+0000 7fd036ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 0x5568df332570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.639+0000 7fd0367fc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df329500 0x5568df332ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.639+0000 7fd036ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 0x5568df332570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43322/0 (socket says 172.21.15.2:43322) 2023-11-16T16:49:37.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.639+0000 7fd036ffd700 1 -- 172.21.15.2:0/1744486335 learned_addr learned my addr 172.21.15.2:0/1744486335 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:37.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.639+0000 7fd0377fe700 1 --2- 172.21.15.2:0/1744486335 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df32a0c0 0x5568df339af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.640+0000 7fd036ffd700 1 -- 172.21.15.2:0/1744486335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df329500 msgr2=0x5568df332ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.640+0000 7fd036ffd700 1 --2- 172.21.15.2:0/1744486335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df329500 0x5568df332ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.640+0000 7fd036ffd700 1 -- 172.21.15.2:0/1744486335 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df32a0c0 msgr2=0x5568df339af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.640+0000 7fd036ffd700 1 --2- 172.21.15.2:0/1744486335 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df32a0c0 0x5568df339af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.640+0000 7fd036ffd700 1 -- 172.21.15.2:0/1744486335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd02c00b9a0 con 0x5568df328bb0 2023-11-16T16:49:37.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.640+0000 7fd0377fe700 1 --2- 172.21.15.2:0/1744486335 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df32a0c0 0x5568df339af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:49:37.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.640+0000 7fd036ffd700 1 --2- 172.21.15.2:0/1744486335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 0x5568df332570 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd02809b2c0 tx=0x7fd0280a1330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc9942391133b1d5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.640+0000 7fd01aff5700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:49:37.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.640+0000 7fd01aff5700 1 -- 172.21.15.2:0/1744486335 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02809aa20 con 0x5568df328bb0 2023-11-16T16:49:37.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.640+0000 7fd01aff5700 1 -- 172.21.15.2:0/1744486335 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fd02809f360 con 0x5568df328bb0 2023-11-16T16:49:37.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.640+0000 7fd050dc0b80 1 -- 172.21.15.2:0/1744486335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5568df31bad0 con 0x5568df328bb0 2023-11-16T16:49:37.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.640+0000 7fd01aff5700 1 -- 172.21.15.2:0/1744486335 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0280a36f0 con 0x5568df328bb0 2023-11-16T16:49:37.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.640+0000 7fd050dc0b80 10 client.15318 fetch_fsmap learned FSMap version 26 2023-11-16T16:49:37.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.640+0000 7fd050dc0b80 1 -- 172.21.15.2:0/1744486335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5568df240140 con 0x5568df328bb0 2023-11-16T16:49:37.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.641+0000 7fd01aff5700 1 -- 172.21.15.2:0/1744486335 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5568df31bad0 con 0x5568df328bb0 2023-11-16T16:49:37.646 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.641+0000 7fd01aff5700 1 -- 172.21.15.2:0/1744486335 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fd02809a320 con 0x5568df328bb0 2023-11-16T16:49:37.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.641+0000 7fd050dc0b80 10 client.15318 fetch_fsmap finished waiting for FSMap version 26 2023-11-16T16:49:37.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.641+0000 7fd050dc0b80 10 client.15318 resolve_mds: resolved name 'i' to daemon mds.i 2023-11-16T16:49:37.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.641+0000 7fd050dc0b80 20 client.15318 populate_metadata read hostname 'smithi002' 2023-11-16T16:49:37.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.641+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/1744486335 >> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] conn(0x5568df33d390 0x5568df33c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.641+0000 7fd050dc0b80 4 client.15318 mds_command: new command op to 24544 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:49:37.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.641+0000 7fd050dc0b80 1 -- 172.21.15.2:0/1744486335 --> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5568df23c050 con 0x5568df33d390 2023-11-16T16:49:37.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.641+0000 7fd0367fc700 1 --2- 172.21.15.2:0/1744486335 >> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] conn(0x5568df33d390 0x5568df33c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.642+0000 7fd0367fc700 1 --2- 172.21.15.2:0/1744486335 >> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] conn(0x5568df33d390 0x5568df33c990 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6ff04bd35244e0b4 server_cookie=10fcc423416f2fd7 in_seq=0 out_seq=0 2023-11-16T16:49:37.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.642+0000 7fd01aff5700 10 client.15318 ms_handle_connect on v2:172.21.15.155:6838/1158457009 2023-11-16T16:49:37.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.643+0000 7fd01aff5700 1 -- 172.21.15.2:0/1744486335 <== mds.1 v2:172.21.15.155:6838/1158457009 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5568df23c050 con 0x5568df33d390 2023-11-16T16:49:37.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.643+0000 7fd01aff5700 10 client.15318 handle_command_reply: tid=0 2023-11-16T16:49:37.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.643+0000 7fd050dc0b80 1 client.15318 shutdown 2023-11-16T16:49:37.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.643+0000 7fd050dc0b80 1 -- 172.21.15.2:0/1744486335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 msgr2=0x5568df332570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.648 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.643+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/1744486335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 0x5568df332570 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd02809b2c0 tx=0x7fd0280a1330 comp rx=0 tx=0).stop 2023-11-16T16:49:37.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.643+0000 7fd050dc0b80 1 -- 172.21.15.2:0/1744486335 shutdown_connections 2023-11-16T16:49:37.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.643+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/1744486335 >> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] conn(0x5568df33d390 0x5568df33c990 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.643+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/1744486335 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df32a0c0 0x5568df339af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.643+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/1744486335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 0x5568df332570 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.643+0000 7fd01aff5700 0 client.15318 ms_handle_reset on v2:172.21.15.155:6838/1158457009 2023-11-16T16:49:37.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.643+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/1744486335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df329500 0x5568df332ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.643+0000 7fd050dc0b80 1 -- 172.21.15.2:0/1744486335 >> 172.21.15.2:0/1744486335 conn(0x5568df31e310 msgr2=0x5568df322d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.643+0000 7fd050dc0b80 1 -- 172.21.15.2:0/1744486335 shutdown_connections 2023-11-16T16:49:37.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.643+0000 7fd050dc0b80 1 -- 172.21.15.2:0/1744486335 wait complete. 2023-11-16T16:49:37.649 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.643+0000 7fd050dc0b80 20 client.15318 trim_cache size 0 max 16384 2023-11-16T16:49:37.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.662+0000 7fd050dc0b80 1 Processor -- start 2023-11-16T16:49:37.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.662+0000 7fd050dc0b80 1 -- start start 2023-11-16T16:49:37.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.662+0000 7fd050dc0b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 0x5568df33c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.662+0000 7fd050dc0b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df329500 0x5568df339a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.662+0000 7fd050dc0b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df32a0c0 0x5568df32cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.662+0000 7fd050dc0b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568df332140 con 0x5568df328bb0 2023-11-16T16:49:37.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.662+0000 7fd050dc0b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568df23c080 con 0x5568df329500 2023-11-16T16:49:37.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.662+0000 7fd050dc0b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568df333760 con 0x5568df32a0c0 2023-11-16T16:49:37.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.662+0000 7fd036ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 0x5568df33c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.662+0000 7fd036ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 0x5568df33c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43334/0 (socket says 172.21.15.2:43334) 2023-11-16T16:49:37.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.662+0000 7fd036ffd700 1 -- 172.21.15.2:0/4055367906 learned_addr learned my addr 172.21.15.2:0/4055367906 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:37.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.663+0000 7fd0377fe700 1 --2- 172.21.15.2:0/4055367906 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df32a0c0 0x5568df32cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.663+0000 7fd0367fc700 1 --2- 172.21.15.2:0/4055367906 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df329500 0x5568df339a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.663+0000 7fd036ffd700 1 -- 172.21.15.2:0/4055367906 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df32a0c0 msgr2=0x5568df32cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.663+0000 7fd036ffd700 1 --2- 172.21.15.2:0/4055367906 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df32a0c0 0x5568df32cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.663+0000 7fd036ffd700 1 -- 172.21.15.2:0/4055367906 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df329500 msgr2=0x5568df339a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.663+0000 7fd036ffd700 1 --2- 172.21.15.2:0/4055367906 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df329500 0x5568df339a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.663+0000 7fd036ffd700 1 -- 172.21.15.2:0/4055367906 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568df23fe90 con 0x5568df328bb0 2023-11-16T16:49:37.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.663+0000 7fd036ffd700 1 --2- 172.21.15.2:0/4055367906 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 0x5568df33c990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd028097830 tx=0x7fd0280998e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c6507d0e8349431 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.663+0000 7fd01dffb700 1 -- 172.21.15.2:0/4055367906 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02809c500 con 0x5568df328bb0 2023-11-16T16:49:37.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.663+0000 7fd01dffb700 1 -- 172.21.15.2:0/4055367906 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd02809b100 con 0x5568df328bb0 2023-11-16T16:49:37.667 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.663+0000 7fd01dffb700 1 -- 172.21.15.2:0/4055367906 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0280a3260 con 0x5568df328bb0 2023-11-16T16:49:37.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.664+0000 7fd050dc0b80 1 -- 172.21.15.2:0/4055367906 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 msgr2=0x5568df33c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.664+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/4055367906 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 0x5568df33c990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd028097830 tx=0x7fd0280998e0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.664+0000 7fd050dc0b80 1 -- 172.21.15.2:0/4055367906 shutdown_connections 2023-11-16T16:49:37.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.664+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/4055367906 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df32a0c0 0x5568df32cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.664+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/4055367906 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 0x5568df33c990 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.664+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/4055367906 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df329500 0x5568df339a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.664+0000 7fd050dc0b80 1 -- 172.21.15.2:0/4055367906 >> 172.21.15.2:0/4055367906 conn(0x5568df31e310 msgr2=0x5568df346160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.664+0000 7fd050dc0b80 1 -- 172.21.15.2:0/4055367906 shutdown_connections 2023-11-16T16:49:37.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.664+0000 7fd050dc0b80 1 -- 172.21.15.2:0/4055367906 wait complete. 2023-11-16T16:49:37.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.664+0000 7fd050dc0b80 1 -- start start 2023-11-16T16:49:37.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.664+0000 7fd050dc0b80 1 Processor -- start 2023-11-16T16:49:37.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.665+0000 7fd050dc0b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 0x5568df3343b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.665+0000 7fd050dc0b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df329500 0x5568df339920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.665+0000 7fd050dc0b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df32a0c0 0x5568df3348f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.665+0000 7fd050dc0b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568df334e70 con 0x5568df328bb0 2023-11-16T16:49:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.665+0000 7fd050dc0b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568df334fe0 con 0x5568df329500 2023-11-16T16:49:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.665+0000 7fd050dc0b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568df3912c0 con 0x5568df32a0c0 2023-11-16T16:49:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.665+0000 7fd050dc0b80 10 client.0 _ll_register_callbacks cb 0x7fd02809af20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:49:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.665+0000 7fd036ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 0x5568df3343b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.665+0000 7fd0367fc700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df329500 0x5568df339920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.665+0000 7fd0377fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df32a0c0 0x5568df3348f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.665+0000 7fd0367fc700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df329500 0x5568df339920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:41318/0 (socket says 172.21.15.2:41318) 2023-11-16T16:49:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.665+0000 7fd0367fc700 1 -- 172.21.15.2:0/2550533362 learned_addr learned my addr 172.21.15.2:0/2550533362 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:37.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.666+0000 7fd0367fc700 1 -- 172.21.15.2:0/2550533362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df32a0c0 msgr2=0x5568df3348f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.666+0000 7fd0367fc700 1 --2- 172.21.15.2:0/2550533362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df32a0c0 0x5568df3348f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.666+0000 7fd0367fc700 1 -- 172.21.15.2:0/2550533362 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 msgr2=0x5568df3343b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.666+0000 7fd0367fc700 1 --2- 172.21.15.2:0/2550533362 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 0x5568df3343b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.666+0000 7fd0367fc700 1 -- 172.21.15.2:0/2550533362 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5568df240140 con 0x5568df329500 2023-11-16T16:49:37.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.666+0000 7fd036ffd700 1 --2- 172.21.15.2:0/2550533362 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 0x5568df3343b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:37.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.666+0000 7fd0367fc700 1 --2- 172.21.15.2:0/2550533362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df329500 0x5568df339920 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x5568df324460 tx=0x7fd02c00bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98183ec85f27344 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.666+0000 7fd01aff5700 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2023-11-16T16:49:37.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.666+0000 7fd01aff5700 1 -- 172.21.15.2:0/2550533362 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c005ee0 con 0x5568df329500 2023-11-16T16:49:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.666+0000 7fd050dc0b80 1 -- 172.21.15.2:0/2550533362 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd02c01a920 con 0x5568df329500 2023-11-16T16:49:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.666+0000 7fd01aff5700 1 -- 172.21.15.2:0/2550533362 <== mon.2 v2:172.21.15.141:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fd02c01ada0 con 0x5568df329500 2023-11-16T16:49:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.667+0000 7fd01aff5700 1 -- 172.21.15.2:0/2550533362 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd02c00fe90 con 0x5568df329500 2023-11-16T16:49:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.667+0000 7fd050dc0b80 10 client.25355 fetch_fsmap learned FSMap version 26 2023-11-16T16:49:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.667+0000 7fd050dc0b80 1 -- 172.21.15.2:0/2550533362 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5568df090c10 con 0x5568df329500 2023-11-16T16:49:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.667+0000 7fd01aff5700 1 -- 172.21.15.2:0/2550533362 <== mon.2 v2:172.21.15.141:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd02c01a920 con 0x5568df329500 2023-11-16T16:49:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.667+0000 7fd01aff5700 1 -- 172.21.15.2:0/2550533362 <== mon.2 v2:172.21.15.141:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fd02c028030 con 0x5568df329500 2023-11-16T16:49:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.667+0000 7fd050dc0b80 10 client.25355 fetch_fsmap finished waiting for FSMap version 26 2023-11-16T16:49:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.667+0000 7fd050dc0b80 10 client.25355 resolve_mds: resolved name 'i' to daemon mds.i 2023-11-16T16:49:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.667+0000 7fd050dc0b80 20 client.25355 populate_metadata read hostname 'smithi002' 2023-11-16T16:49:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.667+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/2550533362 >> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] conn(0x5568df396110 0x5568df392a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.667+0000 7fd050dc0b80 4 client.25355 mds_command: new command op to 24544 tid=0[{"prefix": "status"}] 2023-11-16T16:49:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.667+0000 7fd050dc0b80 1 -- 172.21.15.2:0/2550533362 --> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5568df248610 con 0x5568df396110 2023-11-16T16:49:37.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.668+0000 7fd036ffd700 1 --2- 172.21.15.2:0/2550533362 >> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] conn(0x5568df396110 0x5568df392a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.668+0000 7fd036ffd700 1 --2- 172.21.15.2:0/2550533362 >> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] conn(0x5568df396110 0x5568df392a20 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=56b721a3c72804f4 server_cookie=77920c62cc900083 in_seq=0 out_seq=0 2023-11-16T16:49:37.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.668+0000 7fd01aff5700 10 client.25355 ms_handle_connect on v2:172.21.15.155:6838/1158457009 2023-11-16T16:49:37.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.668+0000 7fd01aff5700 1 -- 172.21.15.2:0/2550533362 <== mds.1 v2:172.21.15.155:6838/1158457009 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5568df248610 con 0x5568df396110 2023-11-16T16:49:37.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.668+0000 7fd01aff5700 10 client.25355 handle_command_reply: tid=0 2023-11-16T16:49:37.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.668+0000 7fd050dc0b80 1 client.25355 shutdown 2023-11-16T16:49:37.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.669+0000 7fd050dc0b80 1 -- 172.21.15.2:0/2550533362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df329500 msgr2=0x5568df339920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.669+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/2550533362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df329500 0x5568df339920 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x5568df324460 tx=0x7fd02c00bf90 comp rx=0 tx=0).stop 2023-11-16T16:49:37.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.669+0000 7fd050dc0b80 1 -- 172.21.15.2:0/2550533362 shutdown_connections 2023-11-16T16:49:37.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.669+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/2550533362 >> [v2:172.21.15.155:6838/1158457009,v1:172.21.15.155:6839/1158457009] conn(0x5568df396110 0x5568df392a20 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.669+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/2550533362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5568df32a0c0 0x5568df3348f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.669+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/2550533362 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5568df328bb0 0x5568df3343b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.674 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.669+0000 7fd050dc0b80 1 --2- 172.21.15.2:0/2550533362 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5568df329500 0x5568df339920 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.669+0000 7fd050dc0b80 1 -- 172.21.15.2:0/2550533362 >> 172.21.15.2:0/2550533362 conn(0x5568df31e310 msgr2=0x5568df33c990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.669+0000 7fd01aff5700 0 client.25355 ms_handle_reset on v2:172.21.15.155:6838/1158457009 2023-11-16T16:49:37.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.669+0000 7fd050dc0b80 1 -- 172.21.15.2:0/2550533362 shutdown_connections 2023-11-16T16:49:37.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.669+0000 7fd050dc0b80 1 -- 172.21.15.2:0/2550533362 wait complete. 2023-11-16T16:49:37.675 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.669+0000 7fd050dc0b80 20 client.25355 trim_cache size 0 max 16384 2023-11-16T16:49:37.675 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T16:49:37.675 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "875fcd24-849e-11ee-8dbe-212e2dc638e7", 2023-11-16T16:49:37.675 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 1, 2023-11-16T16:49:37.675 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24544, 2023-11-16T16:49:37.675 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2023-11-16T16:49:37.676 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2023-11-16T16:49:37.676 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2023-11-16T16:49:37.676 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 106.77950339500001, 2023-11-16T16:49:37.676 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 26, 2023-11-16T16:49:37.676 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 93, 2023-11-16T16:49:37.676 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 91, 2023-11-16T16:49:37.676 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 146.06526434700001 2023-11-16T16:49:37.676 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T16:49:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.669+0000 7fd03d3ff700 1 -- 172.21.15.2:0/4152828567 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd0200a1150 msgr2=0x7fd0200a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.669+0000 7fd03d3ff700 1 --2- 172.21.15.2:0/4152828567 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd0200a1150 0x7fd0200a3600 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd030009a50 tx=0x7fd030009e10 comp rx=0 tx=0).stop 2023-11-16T16:49:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.669+0000 7fd03d3ff700 1 -- 172.21.15.2:0/4152828567 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0380734d0 msgr2=0x7fd0381981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.669+0000 7fd03d3ff700 1 --2- 172.21.15.2:0/4152828567 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0380734d0 0x7fd0381981d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd028005910 tx=0x7fd02800f690 comp rx=0 tx=0).stop 2023-11-16T16:49:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.670+0000 7fd03d3ff700 1 -- 172.21.15.2:0/4152828567 shutdown_connections 2023-11-16T16:49:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.670+0000 7fd03d3ff700 1 --2- 172.21.15.2:0/4152828567 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fd0200a1150 0x7fd0200a3600 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.670+0000 7fd03d3ff700 1 --2- 172.21.15.2:0/4152828567 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd038108120 0x7fd03819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.670+0000 7fd03d3ff700 1 --2- 172.21.15.2:0/4152828567 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0380734d0 0x7fd0381981d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.670+0000 7fd03d3ff700 1 --2- 172.21.15.2:0/4152828567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd038074ab0 0x7fd038198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.670+0000 7fd03d3ff700 1 -- 172.21.15.2:0/4152828567 >> 172.21.15.2:0/4152828567 conn(0x7fd0380fb8b0 msgr2=0x7fd0380fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.670+0000 7fd03d3ff700 1 -- 172.21.15.2:0/4152828567 shutdown_connections 2023-11-16T16:49:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.670+0000 7fd03d3ff700 1 -- 172.21.15.2:0/4152828567 wait complete. 2023-11-16T16:49:37.689 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-11-16T16:49:37.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.895+0000 7ff4e5bff700 1 -- 172.21.15.2:0/592262039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4e0101420 msgr2=0x7ff4e0101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.895+0000 7ff4e5bff700 1 --2- 172.21.15.2:0/592262039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4e0101420 0x7ff4e0101870 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff4d40098f0 tx=0x7ff4d4009c00 comp rx=0 tx=0).stop 2023-11-16T16:49:37.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.895+0000 7ff4e5bff700 1 -- 172.21.15.2:0/592262039 shutdown_connections 2023-11-16T16:49:37.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.895+0000 7ff4e5bff700 1 --2- 172.21.15.2:0/592262039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4e0101db0 0x7ff4e0108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.895+0000 7ff4e5bff700 1 --2- 172.21.15.2:0/592262039 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4e0100230 0x7ff4e0100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.895+0000 7ff4e5bff700 1 --2- 172.21.15.2:0/592262039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4e0101420 0x7ff4e0101870 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.895+0000 7ff4e5bff700 1 -- 172.21.15.2:0/592262039 >> 172.21.15.2:0/592262039 conn(0x7ff4e00fb810 msgr2=0x7ff4e00fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.895+0000 7ff4e5bff700 1 -- 172.21.15.2:0/592262039 shutdown_connections 2023-11-16T16:49:37.897 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.895+0000 7ff4e5bff700 1 -- 172.21.15.2:0/592262039 wait complete. 2023-11-16T16:49:37.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.896+0000 7ff4e5bff700 1 Processor -- start 2023-11-16T16:49:37.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.897+0000 7ff4e5bff700 1 -- start start 2023-11-16T16:49:37.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.897+0000 7ff4e5bff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4e0100230 0x7ff4e0195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.897+0000 7ff4e5bff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4e0101420 0x7ff4e01964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.897+0000 7ff4e5bff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4e0101db0 0x7ff4e019d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.897+0000 7ff4e5bff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e019db10 con 0x7ff4e0101420 2023-11-16T16:49:37.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.897+0000 7ff4e5bff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e019dc50 con 0x7ff4e0101db0 2023-11-16T16:49:37.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.897+0000 7ff4e5bff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e019dde0 con 0x7ff4e0100230 2023-11-16T16:49:37.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.897+0000 7ff4df7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4e0100230 0x7ff4e0195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.897+0000 7ff4deffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4e0101420 0x7ff4e01964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.897+0000 7ff4df7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4e0100230 0x7ff4e0195f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:33356/0 (socket says 172.21.15.2:33356) 2023-11-16T16:49:37.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.897+0000 7ff4df7fe700 1 -- 172.21.15.2:0/625501199 learned_addr learned my addr 172.21.15.2:0/625501199 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:37.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.897+0000 7ff4dffff700 1 --2- 172.21.15.2:0/625501199 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4e0101db0 0x7ff4e019d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.898+0000 7ff4df7fe700 1 -- 172.21.15.2:0/625501199 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4e0101db0 msgr2=0x7ff4e019d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.898+0000 7ff4df7fe700 1 --2- 172.21.15.2:0/625501199 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4e0101db0 0x7ff4e019d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.898+0000 7ff4df7fe700 1 -- 172.21.15.2:0/625501199 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4e0101420 msgr2=0x7ff4e01964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.898+0000 7ff4df7fe700 1 --2- 172.21.15.2:0/625501199 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4e0101420 0x7ff4e01964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.898+0000 7ff4df7fe700 1 -- 172.21.15.2:0/625501199 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d4009520 con 0x7ff4e0100230 2023-11-16T16:49:37.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.898+0000 7ff4df7fe700 1 --2- 172.21.15.2:0/625501199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4e0100230 0x7ff4e0195f90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000e9f0 tx=0x7ff4d000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.898+0000 7ff4dcff9700 1 -- 172.21.15.2:0/625501199 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0004220 con 0x7ff4e0100230 2023-11-16T16:49:37.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.898+0000 7ff4dcff9700 1 -- 172.21.15.2:0/625501199 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4d0004380 con 0x7ff4e0100230 2023-11-16T16:49:37.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.898+0000 7ff4e5bff700 1 -- 172.21.15.2:0/625501199 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e010bb80 con 0x7ff4e0100230 2023-11-16T16:49:37.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.898+0000 7ff4dcff9700 1 -- 172.21.15.2:0/625501199 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0003be0 con 0x7ff4e0100230 2023-11-16T16:49:37.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.899+0000 7ff4e5bff700 1 -- 172.21.15.2:0/625501199 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e010b9f0 con 0x7ff4e0100230 2023-11-16T16:49:37.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.902+0000 7ff4dcff9700 1 -- 172.21.15.2:0/625501199 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7ff4d0003d40 con 0x7ff4e0100230 2023-11-16T16:49:37.906 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.903+0000 7ff4f957eb80 1 Processor -- start 2023-11-16T16:49:37.906 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.903+0000 7ff4dcff9700 1 --2- 172.21.15.2:0/625501199 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff4c80a11d0 0x7ff4c80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.906 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.903+0000 7ff4f957eb80 1 -- start start 2023-11-16T16:49:37.906 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.903+0000 7ff4dcff9700 1 -- 172.21.15.2:0/625501199 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7ff4d0014070 con 0x7ff4e0100230 2023-11-16T16:49:37.906 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.903+0000 7ff4dffff700 1 --2- 172.21.15.2:0/625501199 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff4c80a11d0 0x7ff4c80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.906 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.903+0000 7ff4f957eb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0553bb0 0x55c9f0553fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.906 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.903+0000 7ff4f957eb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f0554500 0x55c9f0554ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.906 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.904+0000 7ff4f957eb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f05550c0 0x55c9f0555530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.904+0000 7ff4f957eb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9f0459010 con 0x55c9f0553bb0 2023-11-16T16:49:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.904+0000 7ff4f957eb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9f04ab850 con 0x55c9f0554500 2023-11-16T16:49:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.904+0000 7ff4f957eb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9f04a4ce0 con 0x55c9f05550c0 2023-11-16T16:49:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.904+0000 7ff4df7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0553bb0 0x55c9f0553fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.904+0000 7ff4df7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0553bb0 0x55c9f0553fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43356/0 (socket says 172.21.15.2:43356) 2023-11-16T16:49:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.904+0000 7ff4df7fe700 1 -- 172.21.15.2:0/3650973107 learned_addr learned my addr 172.21.15.2:0/3650973107 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.904+0000 7ff4deffd700 1 --2- 172.21.15.2:0/3650973107 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f0554500 0x55c9f0554ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.904+0000 7ff4dffff700 1 --2- 172.21.15.2:0/3650973107 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f05550c0 0x55c9f0555530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.904+0000 7ff4df7fe700 1 -- 172.21.15.2:0/3650973107 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f05550c0 msgr2=0x55c9f0555530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.904+0000 7ff4df7fe700 1 --2- 172.21.15.2:0/3650973107 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f05550c0 0x55c9f0555530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.904+0000 7ff4df7fe700 1 -- 172.21.15.2:0/3650973107 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f0554500 msgr2=0x55c9f0554ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.904+0000 7ff4df7fe700 1 --2- 172.21.15.2:0/3650973107 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f0554500 0x55c9f0554ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.904+0000 7ff4df7fe700 1 -- 172.21.15.2:0/3650973107 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9f03265e0 con 0x55c9f0553bb0 2023-11-16T16:49:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.904+0000 7ff4dffff700 1 --2- 172.21.15.2:0/3650973107 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f05550c0 0x55c9f0555530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:49:37.908 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.904+0000 7ff4dffff700 1 --2- 172.21.15.2:0/625501199 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff4c80a11d0 0x7ff4c80a3680 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff4d800f590 tx=0x7ff4d800b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.908 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.904+0000 7ff4df7fe700 1 --2- 172.21.15.2:0/3650973107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0553bb0 0x55c9f0553fc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff4d00967d0 tx=0x7ff4d00999a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ece875656f5263a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.905+0000 7ff4c5ffb700 1 -- 172.21.15.2:0/3650973107 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0099d60 con 0x55c9f0553bb0 2023-11-16T16:49:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.905+0000 7ff4c5ffb700 1 -- 172.21.15.2:0/3650973107 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4d009dd90 con 0x55c9f0553bb0 2023-11-16T16:49:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.905+0000 7ff4c5ffb700 1 -- 172.21.15.2:0/3650973107 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0099250 con 0x55c9f0553bb0 2023-11-16T16:49:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.905+0000 7ff4f957eb80 1 -- 172.21.15.2:0/3650973107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0553bb0 msgr2=0x55c9f0553fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.905+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/3650973107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0553bb0 0x55c9f0553fc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff4d00967d0 tx=0x7ff4d00999a0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.906+0000 7ff4f957eb80 1 -- 172.21.15.2:0/3650973107 shutdown_connections 2023-11-16T16:49:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.906+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/3650973107 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f05550c0 0x55c9f0555530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.906+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/3650973107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0553bb0 0x55c9f0553fc0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.906+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/3650973107 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f0554500 0x55c9f0554ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.906+0000 7ff4f957eb80 1 -- 172.21.15.2:0/3650973107 >> 172.21.15.2:0/3650973107 conn(0x55c9f0549310 msgr2=0x55c9f054e620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.906+0000 7ff4f957eb80 1 -- 172.21.15.2:0/3650973107 shutdown_connections 2023-11-16T16:49:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.906+0000 7ff4f957eb80 1 -- 172.21.15.2:0/3650973107 wait complete. 2023-11-16T16:49:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.906+0000 7ff4f957eb80 1 -- start start 2023-11-16T16:49:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.906+0000 7ff4f957eb80 1 Processor -- start 2023-11-16T16:49:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.907+0000 7ff4f957eb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0553bb0 0x55c9f055d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.907+0000 7ff4f957eb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f0554500 0x55c9f055dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.907+0000 7ff4f957eb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f05550c0 0x55c9f0564af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.907+0000 7ff4f957eb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9f0446890 con 0x55c9f0553bb0 2023-11-16T16:49:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.907+0000 7ff4f957eb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9f0446530 con 0x55c9f0554500 2023-11-16T16:49:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.907+0000 7ff4f957eb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9f03288d0 con 0x55c9f05550c0 2023-11-16T16:49:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.907+0000 7ff4f957eb80 10 client.0 _ll_register_callbacks cb 0x55c9f0464730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:49:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.907+0000 7ff4dffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f05550c0 0x55c9f0564af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.907+0000 7ff4df7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0553bb0 0x55c9f055d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.907+0000 7ff4dffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f05550c0 0x55c9f0564af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:33366/0 (socket says 172.21.15.2:33366) 2023-11-16T16:49:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.907+0000 7ff4dffff700 1 -- 172.21.15.2:0/3234232117 learned_addr learned my addr 172.21.15.2:0/3234232117 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.907+0000 7ff4deffd700 1 --2- 172.21.15.2:0/3234232117 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f0554500 0x55c9f055dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.908+0000 7ff4dffff700 1 -- 172.21.15.2:0/3234232117 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f0554500 msgr2=0x55c9f055dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.908+0000 7ff4dffff700 1 --2- 172.21.15.2:0/3234232117 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f0554500 0x55c9f055dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.908+0000 7ff4dffff700 1 -- 172.21.15.2:0/3234232117 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0553bb0 msgr2=0x55c9f055d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.908+0000 7ff4dffff700 1 --2- 172.21.15.2:0/3234232117 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0553bb0 0x55c9f055d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.908+0000 7ff4dffff700 1 -- 172.21.15.2:0/3234232117 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e010b9f0 con 0x55c9f05550c0 2023-11-16T16:49:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.908+0000 7ff4dffff700 1 --2- 172.21.15.2:0/3234232117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f05550c0 0x55c9f0564af0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8019ef0 tx=0x7ff4d8004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f39d709c7c266ec3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.908+0000 7ff4beffd700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:49:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.908+0000 7ff4beffd700 1 -- 172.21.15.2:0/3234232117 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d800e040 con 0x55c9f05550c0 2023-11-16T16:49:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.908+0000 7ff4beffd700 1 -- 172.21.15.2:0/3234232117 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7ff4d8015bc0 con 0x55c9f05550c0 2023-11-16T16:49:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.908+0000 7ff4f957eb80 1 -- 172.21.15.2:0/3234232117 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c9f0546ad0 con 0x55c9f05550c0 2023-11-16T16:49:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.908+0000 7ff4beffd700 1 -- 172.21.15.2:0/3234232117 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4d8015d80 con 0x55c9f05550c0 2023-11-16T16:49:37.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.908+0000 7ff4f957eb80 10 client.25336 fetch_fsmap learned FSMap version 26 2023-11-16T16:49:37.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.908+0000 7ff4f957eb80 1 -- 172.21.15.2:0/3234232117 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55c9f046b140 con 0x55c9f05550c0 2023-11-16T16:49:37.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.909+0000 7ff4beffd700 1 -- 172.21.15.2:0/3234232117 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c9f0546ad0 con 0x55c9f05550c0 2023-11-16T16:49:37.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.909+0000 7ff4beffd700 1 -- 172.21.15.2:0/3234232117 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7ff4d802c440 con 0x55c9f05550c0 2023-11-16T16:49:37.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.909+0000 7ff4f957eb80 10 client.25336 fetch_fsmap finished waiting for FSMap version 26 2023-11-16T16:49:37.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.909+0000 7ff4f957eb80 10 client.25336 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T16:49:37.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.909+0000 7ff4f957eb80 20 client.25336 populate_metadata read hostname 'smithi002' 2023-11-16T16:49:37.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.909+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/3234232117 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55c9f0568810 0x55c9f0567bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.909+0000 7ff4f957eb80 4 client.25336 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:49:37.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.909+0000 7ff4f957eb80 1 -- 172.21.15.2:0/3234232117 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c9f0467050 con 0x55c9f0568810 2023-11-16T16:49:37.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.909+0000 7ff4df7fe700 1 --2- 172.21.15.2:0/3234232117 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55c9f0568810 0x55c9f0567bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.915 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.910+0000 7ff4df7fe700 1 --2- 172.21.15.2:0/3234232117 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55c9f0568810 0x55c9f0567bc0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c67c9fb925e1eb3 server_cookie=adc797bede5b59a8 in_seq=0 out_seq=0 2023-11-16T16:49:37.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.910+0000 7ff4beffd700 10 client.25336 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:49:37.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.911+0000 7ff4beffd700 1 -- 172.21.15.2:0/3234232117 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55c9f0467050 con 0x55c9f0568810 2023-11-16T16:49:37.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.911+0000 7ff4beffd700 10 client.25336 handle_command_reply: tid=0 2023-11-16T16:49:37.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.911+0000 7ff4f957eb80 1 client.25336 shutdown 2023-11-16T16:49:37.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.911+0000 7ff4f957eb80 1 -- 172.21.15.2:0/3234232117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f05550c0 msgr2=0x55c9f0564af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.911+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/3234232117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f05550c0 0x55c9f0564af0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8019ef0 tx=0x7ff4d8004ad0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.911+0000 7ff4f957eb80 1 -- 172.21.15.2:0/3234232117 shutdown_connections 2023-11-16T16:49:37.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.911+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/3234232117 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55c9f0568810 0x55c9f0567bc0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.911+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/3234232117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f05550c0 0x55c9f0564af0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.911+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/3234232117 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0553bb0 0x55c9f055d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.911+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/3234232117 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f0554500 0x55c9f055dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.911+0000 7ff4f957eb80 1 -- 172.21.15.2:0/3234232117 >> 172.21.15.2:0/3234232117 conn(0x55c9f0549310 msgr2=0x55c9f054dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.911+0000 7ff4beffd700 0 client.25336 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:49:37.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.911+0000 7ff4f957eb80 1 -- 172.21.15.2:0/3234232117 shutdown_connections 2023-11-16T16:49:37.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.911+0000 7ff4f957eb80 1 -- 172.21.15.2:0/3234232117 wait complete. 2023-11-16T16:49:37.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.911+0000 7ff4f957eb80 20 client.25336 trim_cache size 0 max 16384 2023-11-16T16:49:37.931 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.930+0000 7ff4f957eb80 1 Processor -- start 2023-11-16T16:49:37.932 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.930+0000 7ff4f957eb80 1 -- start start 2023-11-16T16:49:37.932 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.930+0000 7ff4f957eb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f0553bb0 0x55c9f0557fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.932 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.930+0000 7ff4f957eb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0554500 0x55c9f05584f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.932 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.930+0000 7ff4f957eb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f05550c0 0x55c9f055d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.932 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.930+0000 7ff4f957eb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9f0467050 con 0x55c9f0554500 2023-11-16T16:49:37.932 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.930+0000 7ff4f957eb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9f03b0ed0 con 0x55c9f05550c0 2023-11-16T16:49:37.932 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.930+0000 7ff4f957eb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9f0440e10 con 0x55c9f0553bb0 2023-11-16T16:49:37.933 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.930+0000 7ff4deffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0554500 0x55c9f05584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.930+0000 7ff4deffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0554500 0x55c9f05584f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43372/0 (socket says 172.21.15.2:43372) 2023-11-16T16:49:37.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.930+0000 7ff4df7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f0553bb0 0x55c9f0557fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.930+0000 7ff4deffd700 1 -- 172.21.15.2:0/3285804920 learned_addr learned my addr 172.21.15.2:0/3285804920 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:37.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.931+0000 7ff4dffff700 1 --2- 172.21.15.2:0/3285804920 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f05550c0 0x55c9f055d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.931+0000 7ff4deffd700 1 -- 172.21.15.2:0/3285804920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f0553bb0 msgr2=0x55c9f0557fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.931+0000 7ff4deffd700 1 --2- 172.21.15.2:0/3285804920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f0553bb0 0x55c9f0557fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.931+0000 7ff4deffd700 1 -- 172.21.15.2:0/3285804920 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f05550c0 msgr2=0x55c9f055d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.931+0000 7ff4deffd700 1 --2- 172.21.15.2:0/3285804920 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f05550c0 0x55c9f055d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.931+0000 7ff4deffd700 1 -- 172.21.15.2:0/3285804920 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9f046ae90 con 0x55c9f0554500 2023-11-16T16:49:37.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.931+0000 7ff4dffff700 1 --2- 172.21.15.2:0/3285804920 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f05550c0 0x55c9f055d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:49:37.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.931+0000 7ff4deffd700 1 --2- 172.21.15.2:0/3285804920 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0554500 0x55c9f05584f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff4d40098f0 tx=0x7ff4d4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dddeb7851a4dd12a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.934 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.931+0000 7ff4c67fc700 1 -- 172.21.15.2:0/3285804920 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d400f610 con 0x55c9f0554500 2023-11-16T16:49:37.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.931+0000 7ff4c67fc700 1 -- 172.21.15.2:0/3285804920 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4d401d040 con 0x55c9f0554500 2023-11-16T16:49:37.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.931+0000 7ff4c67fc700 1 -- 172.21.15.2:0/3285804920 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4005560 con 0x55c9f0554500 2023-11-16T16:49:37.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.932+0000 7ff4f957eb80 1 -- 172.21.15.2:0/3285804920 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0554500 msgr2=0x55c9f05584f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.932+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/3285804920 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0554500 0x55c9f05584f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff4d40098f0 tx=0x7ff4d4005dc0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.932+0000 7ff4f957eb80 1 -- 172.21.15.2:0/3285804920 shutdown_connections 2023-11-16T16:49:37.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.932+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/3285804920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f0553bb0 0x55c9f0557fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.932+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/3285804920 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f05550c0 0x55c9f055d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.932+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/3285804920 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0554500 0x55c9f05584f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.932+0000 7ff4f957eb80 1 -- 172.21.15.2:0/3285804920 >> 172.21.15.2:0/3285804920 conn(0x55c9f0549310 msgr2=0x55c9f0567bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.932+0000 7ff4f957eb80 1 -- 172.21.15.2:0/3285804920 shutdown_connections 2023-11-16T16:49:37.935 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.932+0000 7ff4f957eb80 1 -- 172.21.15.2:0/3285804920 wait complete. 2023-11-16T16:49:37.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.932+0000 7ff4f957eb80 1 -- start start 2023-11-16T16:49:37.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.933+0000 7ff4f957eb80 1 Processor -- start 2023-11-16T16:49:37.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.933+0000 7ff4f957eb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0553bb0 0x55c9f055a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.933+0000 7ff4f957eb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f0554500 0x55c9f05be700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.939 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.933+0000 7ff4f957eb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f05550c0 0x55c9f05c0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.933+0000 7ff4f957eb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9f05c13e0 con 0x55c9f0553bb0 2023-11-16T16:49:37.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.933+0000 7ff4f957eb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9f05c1550 con 0x55c9f05550c0 2023-11-16T16:49:37.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.933+0000 7ff4f957eb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9f05c1710 con 0x55c9f0554500 2023-11-16T16:49:37.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.933+0000 7ff4f957eb80 10 client.0 _ll_register_callbacks cb 0x7ff4d80198a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:49:37.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.933+0000 7ff4df7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0553bb0 0x55c9f055a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.933+0000 7ff4deffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f0554500 0x55c9f05be700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.933+0000 7ff4deffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f0554500 0x55c9f05be700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:33386/0 (socket says 172.21.15.2:33386) 2023-11-16T16:49:37.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.933+0000 7ff4deffd700 1 -- 172.21.15.2:0/87431284 learned_addr learned my addr 172.21.15.2:0/87431284 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:37.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.934+0000 7ff4dffff700 1 --2- 172.21.15.2:0/87431284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f05550c0 0x55c9f05c0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.934+0000 7ff4deffd700 1 -- 172.21.15.2:0/87431284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f05550c0 msgr2=0x55c9f05c0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.934+0000 7ff4deffd700 1 --2- 172.21.15.2:0/87431284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f05550c0 0x55c9f05c0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.934+0000 7ff4deffd700 1 -- 172.21.15.2:0/87431284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0553bb0 msgr2=0x55c9f055a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:49:37.940 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.934+0000 7ff4deffd700 1 --2- 172.21.15.2:0/87431284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0553bb0 0x55c9f055a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.934+0000 7ff4deffd700 1 -- 172.21.15.2:0/87431284 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e0108fa0 con 0x55c9f0554500 2023-11-16T16:49:37.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.934+0000 7ff4df7fe700 1 --2- 172.21.15.2:0/87431284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0553bb0 0x55c9f055a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:37.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.934+0000 7ff4deffd700 1 --2- 172.21.15.2:0/87431284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f0554500 0x55c9f05be700 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4009fd0 tx=0x7ff4d40055e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=221eef043140c974 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:37.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.934+0000 7ff4beffd700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:49:37.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.934+0000 7ff4beffd700 1 -- 172.21.15.2:0/87431284 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d400b970 con 0x55c9f0554500 2023-11-16T16:49:37.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.934+0000 7ff4beffd700 1 -- 172.21.15.2:0/87431284 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7ff4d4003680 con 0x55c9f0554500 2023-11-16T16:49:37.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.934+0000 7ff4f957eb80 1 -- 172.21.15.2:0/87431284 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff4d009dd90 con 0x55c9f0554500 2023-11-16T16:49:37.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.935+0000 7ff4beffd700 1 -- 172.21.15.2:0/87431284 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4d400bad0 con 0x55c9f0554500 2023-11-16T16:49:37.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.935+0000 7ff4f957eb80 10 client.25342 fetch_fsmap learned FSMap version 26 2023-11-16T16:49:37.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.935+0000 7ff4f957eb80 1 -- 172.21.15.2:0/87431284 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55c9f02bbc10 con 0x55c9f0554500 2023-11-16T16:49:37.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.935+0000 7ff4beffd700 1 -- 172.21.15.2:0/87431284 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4d009dd90 con 0x55c9f0554500 2023-11-16T16:49:37.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.935+0000 7ff4beffd700 1 -- 172.21.15.2:0/87431284 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7ff4d4038070 con 0x55c9f0554500 2023-11-16T16:49:37.941 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.935+0000 7ff4f957eb80 10 client.25342 fetch_fsmap finished waiting for FSMap version 26 2023-11-16T16:49:37.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.935+0000 7ff4f957eb80 10 client.25342 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-16T16:49:37.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.935+0000 7ff4f957eb80 20 client.25342 populate_metadata read hostname 'smithi002' 2023-11-16T16:49:37.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.935+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/87431284 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55c9f05c60c0 0x55c9f05c29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:37.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.935+0000 7ff4f957eb80 4 client.25342 mds_command: new command op to 24590 tid=0[{"prefix": "status"}] 2023-11-16T16:49:37.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.935+0000 7ff4f957eb80 1 -- 172.21.15.2:0/87431284 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c9f0473610 con 0x55c9f05c60c0 2023-11-16T16:49:37.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.936+0000 7ff4df7fe700 1 --2- 172.21.15.2:0/87431284 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55c9f05c60c0 0x55c9f05c29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:37.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.936+0000 7ff4df7fe700 1 --2- 172.21.15.2:0/87431284 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55c9f05c60c0 0x55c9f05c29b0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ecb44e12b0317fd server_cookie=e1bffe2b00933321 in_seq=0 out_seq=0 2023-11-16T16:49:37.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.936+0000 7ff4beffd700 10 client.25342 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:49:37.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.936+0000 7ff4beffd700 1 -- 172.21.15.2:0/87431284 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55c9f0473610 con 0x55c9f05c60c0 2023-11-16T16:49:37.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.936+0000 7ff4beffd700 10 client.25342 handle_command_reply: tid=0 2023-11-16T16:49:37.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.936+0000 7ff4f957eb80 1 client.25342 shutdown 2023-11-16T16:49:37.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.936+0000 7ff4f957eb80 1 -- 172.21.15.2:0/87431284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f0554500 msgr2=0x55c9f05be700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.942 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.936+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/87431284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f0554500 0x55c9f05be700 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4009fd0 tx=0x7ff4d40055e0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.936+0000 7ff4f957eb80 1 -- 172.21.15.2:0/87431284 shutdown_connections 2023-11-16T16:49:37.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.936+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/87431284 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c9f05550c0 0x55c9f05c0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.936+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/87431284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c9f0553bb0 0x55c9f055a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.936+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/87431284 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55c9f05c60c0 0x55c9f05c29b0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.936+0000 7ff4f957eb80 1 --2- 172.21.15.2:0/87431284 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9f0554500 0x55c9f05be700 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.936+0000 7ff4beffd700 0 client.25342 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:49:37.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.936+0000 7ff4f957eb80 1 -- 172.21.15.2:0/87431284 >> 172.21.15.2:0/87431284 conn(0x55c9f0549310 msgr2=0x55c9f0567bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.937+0000 7ff4f957eb80 1 -- 172.21.15.2:0/87431284 shutdown_connections 2023-11-16T16:49:37.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.937+0000 7ff4f957eb80 1 -- 172.21.15.2:0/87431284 wait complete. 2023-11-16T16:49:37.943 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.937+0000 7ff4f957eb80 20 client.25342 trim_cache size 0 max 16384 2023-11-16T16:49:37.943 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T16:49:37.944 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "875fcd24-849e-11ee-8dbe-212e2dc638e7", 2023-11-16T16:49:37.944 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2023-11-16T16:49:37.944 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24590, 2023-11-16T16:49:37.944 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2023-11-16T16:49:37.944 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2023-11-16T16:49:37.944 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2023-11-16T16:49:37.944 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 131.58566879599999, 2023-11-16T16:49:37.944 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 26, 2023-11-16T16:49:37.944 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 93, 2023-11-16T16:49:37.944 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 91, 2023-11-16T16:49:37.944 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 137.632295083 2023-11-16T16:49:37.944 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T16:49:37.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.937+0000 7ff4e5bff700 1 -- 172.21.15.2:0/625501199 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff4c80a11d0 msgr2=0x7ff4c80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.937+0000 7ff4e5bff700 1 --2- 172.21.15.2:0/625501199 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff4c80a11d0 0x7ff4c80a3680 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff4d800f590 tx=0x7ff4d800b040 comp rx=0 tx=0).stop 2023-11-16T16:49:37.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.937+0000 7ff4e5bff700 1 -- 172.21.15.2:0/625501199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4e0100230 msgr2=0x7ff4e0195f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:37.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.937+0000 7ff4e5bff700 1 --2- 172.21.15.2:0/625501199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4e0100230 0x7ff4e0195f90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000e9f0 tx=0x7ff4d000ed00 comp rx=0 tx=0).stop 2023-11-16T16:49:37.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.937+0000 7ff4e5bff700 1 -- 172.21.15.2:0/625501199 shutdown_connections 2023-11-16T16:49:37.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.937+0000 7ff4e5bff700 1 --2- 172.21.15.2:0/625501199 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7ff4c80a11d0 0x7ff4c80a3680 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.937+0000 7ff4e5bff700 1 --2- 172.21.15.2:0/625501199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4e0100230 0x7ff4e0195f90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.945 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.937+0000 7ff4e5bff700 1 --2- 172.21.15.2:0/625501199 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff4e0101db0 0x7ff4e019d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.937+0000 7ff4e5bff700 1 --2- 172.21.15.2:0/625501199 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4e0101420 0x7ff4e01964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:37.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.937+0000 7ff4e5bff700 1 -- 172.21.15.2:0/625501199 >> 172.21.15.2:0/625501199 conn(0x7ff4e00fb810 msgr2=0x7ff4e0104930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:37.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.937+0000 7ff4e5bff700 1 -- 172.21.15.2:0/625501199 shutdown_connections 2023-11-16T16:49:37.946 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:37.938+0000 7ff4e5bff700 1 -- 172.21.15.2:0/625501199 wait complete. 2023-11-16T16:49:37.955 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:37 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1634032009' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:37.958 INFO:teuthology.run_tasks:Running task kclient... 2023-11-16T16:49:37.973 INFO:tasks.kclient:Mounting kernel clients... 2023-11-16T16:49:37.974 INFO:tasks.kclient:config is {'client.0': {}} 2023-11-16T16:49:37.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:37.974 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2023-11-16T16:49:38.017 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-11-16T16:49:38.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:38.019 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-11-16T16:49:38.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:38.129 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link delete ceph-brx 2023-11-16T16:49:38.253 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2023-11-16T16:49:38.253 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-16T16:49:38.253 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-16T16:49:38.253 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph fs ls 2023-11-16T16:49:38.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:37 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1634032009' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:38.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:37 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1634032009' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:49:39.114 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T16:49:39.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:38 smithi002 ceph-mon[128712]: pgmap v350: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T16:49:39.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:38 smithi141 ceph-mon[131184]: pgmap v350: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T16:49:39.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:38 smithi155 ceph-mon[131607]: pgmap v350: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T16:49:40.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.618+0000 7f9c02fe7700 1 -- 172.21.15.2:0/1059432533 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bfc100ee0 msgr2=0x7f9bfc104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:40.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.618+0000 7f9c02fe7700 1 --2- 172.21.15.2:0/1059432533 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bfc100ee0 0x7f9bfc104eb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9be80099c0 tx=0x7f9be8009cd0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.618+0000 7f9c02fe7700 1 -- 172.21.15.2:0/1059432533 shutdown_connections 2023-11-16T16:49:40.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.618+0000 7f9c02fe7700 1 --2- 172.21.15.2:0/1059432533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9bfc1055e0 0x7f9bfc10d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.618+0000 7f9c02fe7700 1 --2- 172.21.15.2:0/1059432533 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bfc100540 0x7f9bfc100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.620 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.618+0000 7f9c02fe7700 1 --2- 172.21.15.2:0/1059432533 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bfc100ee0 0x7f9bfc104eb0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.621 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.618+0000 7f9c02fe7700 1 -- 172.21.15.2:0/1059432533 >> 172.21.15.2:0/1059432533 conn(0x7f9bfc0fbdc0 msgr2=0x7f9bfc0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:40.621 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.618+0000 7f9c02fe7700 1 -- 172.21.15.2:0/1059432533 shutdown_connections 2023-11-16T16:49:40.621 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.618+0000 7f9c02fe7700 1 -- 172.21.15.2:0/1059432533 wait complete. 2023-11-16T16:49:40.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.619+0000 7f9c02fe7700 1 Processor -- start 2023-11-16T16:49:40.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.619+0000 7f9c02fe7700 1 -- start start 2023-11-16T16:49:40.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.619+0000 7f9c02fe7700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bfc100540 0x7f9bfc1963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:40.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.619+0000 7f9c02fe7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9bfc100ee0 0x7f9bfc196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:40.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.619+0000 7f9c02fe7700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bfc1055e0 0x7f9bfc19ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:40.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.619+0000 7f9c02fe7700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bfc19b390 con 0x7f9bfc1055e0 2023-11-16T16:49:40.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.619+0000 7f9c02fe7700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bfc19b500 con 0x7f9bfc100540 2023-11-16T16:49:40.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.619+0000 7f9c02fe7700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bfc19b6c0 con 0x7f9bfc100ee0 2023-11-16T16:49:40.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.620+0000 7f9c00d83700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bfc100540 0x7f9bfc1963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:40.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.620+0000 7f9c00d83700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bfc100540 0x7f9bfc1963d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:41410/0 (socket says 172.21.15.2:41410) 2023-11-16T16:49:40.623 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.620+0000 7f9c01584700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bfc1055e0 0x7f9bfc19ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:40.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.620+0000 7f9c00d83700 1 -- 172.21.15.2:0/230393520 learned_addr learned my addr 172.21.15.2:0/230393520 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:40.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.620+0000 7f9bfbfff700 1 --2- 172.21.15.2:0/230393520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9bfc100ee0 0x7f9bfc196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:40.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.620+0000 7f9c00d83700 1 -- 172.21.15.2:0/230393520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9bfc100ee0 msgr2=0x7f9bfc196910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:40.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.620+0000 7f9c00d83700 1 --2- 172.21.15.2:0/230393520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9bfc100ee0 0x7f9bfc196910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.620+0000 7f9c00d83700 1 -- 172.21.15.2:0/230393520 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bfc1055e0 msgr2=0x7f9bfc19ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:40.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.620+0000 7f9c00d83700 1 --2- 172.21.15.2:0/230393520 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bfc1055e0 0x7f9bfc19ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.620+0000 7f9c00d83700 1 -- 172.21.15.2:0/230393520 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be80095f0 con 0x7f9bfc100540 2023-11-16T16:49:40.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.621+0000 7f9c00d83700 1 --2- 172.21.15.2:0/230393520 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bfc100540 0x7f9bfc1963d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9bf000d980 tx=0x7f9bf000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:40.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.621+0000 7f9bf9ffb700 1 -- 172.21.15.2:0/230393520 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0004220 con 0x7f9bfc100540 2023-11-16T16:49:40.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.621+0000 7f9bf9ffb700 1 -- 172.21.15.2:0/230393520 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9bf0007640 con 0x7f9bfc100540 2023-11-16T16:49:40.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.621+0000 7f9c02fe7700 1 -- 172.21.15.2:0/230393520 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bfc110360 con 0x7f9bfc100540 2023-11-16T16:49:40.624 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.621+0000 7f9bf9ffb700 1 -- 172.21.15.2:0/230393520 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0005690 con 0x7f9bfc100540 2023-11-16T16:49:40.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.621+0000 7f9c02fe7700 1 -- 172.21.15.2:0/230393520 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bfc1101d0 con 0x7f9bfc100540 2023-11-16T16:49:40.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.622+0000 7f9c02fe7700 1 -- 172.21.15.2:0/230393520 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bfc1104f0 con 0x7f9bfc100540 2023-11-16T16:49:40.625 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.624+0000 7f9bf9ffb700 1 -- 172.21.15.2:0/230393520 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f9bf0004750 con 0x7f9bfc100540 2023-11-16T16:49:40.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.625+0000 7f9bf9ffb700 1 --2- 172.21.15.2:0/230393520 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9bec0a1150 0x7f9bec0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:40.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.625+0000 7f9bf9ffb700 1 -- 172.21.15.2:0/230393520 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f9bf00ccd60 con 0x7f9bfc100540 2023-11-16T16:49:40.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.625+0000 7f9bfbfff700 1 --2- 172.21.15.2:0/230393520 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9bec0a1150 0x7f9bec0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:40.627 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.626+0000 7f9bfbfff700 1 --2- 172.21.15.2:0/230393520 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9bec0a1150 0x7f9bec0a3600 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9be8006010 tx=0x7f9be8003680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:40.629 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.627+0000 7f9bf9ffb700 1 -- 172.21.15.2:0/230393520 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9bf00960f0 con 0x7f9bfc100540 2023-11-16T16:49:40.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:40 smithi002 ceph-mon[128712]: pgmap v351: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T16:49:40.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:40 smithi155 ceph-mon[131607]: pgmap v351: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T16:49:40.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.878+0000 7f9c02fe7700 1 -- 172.21.15.2:0/230393520 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9bfc1a3450 con 0x7f9bfc100540 2023-11-16T16:49:40.880 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.879+0000 7f9bf9ffb700 1 -- 172.21.15.2:0/230393520 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) v1 ==== 53+0+88 (secure 0 0 0) 0x7f9bf0016020 con 0x7f9bfc100540 2023-11-16T16:49:40.881 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-11-16T16:49:40.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.881+0000 7f9c02fe7700 1 -- 172.21.15.2:0/230393520 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9bec0a1150 msgr2=0x7f9bec0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:40.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.881+0000 7f9c02fe7700 1 --2- 172.21.15.2:0/230393520 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9bec0a1150 0x7f9bec0a3600 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9be8006010 tx=0x7f9be8003680 comp rx=0 tx=0).stop 2023-11-16T16:49:40.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.881+0000 7f9c02fe7700 1 -- 172.21.15.2:0/230393520 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bfc100540 msgr2=0x7f9bfc1963d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:40.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.881+0000 7f9c02fe7700 1 --2- 172.21.15.2:0/230393520 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bfc100540 0x7f9bfc1963d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9bf000d980 tx=0x7f9bf000dc90 comp rx=0 tx=0).stop 2023-11-16T16:49:40.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.881+0000 7f9c02fe7700 1 -- 172.21.15.2:0/230393520 shutdown_connections 2023-11-16T16:49:40.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.881+0000 7f9c02fe7700 1 --2- 172.21.15.2:0/230393520 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f9bec0a1150 0x7f9bec0a3600 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.881+0000 7f9c02fe7700 1 --2- 172.21.15.2:0/230393520 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bfc1055e0 0x7f9bfc19ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.881+0000 7f9c02fe7700 1 --2- 172.21.15.2:0/230393520 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f9bfc100540 0x7f9bfc1963d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.883 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.881+0000 7f9c02fe7700 1 --2- 172.21.15.2:0/230393520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9bfc100ee0 0x7f9bfc196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:40.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.881+0000 7f9c02fe7700 1 -- 172.21.15.2:0/230393520 >> 172.21.15.2:0/230393520 conn(0x7f9bfc0fbdc0 msgr2=0x7f9bfc0fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:40.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.881+0000 7f9c02fe7700 1 -- 172.21.15.2:0/230393520 shutdown_connections 2023-11-16T16:49:40.884 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:40.881+0000 7f9c02fe7700 1 -- 172.21.15.2:0/230393520 wait complete. 2023-11-16T16:49:41.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:40 smithi141 ceph-mon[131184]: pgmap v351: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-11-16T16:49:41.503 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-16T16:49:41.504 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-16T16:49:41.504 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi002.front.sepia.ceph.com 2023-11-16T16:49:41.504 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-16T16:49:41.504 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-16T16:49:41.504 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-16T16:49:41.504 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-16T16:49:41.504 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-16T16:49:41.504 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-16T16:49:41.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:41.505 DEBUG:teuthology.orchestra.run.smithi002:> ip addr 2023-11-16T16:49:41.552 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-16T16:49:41.552 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-16T16:49:41.552 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2023-11-16T16:49:41.552 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2023-11-16T16:49:41.552 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2023-11-16T16:49:41.552 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2023-11-16T16:49:41.552 INFO:teuthology.orchestra.run.smithi002.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T16:49:41.552 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:02 brd ff:ff:ff:ff:ff:ff 2023-11-16T16:49:41.553 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f0 2023-11-16T16:49:41.553 INFO:teuthology.orchestra.run.smithi002.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T16:49:41.553 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:03 brd ff:ff:ff:ff:ff:ff 2023-11-16T16:49:41.553 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f1 2023-11-16T16:49:41.553 INFO:teuthology.orchestra.run.smithi002.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-16T16:49:41.553 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:20 brd ff:ff:ff:ff:ff:ff 2023-11-16T16:49:41.553 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f0 2023-11-16T16:49:41.553 INFO:teuthology.orchestra.run.smithi002.stdout: inet 172.21.15.2/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-11-16T16:49:41.553 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft 41430sec preferred_lft 41430sec 2023-11-16T16:49:41.553 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::ec4:7aff:febd:1920/64 scope link 2023-11-16T16:49:41.553 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2023-11-16T16:49:41.554 INFO:teuthology.orchestra.run.smithi002.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-16T16:49:41.554 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:21 brd ff:ff:ff:ff:ff:ff 2023-11-16T16:49:41.554 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f1 2023-11-16T16:49:41.555 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-16T16:49:41.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:41.555 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T16:49:41.555 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T16:49:41.555 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add name ceph-brx type bridge 2023-11-16T16:49:41.555 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr flush dev ceph-brx 2023-11-16T16:49:41.555 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set ceph-brx up 2023-11-16T16:49:41.555 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-16T16:49:41.555 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T16:49:41.725 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:41 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/230393520' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T16:49:41.764 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:41 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/230393520' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T16:49:41.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:41.766 DEBUG:teuthology.orchestra.run.smithi002:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-16T16:49:41.839 INFO:teuthology.orchestra.run.smithi002.stdout:1 2023-11-16T16:49:41.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:41.842 DEBUG:teuthology.orchestra.run.smithi002:> route 2023-11-16T16:49:41.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:41 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/230393520' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-11-16T16:49:41.890 INFO:teuthology.orchestra.run.smithi002.stdout:Kernel IP routing table 2023-11-16T16:49:41.890 INFO:teuthology.orchestra.run.smithi002.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-16T16:49:41.890 INFO:teuthology.orchestra.run.smithi002.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-11-16T16:49:41.890 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-11-16T16:49:41.890 INFO:teuthology.orchestra.run.smithi002.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-16T16:49:41.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:41.891 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T16:49:41.891 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T16:49:41.892 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-11-16T16:49:41.892 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-11-16T16:49:41.892 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-11-16T16:49:41.892 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T16:49:42.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:42.068 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2023-11-16T16:49:42.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:42.113 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list-id 2023-11-16T16:49:42.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:42.197 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T16:49:42.197 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T16:49:42.197 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-16T16:49:42.197 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-11-16T16:49:42.197 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T16:49:42.386 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-11-16T16:49:42.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:42.386 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T16:49:42.386 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T16:49:42.386 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-11-16T16:49:42.386 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-16T16:49:42.386 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-11-16T16:49:42.386 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-11-16T16:49:42.386 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-11-16T16:49:42.386 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T16:49:42.600 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:42 smithi002 ceph-mon[128712]: pgmap v352: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T16:49:42.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:42.646 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2023-11-16T16:49:42.646 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2023-11-16T16:49:42.646 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set brx.0 up 2023-11-16T16:49:42.647 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set dev brx.0 master ceph-brx 2023-11-16T16:49:42.647 DEBUG:teuthology.orchestra.run.smithi002:> ') 2023-11-16T16:49:42.823 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-16T16:49:42.823 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-16T16:49:42.867 INFO:teuthology.orchestra.run.smithi002.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-16T16:49:42.867 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-16T16:49:42.867 DEBUG:teuthology.orchestra.run.smithi002:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-16T16:49:42.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:42 smithi155 ceph-mon[131607]: pgmap v352: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T16:49:42.951 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2023-11-16T16:49:42.951 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2023-11-16T16:49:42.952 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/_nogroup/sv_1/81f7d666-f2e8-486f-8c44-a77b2ceae0af 2023-11-16T16:49:42.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:42.952 DEBUG:teuthology.orchestra.run.smithi002:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/_nogroup/sv_1/81f7d666-f2e8-486f-8c44-a77b2ceae0af /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2023-11-16T16:49:43.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:42 smithi141 ceph-mon[131184]: pgmap v352: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T16:49:43.098 INFO:teuthology.orchestra.run.smithi002.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2023-11-16T16:49:43.099 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2023-11-16T16:49:43.599 INFO:teuthology.orchestra.run.smithi002.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2023-11-16T16:49:43.599 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2023-11-16T16:49:43.599 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: resolved to: "172.21.15.2:3300,172.21.15.141:3300,172.21.15.155:3300" 2023-11-16T16:49:43.600 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: trying mount with new device syntax: 0@875fcd24-849e-11ee-8dbe-212e2dc638e7.cephfs=/volumes/_nogroup/sv_1/81f7d666-f2e8-486f-8c44-a77b2ceae0af 2023-11-16T16:49:43.600 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync,name=0,key=0,mon_addr=172.21.15.2:3300/172.21.15.141:3300/172.21.15.155:3300" will pass to kernel 2023-11-16T16:49:43.607 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-16T16:49:43.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:43.607 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-16T16:49:43.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:43.681 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-16T16:49:43.770 INFO:teuthology.orchestra.run.smithi002.stdout:client15351 2023-11-16T16:49:43.770 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T16:49:43.770 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-16T16:49:43.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:43.816 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-16T16:49:43.944 INFO:teuthology.orchestra.run.smithi002.stdout:client15351 2023-11-16T16:49:43.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:43.945 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-16T16:49:44.033 INFO:teuthology.orchestra.run.smithi002.stdout:client15351 2023-11-16T16:49:44.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T16:49:44.034 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7.client15351/status') 2023-11-16T16:49:44.144 INFO:teuthology.orchestra.run.smithi002.stdout:instance: client.15351 (3)192.168.0.1:0/1620268271 2023-11-16T16:49:44.144 INFO:teuthology.orchestra.run.smithi002.stdout:blocklisted: false 2023-11-16T16:49:44.145 INFO:teuthology.orchestra.run.smithi002.stderr:0+1 records in 2023-11-16T16:49:44.145 INFO:teuthology.orchestra.run.smithi002.stderr:0+1 records out 2023-11-16T16:49:44.145 INFO:teuthology.orchestra.run.smithi002.stderr:70 bytes copied, 5.3026e-05 s, 1.3 MB/s 2023-11-16T16:49:44.148 INFO:teuthology.run_tasks:Running task check-counter... 2023-11-16T16:49:44.160 INFO:teuthology.run_tasks:Running task exec... 2023-11-16T16:49:44.180 INFO:teuthology.task.exec:Executing custom commands... 2023-11-16T16:49:44.181 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T16:49:44.181 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2023-11-16T16:49:44.383 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.378+0000 7f19ba9ff700 1 -- 172.21.15.2:0/3278344049 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19b4101dc0 msgr2=0x7f19b4108680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:44.384 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.378+0000 7f19ba9ff700 1 --2- 172.21.15.2:0/3278344049 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19b4101dc0 0x7f19b4108680 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f19ac00b0a0 tx=0x7f19ac00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.384 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.378+0000 7f19ba9ff700 1 -- 172.21.15.2:0/3278344049 shutdown_connections 2023-11-16T16:49:44.384 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.378+0000 7f19ba9ff700 1 --2- 172.21.15.2:0/3278344049 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19b4101dc0 0x7f19b4108680 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.384 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.378+0000 7f19ba9ff700 1 --2- 172.21.15.2:0/3278344049 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f19b4100240 0x7f19b4100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.384 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.378+0000 7f19ba9ff700 1 --2- 172.21.15.2:0/3278344049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19b4101430 0x7f19b4101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.384 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.378+0000 7f19ba9ff700 1 -- 172.21.15.2:0/3278344049 >> 172.21.15.2:0/3278344049 conn(0x7f19b40fb840 msgr2=0x7f19b40fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:44.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.379+0000 7f19ba9ff700 1 -- 172.21.15.2:0/3278344049 shutdown_connections 2023-11-16T16:49:44.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.379+0000 7f19ba9ff700 1 -- 172.21.15.2:0/3278344049 wait complete. 2023-11-16T16:49:44.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.380+0000 7f19ba9ff700 1 Processor -- start 2023-11-16T16:49:44.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.380+0000 7f19ba9ff700 1 -- start start 2023-11-16T16:49:44.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.380+0000 7f19ba9ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19b4100240 0x7f19b40748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:44.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.380+0000 7f19ba9ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19b4101430 0x7f19b4072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:44.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.380+0000 7f19ba9ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f19b4101dc0 0x7f19b4073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:44.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.380+0000 7f19ba9ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b4073bc0 con 0x7f19b4100240 2023-11-16T16:49:44.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.380+0000 7f19ba9ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b4073d30 con 0x7f19b4101dc0 2023-11-16T16:49:44.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.380+0000 7f19ba9ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b41a0160 con 0x7f19b4101430 2023-11-16T16:49:44.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.380+0000 7f19b3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19b4100240 0x7f19b40748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:44.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.380+0000 7f19b3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19b4100240 0x7f19b40748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53474/0 (socket says 172.21.15.2:53474) 2023-11-16T16:49:44.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.380+0000 7f19b3fff700 1 -- 172.21.15.2:0/3243044483 learned_addr learned my addr 172.21.15.2:0/3243044483 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:44.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.380+0000 7f19abfff700 1 --2- 172.21.15.2:0/3243044483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19b4101430 0x7f19b4072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:44.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.380+0000 7f19b8f9c700 1 --2- 172.21.15.2:0/3243044483 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f19b4101dc0 0x7f19b4073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:44.386 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.381+0000 7f19b3fff700 1 -- 172.21.15.2:0/3243044483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19b4101430 msgr2=0x7f19b4072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:44.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.381+0000 7f19b3fff700 1 --2- 172.21.15.2:0/3243044483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19b4101430 0x7f19b4072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.381+0000 7f19b3fff700 1 -- 172.21.15.2:0/3243044483 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f19b4101dc0 msgr2=0x7f19b4073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:44.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.381+0000 7f19b3fff700 1 --2- 172.21.15.2:0/3243044483 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f19b4101dc0 0x7f19b4073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:44.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.381+0000 7f19b3fff700 1 -- 172.21.15.2:0/3243044483 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ac009d00 con 0x7f19b4100240 2023-11-16T16:49:44.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.381+0000 7f19b3fff700 1 --2- 172.21.15.2:0/3243044483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19b4100240 0x7f19b40748a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f19a400d980 tx=0x7f19a400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:44.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.381+0000 7f19b1ffb700 1 -- 172.21.15.2:0/3243044483 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a400f890 con 0x7f19b4100240 2023-11-16T16:49:44.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.381+0000 7f19b1ffb700 1 -- 172.21.15.2:0/3243044483 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f19a4010460 con 0x7f19b4100240 2023-11-16T16:49:44.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.381+0000 7f19b1ffb700 1 -- 172.21.15.2:0/3243044483 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a40195c0 con 0x7f19b4100240 2023-11-16T16:49:44.387 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.381+0000 7f19ba9ff700 1 -- 172.21.15.2:0/3243044483 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b410bc20 con 0x7f19b4100240 2023-11-16T16:49:44.388 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.382+0000 7f19ba9ff700 1 -- 172.21.15.2:0/3243044483 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b410ba90 con 0x7f19b4100240 2023-11-16T16:49:44.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.382+0000 7f19ba9ff700 1 -- 172.21.15.2:0/3243044483 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b4107790 con 0x7f19b4100240 2023-11-16T16:49:44.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.384+0000 7f19b1ffb700 1 -- 172.21.15.2:0/3243044483 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 135714+0+0 (secure 0 0 0) 0x7f19a40105d0 con 0x7f19b4100240 2023-11-16T16:49:44.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.386+0000 7f19b1ffb700 1 --2- 172.21.15.2:0/3243044483 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f19a00a11d0 0x7f19a00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:44.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.386+0000 7f19b1ffb700 1 -- 172.21.15.2:0/3243044483 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f19a40cc9b0 con 0x7f19b4100240 2023-11-16T16:49:44.389 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.386+0000 7f19abfff700 1 --2- 172.21.15.2:0/3243044483 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f19a00a11d0 0x7f19a00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:44.390 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.387+0000 7f19abfff700 1 --2- 172.21.15.2:0/3243044483 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f19a00a11d0 0x7f19a00a3680 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f199c005950 tx=0x7f199c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:44.391 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.390+0000 7f19b1ffb700 1 -- 172.21.15.2:0/3243044483 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f19a4095d40 con 0x7f19b4100240 2023-11-16T16:49:44.673 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:44.671+0000 7f19ba9ff700 1 -- 172.21.15.2:0/3243044483 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f19b4061940 con 0x7f19b4100240 2023-11-16T16:49:45.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:44 smithi002 ceph-mon[128712]: pgmap v353: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:49:45.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:44 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3243044483' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-11-16T16:49:45.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:44 smithi141 ceph-mon[131184]: pgmap v353: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:49:45.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:44 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3243044483' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-11-16T16:49:45.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:44 smithi155 ceph-mon[131607]: pgmap v353: 129 pgs: 129 active+clean; 656 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:49:45.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:44 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3243044483' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-11-16T16:49:45.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.525+0000 7f19b1ffb700 1 -- 172.21.15.2:0/3243044483 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 19) v1 ==== 135731+0+0 (secure 0 0 0) 0x7f19a40108b0 con 0x7f19b4100240 2023-11-16T16:49:45.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.526+0000 7f19b1ffb700 1 -- 172.21.15.2:0/3243044483 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7f19a40acbc0 con 0x7f19b4100240 2023-11-16T16:49:45.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.528+0000 7f19ba9ff700 1 -- 172.21.15.2:0/3243044483 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f19a00a11d0 msgr2=0x7f19a00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.528+0000 7f19ba9ff700 1 --2- 172.21.15.2:0/3243044483 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f19a00a11d0 0x7f19a00a3680 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f199c005950 tx=0x7f199c00b410 comp rx=0 tx=0).stop 2023-11-16T16:49:45.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.528+0000 7f19ba9ff700 1 -- 172.21.15.2:0/3243044483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19b4100240 msgr2=0x7f19b40748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.528+0000 7f19ba9ff700 1 --2- 172.21.15.2:0/3243044483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19b4100240 0x7f19b40748a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f19a400d980 tx=0x7f19a400dc90 comp rx=0 tx=0).stop 2023-11-16T16:49:45.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.528+0000 7f19ba9ff700 1 -- 172.21.15.2:0/3243044483 shutdown_connections 2023-11-16T16:49:45.531 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.528+0000 7f19ba9ff700 1 --2- 172.21.15.2:0/3243044483 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f19a00a11d0 0x7f19a00a3680 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.528+0000 7f19ba9ff700 1 --2- 172.21.15.2:0/3243044483 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f19b4101dc0 0x7f19b4073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.528+0000 7f19ba9ff700 1 --2- 172.21.15.2:0/3243044483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19b4100240 0x7f19b40748a0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.528+0000 7f19ba9ff700 1 --2- 172.21.15.2:0/3243044483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19b4101430 0x7f19b4072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.528+0000 7f19ba9ff700 1 -- 172.21.15.2:0/3243044483 >> 172.21.15.2:0/3243044483 conn(0x7f19b40fb840 msgr2=0x7f19b41048d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:45.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.529+0000 7f19ba9ff700 1 -- 172.21.15.2:0/3243044483 shutdown_connections 2023-11-16T16:49:45.532 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.529+0000 7f19ba9ff700 1 -- 172.21.15.2:0/3243044483 wait complete. 2023-11-16T16:49:45.551 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2023-11-16T16:49:45.777 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.774+0000 7f5d661ff700 1 -- 172.21.15.2:0/2855444601 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d60103620 msgr2=0x7f5d60105a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.777 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.774+0000 7f5d661ff700 1 --2- 172.21.15.2:0/2855444601 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d60103620 0x7f5d60105a00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5d54009990 tx=0x7f5d54009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.777 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.774+0000 7f5d661ff700 1 -- 172.21.15.2:0/2855444601 shutdown_connections 2023-11-16T16:49:45.777 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.774+0000 7f5d661ff700 1 --2- 172.21.15.2:0/2855444601 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d60105f40 0x7f5d601083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.777 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.774+0000 7f5d661ff700 1 --2- 172.21.15.2:0/2855444601 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d60100d00 0x7f5d601030e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.777 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.774+0000 7f5d661ff700 1 --2- 172.21.15.2:0/2855444601 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d60103620 0x7f5d60105a00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.774+0000 7f5d661ff700 1 -- 172.21.15.2:0/2855444601 >> 172.21.15.2:0/2855444601 conn(0x7f5d600fa7a0 msgr2=0x7f5d600fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:45.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.774+0000 7f5d661ff700 1 -- 172.21.15.2:0/2855444601 shutdown_connections 2023-11-16T16:49:45.778 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.775+0000 7f5d661ff700 1 -- 172.21.15.2:0/2855444601 wait complete. 2023-11-16T16:49:45.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.776+0000 7f5d661ff700 1 Processor -- start 2023-11-16T16:49:45.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.776+0000 7f5d661ff700 1 -- start start 2023-11-16T16:49:45.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.777+0000 7f5d661ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d60100d00 0x7f5d60100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.777+0000 7f5d661ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d60103620 0x7f5d600fecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.777+0000 7f5d661ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d60105f40 0x7f5d600ff1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.777+0000 7f5d661ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d600ff990 con 0x7f5d60103620 2023-11-16T16:49:45.781 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.777+0000 7f5d661ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d600ffb00 con 0x7f5d60100d00 2023-11-16T16:49:45.782 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.777+0000 7f5d661ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6019fff0 con 0x7f5d60105f40 2023-11-16T16:49:45.782 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.777+0000 7f5d5f7fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d60100d00 0x7f5d60100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.782 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.777+0000 7f5d5effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d60103620 0x7f5d600fecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.782 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.777+0000 7f5d5effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d60103620 0x7f5d600fecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53514/0 (socket says 172.21.15.2:53514) 2023-11-16T16:49:45.782 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.777+0000 7f5d5f7fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d60100d00 0x7f5d60100670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:55624/0 (socket says 172.21.15.2:55624) 2023-11-16T16:49:45.782 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.777+0000 7f5d5effd700 1 -- 172.21.15.2:0/413648035 learned_addr learned my addr 172.21.15.2:0/413648035 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:45.782 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.777+0000 7f5d5ffff700 1 --2- 172.21.15.2:0/413648035 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d60105f40 0x7f5d600ff1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:45.782 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.778+0000 7f5d5f7fe700 1 -- 172.21.15.2:0/413648035 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d60105f40 msgr2=0x7f5d600ff1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.778+0000 7f5d5f7fe700 1 --2- 172.21.15.2:0/413648035 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d60105f40 0x7f5d600ff1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.778+0000 7f5d5f7fe700 1 -- 172.21.15.2:0/413648035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d60103620 msgr2=0x7f5d600fecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:45.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.778+0000 7f5d5f7fe700 1 --2- 172.21.15.2:0/413648035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d60103620 0x7f5d600fecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:45.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.778+0000 7f5d5f7fe700 1 -- 172.21.15.2:0/413648035 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d50009520 con 0x7f5d60100d00 2023-11-16T16:49:45.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.778+0000 7f5d5ffff700 1 --2- 172.21.15.2:0/413648035 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d60105f40 0x7f5d600ff1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:45.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.778+0000 7f5d5f7fe700 1 --2- 172.21.15.2:0/413648035 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d60100d00 0x7f5d60100670 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000e9f0 tx=0x7f5d5000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:45.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.778+0000 7f5d5cff9700 1 -- 172.21.15.2:0/413648035 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5000cb70 con 0x7f5d60100d00 2023-11-16T16:49:45.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.778+0000 7f5d5cff9700 1 -- 172.21.15.2:0/413648035 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d5000ee90 con 0x7f5d60100d00 2023-11-16T16:49:45.783 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.778+0000 7f5d661ff700 1 -- 172.21.15.2:0/413648035 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d540095f0 con 0x7f5d60100d00 2023-11-16T16:49:45.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.778+0000 7f5d5cff9700 1 -- 172.21.15.2:0/413648035 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d50003cf0 con 0x7f5d60100d00 2023-11-16T16:49:45.784 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.779+0000 7f5d661ff700 1 -- 172.21.15.2:0/413648035 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d6010bc00 con 0x7f5d60100d00 2023-11-16T16:49:45.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.780+0000 7f5d661ff700 1 -- 172.21.15.2:0/413648035 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d600fc310 con 0x7f5d60100d00 2023-11-16T16:49:45.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.782+0000 7f5d5cff9700 1 -- 172.21.15.2:0/413648035 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 19) v1 ==== 135731+0+0 (secure 0 0 0) 0x7f5d50004020 con 0x7f5d60100d00 2023-11-16T16:49:45.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.783+0000 7f5d5cff9700 1 --2- 172.21.15.2:0/413648035 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5d480a11a0 0x7f5d480a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:45.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.783+0000 7f5d5effd700 1 -- 172.21.15.2:0/413648035 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5d480a11a0 msgr2=0x7f5d480a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1793067160 2023-11-16T16:49:45.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.783+0000 7f5d5cff9700 1 -- 172.21.15.2:0/413648035 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f5d50014070 con 0x7f5d60100d00 2023-11-16T16:49:45.785 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.783+0000 7f5d5effd700 1 --2- 172.21.15.2:0/413648035 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5d480a11a0 0x7f5d480a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:49:45.789 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.787+0000 7f5d5cff9700 1 -- 172.21.15.2:0/413648035 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5d500963a0 con 0x7f5d60100d00 2023-11-16T16:49:45.823 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:45 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: ignoring --setuser ceph since I am not root 2023-11-16T16:49:45.823 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:45 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: ignoring --setgroup ceph since I am not root 2023-11-16T16:49:45.824 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:45 smithi002 ceph-mgr[128983]: -- 172.21.15.2:0/4179872717 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562d56a09520 con 0x562d569e1000 2023-11-16T16:49:45.867 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:45 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: ignoring --setuser ceph since I am not root 2023-11-16T16:49:45.867 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:45 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: ignoring --setgroup ceph since I am not root 2023-11-16T16:49:45.868 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:45 smithi155 ceph-mgr[133748]: -- 172.21.15.155:0/3469095478 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56065a4f5520 con 0x56065a4ca800 2023-11-16T16:49:45.868 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:45 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:45.691+0000 7f4b8e3d1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T16:49:45.868 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:45 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:45.770+0000 7f4b8e3d1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T16:49:45.870 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:45 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: ignoring --setuser ceph since I am not root 2023-11-16T16:49:45.870 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:45 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: ignoring --setgroup ceph since I am not root 2023-11-16T16:49:45.870 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:45 smithi141 ceph-mgr[132674]: -- 172.21.15.141:0/1515088828 <== mon.2 v2:172.21.15.141:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b75841f520 con 0x55b7583f5000 2023-11-16T16:49:45.870 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:45 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:45.692+0000 7efe03059200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T16:49:45.870 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:45 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:45.771+0000 7efe03059200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T16:49:45.871 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:45 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:45.868+0000 7efe03059200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T16:49:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.984+0000 7f5d5effd700 1 -- 172.21.15.2:0/413648035 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5d480a11a0 msgr2=0x7f5d480a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1793067160 2023-11-16T16:49:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:45.984+0000 7f5d5effd700 1 --2- 172.21.15.2:0/413648035 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5d480a11a0 0x7f5d480a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T16:49:46.008 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.007+0000 7f5d661ff700 1 -- 172.21.15.2:0/413648035 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f5d60060f30 con 0x7f5d60100d00 2023-11-16T16:49:46.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.014+0000 7f5d5cff9700 1 -- 172.21.15.2:0/413648035 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5d500999c0 con 0x7f5d60100d00 2023-11-16T16:49:46.019 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.018+0000 7f5d661ff700 1 -- 172.21.15.2:0/413648035 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5d480a11a0 msgr2=0x7f5d480a3650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:49:46.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.018+0000 7f5d661ff700 1 --2- 172.21.15.2:0/413648035 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5d480a11a0 0x7f5d480a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.018+0000 7f5d661ff700 1 -- 172.21.15.2:0/413648035 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d60100d00 msgr2=0x7f5d60100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.018+0000 7f5d661ff700 1 --2- 172.21.15.2:0/413648035 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d60100d00 0x7f5d60100670 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000e9f0 tx=0x7f5d5000ed00 comp rx=0 tx=0).stop 2023-11-16T16:49:46.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.018+0000 7f5d661ff700 1 -- 172.21.15.2:0/413648035 shutdown_connections 2023-11-16T16:49:46.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.018+0000 7f5d661ff700 1 --2- 172.21.15.2:0/413648035 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f5d480a11a0 0x7f5d480a3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.018+0000 7f5d661ff700 1 --2- 172.21.15.2:0/413648035 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d60105f40 0x7f5d600ff1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.018+0000 7f5d661ff700 1 --2- 172.21.15.2:0/413648035 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5d60100d00 0x7f5d60100670 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.018+0000 7f5d661ff700 1 --2- 172.21.15.2:0/413648035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d60103620 0x7f5d600fecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.018+0000 7f5d661ff700 1 -- 172.21.15.2:0/413648035 >> 172.21.15.2:0/413648035 conn(0x7f5d600fa7a0 msgr2=0x7f5d60101960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:46.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.018+0000 7f5d661ff700 1 -- 172.21.15.2:0/413648035 shutdown_connections 2023-11-16T16:49:46.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.018+0000 7f5d661ff700 1 -- 172.21.15.2:0/413648035 wait complete. 2023-11-16T16:49:46.039 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2023-11-16T16:49:46.102 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:45 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:45.821+0000 7fa655e1d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-16T16:49:46.102 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:45 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:45.937+0000 7fa655e1d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-16T16:49:46.102 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:46 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:46.064+0000 7fa655e1d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T16:49:46.103 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:45 smithi002 ceph-mon[128712]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:49:46.103 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:45 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3243044483' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-11-16T16:49:46.103 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:45 smithi002 ceph-mon[128712]: mgrmap e19: x(active, since 9m), standbys: z, y 2023-11-16T16:49:46.152 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:45 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:45.971+0000 7efe03059200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T16:49:46.152 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:45 smithi141 ceph-mon[131184]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:49:46.152 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:45 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3243044483' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-11-16T16:49:46.152 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:45 smithi141 ceph-mon[131184]: mgrmap e19: x(active, since 9m), standbys: z, y 2023-11-16T16:49:46.160 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:45 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:45.875+0000 7f4b8e3d1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-16T16:49:46.161 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:45 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:45.977+0000 7f4b8e3d1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T16:49:46.161 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:45 smithi155 ceph-mon[131607]: from='mgr.14150 172.21.15.2:0/1450143673' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:49:46.161 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:45 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3243044483' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-11-16T16:49:46.161 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:45 smithi155 ceph-mon[131607]: mgrmap e19: x(active, since 9m), standbys: z, y 2023-11-16T16:49:46.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.246+0000 7f31c37ff700 1 -- 172.21.15.2:0/3577464100 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31bc073550 msgr2=0x7f31bc0739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.246+0000 7f31c37ff700 1 --2- 172.21.15.2:0/3577464100 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31bc073550 0x7f31bc0739c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f31ac009990 tx=0x7f31ac009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.246+0000 7f31c37ff700 1 -- 172.21.15.2:0/3577464100 shutdown_connections 2023-11-16T16:49:46.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.246+0000 7f31c37ff700 1 --2- 172.21.15.2:0/3577464100 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31bc10c810 0x7f31bc10ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.248 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.246+0000 7f31c37ff700 1 --2- 172.21.15.2:0/3577464100 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31bc074b30 0x7f31bc072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.246+0000 7f31c37ff700 1 --2- 172.21.15.2:0/3577464100 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31bc073550 0x7f31bc0739c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.246+0000 7f31c37ff700 1 -- 172.21.15.2:0/3577464100 >> 172.21.15.2:0/3577464100 conn(0x7f31bc0fb840 msgr2=0x7f31bc0fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:46.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.246+0000 7f31c37ff700 1 -- 172.21.15.2:0/3577464100 shutdown_connections 2023-11-16T16:49:46.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.246+0000 7f31c37ff700 1 -- 172.21.15.2:0/3577464100 wait complete. 2023-11-16T16:49:46.249 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.248+0000 7f31c37ff700 1 Processor -- start 2023-11-16T16:49:46.250 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.248+0000 7f31c37ff700 1 -- start start 2023-11-16T16:49:46.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.249+0000 7f31c37ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31bc073550 0x7f31bc19c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.249+0000 7f31c37ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31bc074b30 0x7f31bc19cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.249+0000 7f31c37ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31bc10c810 0x7f31bc1a3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.249+0000 7f31c37ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc1a41f0 con 0x7f31bc073550 2023-11-16T16:49:46.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.249+0000 7f31c37ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc1a4330 con 0x7f31bc074b30 2023-11-16T16:49:46.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.249+0000 7f31c37ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc1a44c0 con 0x7f31bc10c810 2023-11-16T16:49:46.251 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.249+0000 7f31c0d9a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31bc074b30 0x7f31bc19cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.249+0000 7f31c159b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31bc073550 0x7f31bc19c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.249+0000 7f31c0d9a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31bc074b30 0x7f31bc19cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:55646/0 (socket says 172.21.15.2:55646) 2023-11-16T16:49:46.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.249+0000 7f31c0d9a700 1 -- 172.21.15.2:0/1358127327 learned_addr learned my addr 172.21.15.2:0/1358127327 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:46.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.249+0000 7f31c1d9c700 1 --2- 172.21.15.2:0/1358127327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31bc10c810 0x7f31bc1a3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.249+0000 7f31c0d9a700 1 -- 172.21.15.2:0/1358127327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31bc10c810 msgr2=0x7f31bc1a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.249+0000 7f31c0d9a700 1 --2- 172.21.15.2:0/1358127327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31bc10c810 0x7f31bc1a3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.249+0000 7f31c0d9a700 1 -- 172.21.15.2:0/1358127327 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31bc073550 msgr2=0x7f31bc19c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.249+0000 7f31c0d9a700 1 --2- 172.21.15.2:0/1358127327 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31bc073550 0x7f31bc19c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.249+0000 7f31c0d9a700 1 -- 172.21.15.2:0/1358127327 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ac0095f0 con 0x7f31bc074b30 2023-11-16T16:49:46.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.249+0000 7f31c159b700 1 --2- 172.21.15.2:0/1358127327 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31bc073550 0x7f31bc19c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:46.252 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.250+0000 7f31c0d9a700 1 --2- 172.21.15.2:0/1358127327 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31bc074b30 0x7f31bc19cbb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f31ac00b680 tx=0x7f31ac005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:46.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.250+0000 7f31b27fc700 1 -- 172.21.15.2:0/1358127327 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ac01d070 con 0x7f31bc074b30 2023-11-16T16:49:46.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.250+0000 7f31c37ff700 1 -- 172.21.15.2:0/1358127327 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31bc112220 con 0x7f31bc074b30 2023-11-16T16:49:46.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.250+0000 7f31b27fc700 1 -- 172.21.15.2:0/1358127327 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f31ac021410 con 0x7f31bc074b30 2023-11-16T16:49:46.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.250+0000 7f31b27fc700 1 -- 172.21.15.2:0/1358127327 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ac017650 con 0x7f31bc074b30 2023-11-16T16:49:46.253 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.250+0000 7f31c37ff700 1 -- 172.21.15.2:0/1358127327 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31bc112090 con 0x7f31bc074b30 2023-11-16T16:49:46.254 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.251+0000 7f31c37ff700 1 -- 172.21.15.2:0/1358127327 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31bc102360 con 0x7f31bc074b30 2023-11-16T16:49:46.256 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.255+0000 7f31b27fc700 1 -- 172.21.15.2:0/1358127327 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 19) v1 ==== 135731+0+0 (secure 0 0 0) 0x7f31ac021580 con 0x7f31bc074b30 2023-11-16T16:49:46.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.256+0000 7f31b27fc700 1 --2- 172.21.15.2:0/1358127327 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31a80a11a0 0x7f31a80a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.256+0000 7f31c159b700 1 -- 172.21.15.2:0/1358127327 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31a80a11a0 msgr2=0x7f31a80a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1793067160 2023-11-16T16:49:46.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.256+0000 7f31b27fc700 1 -- 172.21.15.2:0/1358127327 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f31ac0ce860 con 0x7f31bc074b30 2023-11-16T16:49:46.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.256+0000 7f31c159b700 1 --2- 172.21.15.2:0/1358127327 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31a80a11a0 0x7f31a80a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:49:46.258 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.257+0000 7f31b27fc700 1 -- 172.21.15.2:0/1358127327 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f31ac097c80 con 0x7f31bc074b30 2023-11-16T16:49:46.405 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:46 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:46.217+0000 7fa655e1d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-16T16:49:46.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.457+0000 7f31c159b700 1 -- 172.21.15.2:0/1358127327 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31a80a11a0 msgr2=0x7f31a80a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1793067160 2023-11-16T16:49:46.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.457+0000 7f31c159b700 1 --2- 172.21.15.2:0/1358127327 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31a80a11a0 0x7f31a80a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T16:49:46.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.476+0000 7f31c37ff700 1 -- 172.21.15.2:0/1358127327 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f31bc02cfe0 con 0x7f31bc074b30 2023-11-16T16:49:46.485 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.483+0000 7f31b27fc700 1 -- 172.21.15.2:0/1358127327 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7f31ac09b2a0 con 0x7f31bc074b30 2023-11-16T16:49:46.487 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:46 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:46.157+0000 7f4b8e3d1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T16:49:46.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.487+0000 7f31c37ff700 1 -- 172.21.15.2:0/1358127327 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31a80a11a0 msgr2=0x7f31a80a3650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:49:46.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.487+0000 7f31c37ff700 1 --2- 172.21.15.2:0/1358127327 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31a80a11a0 0x7f31a80a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.487+0000 7f31c37ff700 1 -- 172.21.15.2:0/1358127327 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31bc074b30 msgr2=0x7f31bc19cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.487+0000 7f31c37ff700 1 --2- 172.21.15.2:0/1358127327 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31bc074b30 0x7f31bc19cbb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f31ac00b680 tx=0x7f31ac005740 comp rx=0 tx=0).stop 2023-11-16T16:49:46.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.487+0000 7f31c37ff700 1 -- 172.21.15.2:0/1358127327 shutdown_connections 2023-11-16T16:49:46.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.487+0000 7f31c37ff700 1 --2- 172.21.15.2:0/1358127327 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7f31a80a11a0 0x7f31a80a3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.487+0000 7f31c37ff700 1 --2- 172.21.15.2:0/1358127327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f31bc10c810 0x7f31bc1a3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.489 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.487+0000 7f31c37ff700 1 --2- 172.21.15.2:0/1358127327 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31bc073550 0x7f31bc19c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.487+0000 7f31c37ff700 1 --2- 172.21.15.2:0/1358127327 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f31bc074b30 0x7f31bc19cbb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.487+0000 7f31c37ff700 1 -- 172.21.15.2:0/1358127327 >> 172.21.15.2:0/1358127327 conn(0x7f31bc0fb840 msgr2=0x7f31bc0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:46.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.487+0000 7f31c37ff700 1 -- 172.21.15.2:0/1358127327 shutdown_connections 2023-11-16T16:49:46.490 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.487+0000 7f31c37ff700 1 -- 172.21.15.2:0/1358127327 wait complete. 2023-11-16T16:49:46.511 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M' 2023-11-16T16:49:46.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.735+0000 7fafbddff700 1 -- 172.21.15.2:0/2748771873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fafb8100cc0 msgr2=0x7fafb81030a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.735+0000 7fafbddff700 1 --2- 172.21.15.2:0/2748771873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fafb8100cc0 0x7fafb81030a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009990 tx=0x7fafa8009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.737 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.735+0000 7fafbddff700 1 -- 172.21.15.2:0/2748771873 shutdown_connections 2023-11-16T16:49:46.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.735+0000 7fafbddff700 1 --2- 172.21.15.2:0/2748771873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fafb8105f00 0x7fafb8108360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.735+0000 7fafbddff700 1 --2- 172.21.15.2:0/2748771873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fafb8100cc0 0x7fafb81030a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.735+0000 7fafbddff700 1 --2- 172.21.15.2:0/2748771873 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fafb81035e0 0x7fafb81059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.735+0000 7fafbddff700 1 -- 172.21.15.2:0/2748771873 >> 172.21.15.2:0/2748771873 conn(0x7fafb80fa720 msgr2=0x7fafb80fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:46.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.735+0000 7fafbddff700 1 -- 172.21.15.2:0/2748771873 shutdown_connections 2023-11-16T16:49:46.738 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.735+0000 7fafbddff700 1 -- 172.21.15.2:0/2748771873 wait complete. 2023-11-16T16:49:46.739 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.736+0000 7fafbddff700 1 Processor -- start 2023-11-16T16:49:46.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.737+0000 7fafbddff700 1 -- start start 2023-11-16T16:49:46.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.737+0000 7fafbddff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fafb8100cc0 0x7fafb81981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.737+0000 7fafbddff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fafb81035e0 0x7fafb8198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.737+0000 7fafbddff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fafb8105f00 0x7fafb819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.737+0000 7fafbddff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb819fd50 con 0x7fafb8105f00 2023-11-16T16:49:46.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.737+0000 7fafbddff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb819fe90 con 0x7fafb8100cc0 2023-11-16T16:49:46.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.737+0000 7fafbddff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb81a0020 con 0x7fafb81035e0 2023-11-16T16:49:46.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.737+0000 7fafb77fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fafb8100cc0 0x7fafb81981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.737+0000 7fafb7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fafb8105f00 0x7fafb819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.737+0000 7fafb77fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fafb8100cc0 0x7fafb81981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:55660/0 (socket says 172.21.15.2:55660) 2023-11-16T16:49:46.740 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.737+0000 7fafb7fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fafb8105f00 0x7fafb819f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53552/0 (socket says 172.21.15.2:53552) 2023-11-16T16:49:46.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.737+0000 7fafb77fe700 1 -- 172.21.15.2:0/1560816219 learned_addr learned my addr 172.21.15.2:0/1560816219 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:46.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.737+0000 7fafb6ffd700 1 --2- 172.21.15.2:0/1560816219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fafb81035e0 0x7fafb8198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:46.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.738+0000 7fafb7fff700 1 -- 172.21.15.2:0/1560816219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fafb81035e0 msgr2=0x7fafb8198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.738+0000 7fafb7fff700 1 --2- 172.21.15.2:0/1560816219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fafb81035e0 0x7fafb8198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.738+0000 7fafb7fff700 1 -- 172.21.15.2:0/1560816219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fafb8100cc0 msgr2=0x7fafb81981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:46.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.738+0000 7fafb7fff700 1 --2- 172.21.15.2:0/1560816219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fafb8100cc0 0x7fafb81981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:46.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.738+0000 7fafb7fff700 1 -- 172.21.15.2:0/1560816219 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa80095f0 con 0x7fafb8105f00 2023-11-16T16:49:46.741 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.738+0000 7fafb7fff700 1 --2- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fafb8105f00 0x7fafb819f780 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fafb000eb80 tx=0x7fafb000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:46.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.738+0000 7fafb4ff9700 1 -- 172.21.15.2:0/1560816219 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb0018070 con 0x7fafb8105f00 2023-11-16T16:49:46.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.738+0000 7fafb4ff9700 1 -- 172.21.15.2:0/1560816219 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fafb000f040 con 0x7fafb8105f00 2023-11-16T16:49:46.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.738+0000 7fafbddff700 1 -- 172.21.15.2:0/1560816219 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb810bc10 con 0x7fafb8105f00 2023-11-16T16:49:46.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.738+0000 7fafb4ff9700 1 -- 172.21.15.2:0/1560816219 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb000cd50 con 0x7fafb8105f00 2023-11-16T16:49:46.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.739+0000 7fafbddff700 1 -- 172.21.15.2:0/1560816219 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb810ba80 con 0x7fafb8105f00 2023-11-16T16:49:46.742 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.740+0000 7fafbddff700 1 -- 172.21.15.2:0/1560816219 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb8107ab0 con 0x7fafb8105f00 2023-11-16T16:49:46.745 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.744+0000 7fafb4ff9700 1 -- 172.21.15.2:0/1560816219 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 135731+0+0 (secure 0 0 0) 0x7fafb0003a50 con 0x7fafb8105f00 2023-11-16T16:49:46.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.746+0000 7fafb4ff9700 1 --2- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fafa00a11a0 0x7fafa00a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:46.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.746+0000 7fafb77fe700 1 -- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fafa00a11a0 msgr2=0x7fafa00a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1793067160 2023-11-16T16:49:46.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.746+0000 7fafb4ff9700 1 -- 172.21.15.2:0/1560816219 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fafb00cd280 con 0x7fafb8105f00 2023-11-16T16:49:46.748 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.746+0000 7fafb77fe700 1 --2- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fafa00a11a0 0x7fafa00a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-16T16:49:46.749 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.748+0000 7fafb4ff9700 1 -- 172.21.15.2:0/1560816219 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fafb00966e0 con 0x7fafb8105f00 2023-11-16T16:49:46.764 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:46 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:46.149+0000 7efe03059200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T16:49:46.764 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:46 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:46.473+0000 7efe03059200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T16:49:46.764 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:46 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:46.570+0000 7efe03059200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T16:49:46.765 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:46 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:46.706+0000 7efe03059200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T16:49:46.820 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:46 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:46.485+0000 7f4b8e3d1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T16:49:46.820 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:46 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:46.576+0000 7f4b8e3d1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T16:49:46.820 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:46 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:46.716+0000 7f4b8e3d1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T16:49:46.905 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:46 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:46.476+0000 7fa655e1d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-16T16:49:46.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.947+0000 7fafb77fe700 1 -- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fafa00a11a0 msgr2=0x7fafa00a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1793067160 2023-11-16T16:49:46.948 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:46.947+0000 7fafb77fe700 1 --2- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fafa00a11a0 0x7fafa00a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-16T16:49:47.027 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:47.025+0000 7fafbddff700 1 -- 172.21.15.2:0/1560816219 --> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fafb8061a30 con 0x7fafa00a11a0 2023-11-16T16:49:47.117 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:46 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:46.818+0000 7f4b8e3d1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T16:49:47.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:47 smithi155 ceph-mon[131607]: from='client.? ' entity='client.admin' 2023-11-16T16:49:47.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:47 smithi155 ceph-mon[131607]: from='client.? ' entity='client.admin' 2023-11-16T16:49:47.252 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:46 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:46.935+0000 7fa655e1d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-16T16:49:47.252 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:47 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:47.060+0000 7fa655e1d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-16T16:49:47.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:47 smithi002 ceph-mon[128712]: from='client.? ' entity='client.admin' 2023-11-16T16:49:47.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:47 smithi002 ceph-mon[128712]: from='client.? ' entity='client.admin' 2023-11-16T16:49:47.260 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:46 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:46.803+0000 7efe03059200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T16:49:47.260 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:47 smithi141 ceph-mon[131184]: from='client.? ' entity='client.admin' 2023-11-16T16:49:47.261 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:47 smithi141 ceph-mon[131184]: from='client.? ' entity='client.admin' 2023-11-16T16:49:47.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:47.348+0000 7fafb77fe700 1 -- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fafa00a11a0 msgr2=0x7fafa00a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1793067160 2023-11-16T16:49:47.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:47.348+0000 7fafb77fe700 1 --2- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fafa00a11a0 0x7fafa00a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-16T16:49:47.514 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:47 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:47.259+0000 7efe03059200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T16:49:47.617 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:47 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:47.268+0000 7f4b8e3d1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T16:49:47.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:47 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:47.250+0000 7fa655e1d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-16T16:49:47.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:47 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:47.389+0000 7fa655e1d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-16T16:49:48.025 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:47 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:47.851+0000 7f4b8e3d1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T16:49:48.026 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:48 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:48.023+0000 7f4b8e3d1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T16:49:48.034 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:47 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:47.862+0000 7efe03059200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T16:49:48.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:48.149+0000 7fafb77fe700 1 -- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fafa00a11a0 msgr2=0x7fafa00a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1793067160 2023-11-16T16:49:48.150 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:48.149+0000 7fafb77fe700 1 --2- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fafa00a11a0 0x7fafa00a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-16T16:49:48.405 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:47 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:47.970+0000 7fa655e1d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-16T16:49:48.514 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:48 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:48.032+0000 7efe03059200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T16:49:49.052 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:48 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:48.792+0000 7efe03059200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:49:49.053 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:48 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:48.882+0000 7efe03059200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T16:49:49.055 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:48 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:48.790+0000 7f4b8e3d1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:49:49.055 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:48 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:48.881+0000 7f4b8e3d1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T16:49:49.155 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:48 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:48.812+0000 7fa655e1d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-16T16:49:49.155 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:49 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:49.028+0000 7fa655e1d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-16T16:49:49.326 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:49 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:49.051+0000 7efe03059200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:49:49.333 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:49 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:49.052+0000 7f4b8e3d1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:49:49.617 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:49 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:49.331+0000 7f4b8e3d1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T16:49:49.617 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:49 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:49.559+0000 7f4b8e3d1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T16:49:49.699 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:49 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:49.325+0000 7efe03059200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T16:49:49.699 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:49 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:49.552+0000 7efe03059200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T16:49:49.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:49.751+0000 7fafb77fe700 1 -- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fafa00a11a0 msgr2=0x7fafa00a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1793067160 2023-11-16T16:49:49.752 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:49.751+0000 7fafb77fe700 1 --2- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fafa00a11a0 0x7fafa00a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-16T16:49:49.958 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:49 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:49.697+0000 7efe03059200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T16:49:49.973 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:49 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:49.707+0000 7f4b8e3d1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T16:49:50.264 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:49 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:49.956+0000 7efe03059200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T16:49:50.367 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:49 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:49.971+0000 7f4b8e3d1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T16:49:50.389 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:50 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:50.024+0000 7fa655e1d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:49:50.389 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:50 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:50.150+0000 7fa655e1d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-16T16:49:50.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:50 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:50.387+0000 7fa655e1d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-16T16:49:50.974 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:50 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:50.654+0000 7efe03059200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T16:49:50.974 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:50 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:50.783+0000 7efe03059200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T16:49:50.999 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:50 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:50.676+0000 7f4b8e3d1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T16:49:50.999 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:50 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:50.807+0000 7f4b8e3d1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T16:49:51.057 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:50 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:50.768+0000 7fa655e1d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-16T16:49:51.264 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:50 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:50.972+0000 7efe03059200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T16:49:51.264 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:51 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:51.083+0000 7efe03059200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T16:49:51.328 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:50 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:50.997+0000 7f4b8e3d1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T16:49:51.328 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:51 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:51.112+0000 7f4b8e3d1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T16:49:51.405 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:51 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:51.055+0000 7fa655e1d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-16T16:49:51.405 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:51 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:51.258+0000 7fa655e1d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-16T16:49:51.617 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:51 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:51.326+0000 7f4b8e3d1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T16:49:51.764 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:51 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:51.297+0000 7efe03059200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T16:49:51.905 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:51 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:51.619+0000 7fa655e1d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-16T16:49:52.225 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:52.223+0000 7fafb4ff9700 1 -- 172.21.15.2:0/1560816219 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 20) v1 ==== 135731+0+0 (secure 0 0 0) 0x7fafb0096ab0 con 0x7fafb8105f00 2023-11-16T16:49:52.514 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 16:49:52 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y[132650]: 2023-11-16T16:49:52.149+0000 7efe03059200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T16:49:52.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:52 smithi141 ceph-mon[131184]: Standby manager daemon y restarted 2023-11-16T16:49:52.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:52 smithi141 ceph-mon[131184]: Standby manager daemon y started 2023-11-16T16:49:52.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:52 smithi141 ceph-mon[131184]: from='mgr.? 172.21.15.141:0/1649738418' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T16:49:52.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:52 smithi141 ceph-mon[131184]: from='mgr.? 172.21.15.141:0/1649738418' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:49:52.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:52 smithi141 ceph-mon[131184]: from='mgr.? 172.21.15.141:0/1649738418' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T16:49:52.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:52 smithi141 ceph-mon[131184]: from='mgr.? 172.21.15.141:0/1649738418' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:49:52.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:52 smithi141 ceph-mon[131184]: Standby manager daemon z restarted 2023-11-16T16:49:52.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:52 smithi141 ceph-mon[131184]: Standby manager daemon z started 2023-11-16T16:49:52.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:52 smithi141 ceph-mon[131184]: from='mgr.? 172.21.15.155:0/193886742' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T16:49:52.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:52 smithi141 ceph-mon[131184]: from='mgr.? 172.21.15.155:0/193886742' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:49:52.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:52 smithi141 ceph-mon[131184]: from='mgr.? 172.21.15.155:0/193886742' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T16:49:52.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:52 smithi141 ceph-mon[131184]: from='mgr.? 172.21.15.155:0/193886742' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:49:52.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:52 smithi002 ceph-mon[128712]: Standby manager daemon y restarted 2023-11-16T16:49:52.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:52 smithi002 ceph-mon[128712]: Standby manager daemon y started 2023-11-16T16:49:52.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:52 smithi002 ceph-mon[128712]: from='mgr.? 172.21.15.141:0/1649738418' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T16:49:52.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:52 smithi002 ceph-mon[128712]: from='mgr.? 172.21.15.141:0/1649738418' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:49:52.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:52 smithi002 ceph-mon[128712]: from='mgr.? 172.21.15.141:0/1649738418' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T16:49:52.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:52 smithi002 ceph-mon[128712]: from='mgr.? 172.21.15.141:0/1649738418' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:49:52.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:52 smithi002 ceph-mon[128712]: Standby manager daemon z restarted 2023-11-16T16:49:52.608 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:52 smithi002 ceph-mon[128712]: Standby manager daemon z started 2023-11-16T16:49:52.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:52 smithi002 ceph-mon[128712]: from='mgr.? 172.21.15.155:0/193886742' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T16:49:52.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:52 smithi002 ceph-mon[128712]: from='mgr.? 172.21.15.155:0/193886742' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:49:52.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:52 smithi002 ceph-mon[128712]: from='mgr.? 172.21.15.155:0/193886742' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T16:49:52.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:52 smithi002 ceph-mon[128712]: from='mgr.? 172.21.15.155:0/193886742' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:49:52.617 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 16:49:52 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z[133724]: 2023-11-16T16:49:52.185+0000 7f4b8e3d1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T16:49:52.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:52 smithi155 ceph-mon[131607]: Standby manager daemon y restarted 2023-11-16T16:49:52.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:52 smithi155 ceph-mon[131607]: Standby manager daemon y started 2023-11-16T16:49:52.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:52 smithi155 ceph-mon[131607]: from='mgr.? 172.21.15.141:0/1649738418' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-11-16T16:49:52.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:52 smithi155 ceph-mon[131607]: from='mgr.? 172.21.15.141:0/1649738418' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:49:52.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:52 smithi155 ceph-mon[131607]: from='mgr.? 172.21.15.141:0/1649738418' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-11-16T16:49:52.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:52 smithi155 ceph-mon[131607]: from='mgr.? 172.21.15.141:0/1649738418' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:49:52.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:52 smithi155 ceph-mon[131607]: Standby manager daemon z restarted 2023-11-16T16:49:52.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:52 smithi155 ceph-mon[131607]: Standby manager daemon z started 2023-11-16T16:49:52.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:52 smithi155 ceph-mon[131607]: from='mgr.? 172.21.15.155:0/193886742' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-11-16T16:49:52.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:52 smithi155 ceph-mon[131607]: from='mgr.? 172.21.15.155:0/193886742' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-16T16:49:52.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:52 smithi155 ceph-mon[131607]: from='mgr.? 172.21.15.155:0/193886742' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-11-16T16:49:52.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:52 smithi155 ceph-mon[131607]: from='mgr.? 172.21.15.155:0/193886742' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-16T16:49:52.905 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:52 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:52.606+0000 7fa655e1d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-16T16:49:52.905 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:52 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:52.788+0000 7fa655e1d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-16T16:49:52.955 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:52.954+0000 7fafb77fe700 1 -- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fafa00a11a0 msgr2=0x7fafa00a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1793067160 2023-11-16T16:49:52.956 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:52.954+0000 7fafb77fe700 1 --2- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fafa00a11a0 0x7fafa00a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-16T16:49:53.405 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:53 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:53.049+0000 7fa655e1d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-16T16:49:53.405 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:53 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:53.206+0000 7fa655e1d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-16T16:49:53.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:53 smithi002 ceph-mon[128712]: mgrmap e20: x(active, since 9m), standbys: z, y 2023-11-16T16:49:53.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:53 smithi141 ceph-mon[131184]: mgrmap e20: x(active, since 9m), standbys: z, y 2023-11-16T16:49:53.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:53 smithi155 ceph-mon[131607]: mgrmap e20: x(active, since 9m), standbys: z, y 2023-11-16T16:49:53.905 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:53 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:53.502+0000 7fa655e1d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-16T16:49:54.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:54.637+0000 7fafb4ff9700 1 -- 172.21.15.2:0/1560816219 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mgrmap(e 21) v1 ==== 135400+0+0 (secure 0 0 0) 0x7fafb00caa80 con 0x7fafb8105f00 2023-11-16T16:49:54.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:54.638+0000 7fafb4ff9700 1 -- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fafa00a11a0 msgr2=0x7fafa00a3650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:49:54.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:54.638+0000 7fafb4ff9700 1 --2- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1793067160,v1:172.21.15.2:6801/1793067160] conn(0x7fafa00a11a0 0x7fafa00a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:54.905 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:54 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:54.623+0000 7fa655e1d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-16T16:49:54.906 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:54 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:54.868+0000 7fa5b1a74700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:54.906 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:54 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:54.868+0000 7fa5b1a74700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:54.906 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:54 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:54.868+0000 7fa5b1a74700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:54.906 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:54 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:54.868+0000 7fa5b1a74700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:54.906 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:54 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:54.868+0000 7fa5b1a74700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:54.906 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:54 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:54.869+0000 7fa5b3ab8700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:54.906 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:54 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:54.869+0000 7fa5b3ab8700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:54.906 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:54 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:54.869+0000 7fa5b3ab8700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:54.906 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:54 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:54.869+0000 7fa5b3ab8700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:54.906 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:49:54 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:49:54.869+0000 7fa5b3ab8700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:49:55.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.640+0000 7fafb4ff9700 1 -- 172.21.15.2:0/1560816219 <== mon.0 v2:172.21.15.2:3300/0 9 ==== mgrmap(e 22) v1 ==== 135527+0+0 (secure 0 0 0) 0x7fafb00ba0c0 con 0x7fafb8105f00 2023-11-16T16:49:55.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.642+0000 7fafb4ff9700 1 --2- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fafa001b4a0 0x7fafa001b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:55.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.642+0000 7fafb4ff9700 1 -- 172.21.15.2:0/1560816219 --> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fafb8061a30 con 0x7fafa001b4a0 2023-11-16T16:49:55.647 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.645+0000 7fafb77fe700 1 --2- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fafa001b4a0 0x7fafa001b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.653+0000 7fafb77fe700 1 --2- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fafa001b4a0 0x7fafa001b870 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fafa800b2e0 tx=0x7fafa8005fb0 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:55.719 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.718+0000 7fafb4ff9700 1 -- 172.21.15.2:0/1560816219 <== mgr.15372 v2:172.21.15.2:6800/1352003968 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fafb8061a30 con 0x7fafa001b4a0 2023-11-16T16:49:55.720 INFO:teuthology.orchestra.run.smithi002.stdout:Schedule set for path / 2023-11-16T16:49:55.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.720+0000 7fafbddff700 1 -- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fafa001b4a0 msgr2=0x7fafa001b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:55.722 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.720+0000 7fafbddff700 1 --2- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fafa001b4a0 0x7fafa001b870 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fafa800b2e0 tx=0x7fafa8005fb0 comp rx=0 tx=0).stop 2023-11-16T16:49:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.720+0000 7fafbddff700 1 -- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fafb8105f00 msgr2=0x7fafb819f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.720+0000 7fafbddff700 1 --2- 172.21.15.2:0/1560816219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fafb8105f00 0x7fafb819f780 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fafb000eb80 tx=0x7fafb000ee90 comp rx=0 tx=0).stop 2023-11-16T16:49:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.720+0000 7fafb7fff700 1 -- 172.21.15.2:0/1560816219 reap_dead start 2023-11-16T16:49:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.721+0000 7fafbddff700 1 -- 172.21.15.2:0/1560816219 shutdown_connections 2023-11-16T16:49:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.721+0000 7fafbddff700 1 -- 172.21.15.2:0/1560816219 >> 172.21.15.2:0/1560816219 conn(0x7fafb80fa720 msgr2=0x7fafb80ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.721+0000 7fafbddff700 1 -- 172.21.15.2:0/1560816219 shutdown_connections 2023-11-16T16:49:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.721+0000 7fafbddff700 1 -- 172.21.15.2:0/1560816219 wait complete. 2023-11-16T16:49:55.746 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h' 2023-11-16T16:49:55.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: Active manager daemon x restarted 2023-11-16T16:49:55.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: Activating manager daemon x 2023-11-16T16:49:55.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: osdmap e94: 12 total, 12 up, 12 in 2023-11-16T16:49:55.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: mgrmap e21: x(active, starting, since 0.0140916s), standbys: z, y 2023-11-16T16:49:55.765 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:49:55.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:49:55.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:49:55.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T16:49:55.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T16:49:55.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T16:49:55.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T16:49:55.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T16:49:55.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T16:49:55.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T16:49:55.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T16:49:55.766 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T16:49:55.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T16:49:55.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T16:49:55.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T16:49:55.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T16:49:55.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T16:49:55.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T16:49:55.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:49:55.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:49:55.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:49:55.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:49:55.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:49:55.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:49:55.767 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:49:55.768 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:49:55.768 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:49:55.768 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:49:55.768 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:49:55.768 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:49:55.768 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T16:49:55.768 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T16:49:55.768 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T16:49:55.768 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: Manager daemon x is now available 2023-11-16T16:49:55.768 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:49:55.768 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:49:55.768 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T16:49:55.769 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:55 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T16:49:55.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: Active manager daemon x restarted 2023-11-16T16:49:55.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: Activating manager daemon x 2023-11-16T16:49:55.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: osdmap e94: 12 total, 12 up, 12 in 2023-11-16T16:49:55.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: mgrmap e21: x(active, starting, since 0.0140916s), standbys: z, y 2023-11-16T16:49:55.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:49:55.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:49:55.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:49:55.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T16:49:55.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T16:49:55.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T16:49:55.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T16:49:55.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T16:49:55.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T16:49:55.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T16:49:55.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T16:49:55.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T16:49:55.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T16:49:55.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T16:49:55.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T16:49:55.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T16:49:55.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T16:49:55.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T16:49:55.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:49:55.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:49:55.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:49:55.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:49:55.814 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:49:55.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:49:55.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:49:55.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:49:55.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:49:55.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:49:55.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:49:55.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:49:55.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T16:49:55.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T16:49:55.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T16:49:55.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: Manager daemon x is now available 2023-11-16T16:49:55.815 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:49:55.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:49:55.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T16:49:55.816 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:55 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T16:49:55.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: Active manager daemon x restarted 2023-11-16T16:49:55.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: Activating manager daemon x 2023-11-16T16:49:55.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: osdmap e94: 12 total, 12 up, 12 in 2023-11-16T16:49:55.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: mgrmap e21: x(active, starting, since 0.0140916s), standbys: z, y 2023-11-16T16:49:55.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-16T16:49:55.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-16T16:49:55.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-16T16:49:55.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-11-16T16:49:55.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-11-16T16:49:55.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-11-16T16:49:55.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-11-16T16:49:55.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-11-16T16:49:55.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-11-16T16:49:55.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T16:49:55.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T16:49:55.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T16:49:55.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T16:49:55.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T16:49:55.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T16:49:55.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-16T16:49:55.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-11-16T16:49:55.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-16T16:49:55.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-16T16:49:55.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-16T16:49:55.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-16T16:49:55.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-16T16:49:55.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-16T16:49:55.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-16T16:49:55.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-16T16:49:55.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-16T16:49:55.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-11-16T16:49:55.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-11-16T16:49:55.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-11-16T16:49:55.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-11-16T16:49:55.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-16T16:49:55.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-16T16:49:55.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-16T16:49:55.870 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: Manager daemon x is now available 2023-11-16T16:49:55.871 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:49:55.871 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:49:55.871 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-11-16T16:49:55.871 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:55 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-11-16T16:49:55.960 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.958+0000 7f93821ff700 1 -- 172.21.15.2:0/573836301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f937c1002c0 msgr2=0x7f937c1006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:55.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.958+0000 7f93821ff700 1 --2- 172.21.15.2:0/573836301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f937c1002c0 0x7f937c1006d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f936c009990 tx=0x7f936c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:55.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.958+0000 7f93821ff700 1 -- 172.21.15.2:0/573836301 shutdown_connections 2023-11-16T16:49:55.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.958+0000 7f93821ff700 1 --2- 172.21.15.2:0/573836301 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f937c101e40 0x7f937c108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:55.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.958+0000 7f93821ff700 1 --2- 172.21.15.2:0/573836301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f937c1002c0 0x7f937c1006d0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:55.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.958+0000 7f93821ff700 1 --2- 172.21.15.2:0/573836301 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f937c1014b0 0x7f937c101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:55.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.958+0000 7f93821ff700 1 -- 172.21.15.2:0/573836301 >> 172.21.15.2:0/573836301 conn(0x7f937c0fb8e0 msgr2=0x7f937c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:55.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.959+0000 7f93821ff700 1 -- 172.21.15.2:0/573836301 shutdown_connections 2023-11-16T16:49:55.961 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.959+0000 7f93821ff700 1 -- 172.21.15.2:0/573836301 wait complete. 2023-11-16T16:49:55.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.960+0000 7f93821ff700 1 Processor -- start 2023-11-16T16:49:55.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.960+0000 7f93821ff700 1 -- start start 2023-11-16T16:49:55.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.960+0000 7f93821ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f937c1002c0 0x7f937c1981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:55.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.960+0000 7f93821ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f937c1014b0 0x7f937c198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:55.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.960+0000 7f93821ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f937c101e40 0x7f937c19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:55.963 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.960+0000 7f93821ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937c19fd40 con 0x7f937c101e40 2023-11-16T16:49:55.964 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.960+0000 7f93821ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937c19fe80 con 0x7f937c1014b0 2023-11-16T16:49:55.964 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.960+0000 7f93821ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937c1a0010 con 0x7f937c1002c0 2023-11-16T16:49:55.964 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.960+0000 7f937bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f937c101e40 0x7f937c19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:55.964 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.960+0000 7f937b7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f937c1002c0 0x7f937c1981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:55.964 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.960+0000 7f937bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f937c101e40 0x7f937c19f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60586/0 (socket says 172.21.15.2:60586) 2023-11-16T16:49:55.964 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.960+0000 7f937bfff700 1 -- 172.21.15.2:0/3849691983 learned_addr learned my addr 172.21.15.2:0/3849691983 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:55.964 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.960+0000 7f937affd700 1 --2- 172.21.15.2:0/3849691983 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f937c1014b0 0x7f937c198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:55.964 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.961+0000 7f937b7fe700 1 -- 172.21.15.2:0/3849691983 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f937c1014b0 msgr2=0x7f937c198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:55.964 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.961+0000 7f937b7fe700 1 --2- 172.21.15.2:0/3849691983 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f937c1014b0 0x7f937c198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:55.964 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.961+0000 7f937b7fe700 1 -- 172.21.15.2:0/3849691983 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f937c101e40 msgr2=0x7f937c19f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:55.964 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.961+0000 7f937b7fe700 1 --2- 172.21.15.2:0/3849691983 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f937c101e40 0x7f937c19f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:55.964 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.961+0000 7f937b7fe700 1 -- 172.21.15.2:0/3849691983 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936c0095f0 con 0x7f937c1002c0 2023-11-16T16:49:55.964 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.961+0000 7f937b7fe700 1 --2- 172.21.15.2:0/3849691983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f937c1002c0 0x7f937c1981c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f936c00b620 tx=0x7f936c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:55.965 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.961+0000 7f9378ff9700 1 -- 172.21.15.2:0/3849691983 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c01d070 con 0x7f937c1002c0 2023-11-16T16:49:55.965 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.961+0000 7f9378ff9700 1 -- 172.21.15.2:0/3849691983 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f936c022470 con 0x7f937c1002c0 2023-11-16T16:49:55.965 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.961+0000 7f93821ff700 1 -- 172.21.15.2:0/3849691983 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f937c10bbc0 con 0x7f937c1002c0 2023-11-16T16:49:55.965 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.961+0000 7f9378ff9700 1 -- 172.21.15.2:0/3849691983 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c017650 con 0x7f937c1002c0 2023-11-16T16:49:55.965 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.962+0000 7f93821ff700 1 -- 172.21.15.2:0/3849691983 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f937c10ba30 con 0x7f937c1002c0 2023-11-16T16:49:55.965 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.963+0000 7f93821ff700 1 -- 172.21.15.2:0/3849691983 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f937c10bd50 con 0x7f937c1002c0 2023-11-16T16:49:55.967 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.965+0000 7f9378ff9700 1 -- 172.21.15.2:0/3849691983 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 22) v1 ==== 135527+0+0 (secure 0 0 0) 0x7f936c0225e0 con 0x7f937c1002c0 2023-11-16T16:49:55.968 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.967+0000 7f9378ff9700 1 --2- 172.21.15.2:0/3849691983 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f93640a0f50 0x7f93640a3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:55.968 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.967+0000 7f9378ff9700 1 -- 172.21.15.2:0/3849691983 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f936c0ce490 con 0x7f937c1002c0 2023-11-16T16:49:55.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.967+0000 7f937affd700 1 --2- 172.21.15.2:0/3849691983 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f93640a0f50 0x7f93640a3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:55.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.967+0000 7f937affd700 1 --2- 172.21.15.2:0/3849691983 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f93640a0f50 0x7f93640a3400 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9370009c60 tx=0x7f9370009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:55.969 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:55.968+0000 7f9378ff9700 1 -- 172.21.15.2:0/3849691983 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f936c097b60 con 0x7f937c1002c0 2023-11-16T16:49:56.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.213+0000 7f93821ff700 1 -- 172.21.15.2:0/3849691983 --> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}) v1 -- 0x7f937c061a50 con 0x7f93640a0f50 2023-11-16T16:49:56.276 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.274+0000 7f9378ff9700 1 -- 172.21.15.2:0/3849691983 <== mgr.15372 v2:172.21.15.2:6800/1352003968 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f937c061a50 con 0x7f93640a0f50 2023-11-16T16:49:56.276 INFO:teuthology.orchestra.run.smithi002.stdout:Retention added to path / 2023-11-16T16:49:56.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.277+0000 7f93821ff700 1 -- 172.21.15.2:0/3849691983 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f93640a0f50 msgr2=0x7f93640a3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:56.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.277+0000 7f93821ff700 1 --2- 172.21.15.2:0/3849691983 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f93640a0f50 0x7f93640a3400 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9370009c60 tx=0x7f9370009260 comp rx=0 tx=0).stop 2023-11-16T16:49:56.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.277+0000 7f93821ff700 1 -- 172.21.15.2:0/3849691983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f937c1002c0 msgr2=0x7f937c1981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:56.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.277+0000 7f93821ff700 1 --2- 172.21.15.2:0/3849691983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f937c1002c0 0x7f937c1981c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f936c00b620 tx=0x7f936c004730 comp rx=0 tx=0).stop 2023-11-16T16:49:56.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.277+0000 7f93821ff700 1 -- 172.21.15.2:0/3849691983 shutdown_connections 2023-11-16T16:49:56.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.277+0000 7f93821ff700 1 --2- 172.21.15.2:0/3849691983 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f93640a0f50 0x7f93640a3400 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.277+0000 7f93821ff700 1 --2- 172.21.15.2:0/3849691983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f937c1002c0 0x7f937c1981c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.277+0000 7f93821ff700 1 --2- 172.21.15.2:0/3849691983 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f937c101e40 0x7f937c19f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.277+0000 7f93821ff700 1 --2- 172.21.15.2:0/3849691983 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f937c1014b0 0x7f937c198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.277+0000 7f93821ff700 1 -- 172.21.15.2:0/3849691983 >> 172.21.15.2:0/3849691983 conn(0x7f937c0fb8e0 msgr2=0x7f937c106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:56.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.277+0000 7f93821ff700 1 -- 172.21.15.2:0/3849691983 shutdown_connections 2023-11-16T16:49:56.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.277+0000 7f93821ff700 1 -- 172.21.15.2:0/3849691983 wait complete. 2023-11-16T16:49:56.299 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2023-11-16T16:49:56.510 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.508+0000 7f8dea7ff700 1 -- 172.21.15.2:0/2844718333 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8de4074ad0 msgr2=0x7f8de4072f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:56.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.508+0000 7f8dea7ff700 1 --2- 172.21.15.2:0/2844718333 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8de4074ad0 0x7f8de4072f20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4009990 tx=0x7f8dd4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.508+0000 7f8dea7ff700 1 -- 172.21.15.2:0/2844718333 shutdown_connections 2023-11-16T16:49:56.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.508+0000 7f8dea7ff700 1 --2- 172.21.15.2:0/2844718333 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8de4074ad0 0x7f8de4072f20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.508+0000 7f8dea7ff700 1 --2- 172.21.15.2:0/2844718333 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8de41080d0 0x7f8de410a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.508+0000 7f8dea7ff700 1 --2- 172.21.15.2:0/2844718333 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8de40734f0 0x7f8de4073960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.508+0000 7f8dea7ff700 1 -- 172.21.15.2:0/2844718333 >> 172.21.15.2:0/2844718333 conn(0x7f8de40fb840 msgr2=0x7f8de40fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:56.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.509+0000 7f8dea7ff700 1 -- 172.21.15.2:0/2844718333 shutdown_connections 2023-11-16T16:49:56.511 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.509+0000 7f8dea7ff700 1 -- 172.21.15.2:0/2844718333 wait complete. 2023-11-16T16:49:56.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.510+0000 7f8dea7ff700 1 Processor -- start 2023-11-16T16:49:56.512 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.510+0000 7f8dea7ff700 1 -- start start 2023-11-16T16:49:56.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.511+0000 7f8dea7ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8de40734f0 0x7f8de41981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:56.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.511+0000 7f8dea7ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8de4074ad0 0x7f8de4198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:56.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.511+0000 7f8dea7ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8de41080d0 0x7f8de419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:56.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.511+0000 7f8dea7ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de419fd40 con 0x7f8de40734f0 2023-11-16T16:49:56.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.511+0000 7f8dea7ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de419fe80 con 0x7f8de41080d0 2023-11-16T16:49:56.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.511+0000 7f8dea7ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de41a0010 con 0x7f8de4074ad0 2023-11-16T16:49:56.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.511+0000 7f8de37fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8de4074ad0 0x7f8de4198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:56.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.511+0000 7f8de3fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8de40734f0 0x7f8de41981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:56.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.511+0000 7f8de37fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8de4074ad0 0x7f8de4198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:36592/0 (socket says 172.21.15.2:36592) 2023-11-16T16:49:56.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.511+0000 7f8de37fe700 1 -- 172.21.15.2:0/2682044467 learned_addr learned my addr 172.21.15.2:0/2682044467 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:56.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.511+0000 7f8de8d9c700 1 --2- 172.21.15.2:0/2682044467 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8de41080d0 0x7f8de419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:56.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.512+0000 7f8de37fe700 1 -- 172.21.15.2:0/2682044467 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8de41080d0 msgr2=0x7f8de419f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:56.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.512+0000 7f8de37fe700 1 --2- 172.21.15.2:0/2682044467 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8de41080d0 0x7f8de419f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.512+0000 7f8de37fe700 1 -- 172.21.15.2:0/2682044467 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8de40734f0 msgr2=0x7f8de41981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:56.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.512+0000 7f8de37fe700 1 --2- 172.21.15.2:0/2682044467 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8de40734f0 0x7f8de41981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.512+0000 7f8de37fe700 1 -- 172.21.15.2:0/2682044467 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd8009520 con 0x7f8de4074ad0 2023-11-16T16:49:56.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.512+0000 7f8de8d9c700 1 --2- 172.21.15.2:0/2682044467 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8de41080d0 0x7f8de419f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:56.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.512+0000 7f8de3fff700 1 --2- 172.21.15.2:0/2682044467 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8de40734f0 0x7f8de41981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:49:56.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.512+0000 7f8de37fe700 1 --2- 172.21.15.2:0/2682044467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8de4074ad0 0x7f8de4198700 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8dd800eb30 tx=0x7f8dd800ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:56.515 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.512+0000 7f8de17fa700 1 -- 172.21.15.2:0/2682044467 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd8009880 con 0x7f8de4074ad0 2023-11-16T16:49:56.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.512+0000 7f8de17fa700 1 -- 172.21.15.2:0/2682044467 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8dd80044e0 con 0x7f8de4074ad0 2023-11-16T16:49:56.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.512+0000 7f8dea7ff700 1 -- 172.21.15.2:0/2682044467 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ddc009c40 con 0x7f8de4074ad0 2023-11-16T16:49:56.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.512+0000 7f8de17fa700 1 -- 172.21.15.2:0/2682044467 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd8003d80 con 0x7f8de4074ad0 2023-11-16T16:49:56.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.512+0000 7f8dea7ff700 1 -- 172.21.15.2:0/2682044467 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd40095f0 con 0x7f8de4074ad0 2023-11-16T16:49:56.516 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.514+0000 7f8dea7ff700 1 -- 172.21.15.2:0/2682044467 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de4106810 con 0x7f8de4074ad0 2023-11-16T16:49:56.517 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.516+0000 7f8de17fa700 1 -- 172.21.15.2:0/2682044467 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 22) v1 ==== 135527+0+0 (secure 0 0 0) 0x7f8dd8003ee0 con 0x7f8de4074ad0 2023-11-16T16:49:56.518 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.517+0000 7f8de17fa700 1 --2- 172.21.15.2:0/2682044467 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f8dcc0a0f50 0x7f8dcc0a3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:56.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.517+0000 7f8de17fa700 1 -- 172.21.15.2:0/2682044467 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f8dd8014070 con 0x7f8de4074ad0 2023-11-16T16:49:56.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.517+0000 7f8de3fff700 1 --2- 172.21.15.2:0/2682044467 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f8dcc0a0f50 0x7f8dcc0a3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:56.519 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.518+0000 7f8de3fff700 1 --2- 172.21.15.2:0/2682044467 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f8dcc0a0f50 0x7f8dcc0a3400 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8dd400b2e0 tx=0x7f8dd4005fb0 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:56.523 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.521+0000 7f8de17fa700 1 -- 172.21.15.2:0/2682044467 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8dd8096320 con 0x7f8de4074ad0 2023-11-16T16:49:56.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.652+0000 7f8de17fa700 1 -- 172.21.15.2:0/2682044467 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mgrmap(e 23) v1 ==== 135890+0+0 (secure 0 0 0) 0x7f8dd80966f0 con 0x7f8de4074ad0 2023-11-16T16:49:56.786 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.784+0000 7f8dea7ff700 1 -- 172.21.15.2:0/2682044467 --> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f8de4060f30 con 0x7f8dcc0a0f50 2023-11-16T16:49:56.794 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.793+0000 7f8de17fa700 1 -- 172.21.15.2:0/2682044467 <== mgr.15372 v2:172.21.15.2:6800/1352003968 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+283 (secure 0 0 0) 0x7f8de4060f30 con 0x7f8dcc0a0f50 2023-11-16T16:49:56.794 INFO:teuthology.orchestra.run.smithi002.stdout:{"fs": "cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "1M", "retention": {"h": 3, "M": 6}, "start": "2023-11-16T00:00:00", "created": "2023-11-16T16:49:55", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2023-11-16T16:49:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.795+0000 7f8dea7ff700 1 -- 172.21.15.2:0/2682044467 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f8dcc0a0f50 msgr2=0x7f8dcc0a3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.795+0000 7f8dea7ff700 1 --2- 172.21.15.2:0/2682044467 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f8dcc0a0f50 0x7f8dcc0a3400 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8dd400b2e0 tx=0x7f8dd4005fb0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.795+0000 7f8dea7ff700 1 -- 172.21.15.2:0/2682044467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8de4074ad0 msgr2=0x7f8de4198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.795+0000 7f8dea7ff700 1 --2- 172.21.15.2:0/2682044467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8de4074ad0 0x7f8de4198700 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8dd800eb30 tx=0x7f8dd800ee40 comp rx=0 tx=0).stop 2023-11-16T16:49:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.795+0000 7f8dea7ff700 1 -- 172.21.15.2:0/2682044467 shutdown_connections 2023-11-16T16:49:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.795+0000 7f8dea7ff700 1 --2- 172.21.15.2:0/2682044467 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f8dcc0a0f50 0x7f8dcc0a3400 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.795+0000 7f8dea7ff700 1 --2- 172.21.15.2:0/2682044467 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f8de41080d0 0x7f8de419f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.795+0000 7f8dea7ff700 1 --2- 172.21.15.2:0/2682044467 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8de40734f0 0x7f8de41981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.795+0000 7f8dea7ff700 1 --2- 172.21.15.2:0/2682044467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8de4074ad0 0x7f8de4198700 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.795+0000 7f8dea7ff700 1 -- 172.21.15.2:0/2682044467 >> 172.21.15.2:0/2682044467 conn(0x7f8de40fb840 msgr2=0x7f8de40fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.795+0000 7f8dea7ff700 1 -- 172.21.15.2:0/2682044467 shutdown_connections 2023-11-16T16:49:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:56.795+0000 7f8dea7ff700 1 -- 172.21.15.2:0/2682044467 wait complete. 2023-11-16T16:49:56.817 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2023-11-16T16:49:56.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:56 smithi002 ceph-mon[128712]: [16/Nov/2023:16:49:55] ENGINE Bus STARTING 2023-11-16T16:49:56.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:56 smithi002 ceph-mon[128712]: [16/Nov/2023:16:49:55] ENGINE Serving on https://172.21.15.2:7150 2023-11-16T16:49:56.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:56 smithi002 ceph-mon[128712]: [16/Nov/2023:16:49:55] ENGINE Serving on http://172.21.15.2:8765 2023-11-16T16:49:56.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:56 smithi002 ceph-mon[128712]: [16/Nov/2023:16:49:55] ENGINE Bus STARTED 2023-11-16T16:49:56.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:56 smithi002 ceph-mon[128712]: mgrmap e22: x(active, since 1.01993s), standbys: z, y 2023-11-16T16:49:57.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:56 smithi141 ceph-mon[131184]: [16/Nov/2023:16:49:55] ENGINE Bus STARTING 2023-11-16T16:49:57.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:56 smithi141 ceph-mon[131184]: [16/Nov/2023:16:49:55] ENGINE Serving on https://172.21.15.2:7150 2023-11-16T16:49:57.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:56 smithi141 ceph-mon[131184]: [16/Nov/2023:16:49:55] ENGINE Serving on http://172.21.15.2:8765 2023-11-16T16:49:57.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:56 smithi141 ceph-mon[131184]: [16/Nov/2023:16:49:55] ENGINE Bus STARTED 2023-11-16T16:49:57.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:56 smithi141 ceph-mon[131184]: mgrmap e22: x(active, since 1.01993s), standbys: z, y 2023-11-16T16:49:57.032 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.031+0000 7f03345ff700 1 -- 172.21.15.2:0/1527320905 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f032c073550 msgr2=0x7f032c0739c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:57.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.031+0000 7f03345ff700 1 --2- 172.21.15.2:0/1527320905 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f032c073550 0x7f032c0739c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f031c0098f0 tx=0x7f031c009c00 comp rx=0 tx=0).stop 2023-11-16T16:49:57.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.031+0000 7f03345ff700 1 -- 172.21.15.2:0/1527320905 shutdown_connections 2023-11-16T16:49:57.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.031+0000 7f03345ff700 1 --2- 172.21.15.2:0/1527320905 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f032c10c810 0x7f032c10ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.031+0000 7f03345ff700 1 --2- 172.21.15.2:0/1527320905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f032c074b30 0x7f032c072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.031+0000 7f03345ff700 1 --2- 172.21.15.2:0/1527320905 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f032c073550 0x7f032c0739c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.031+0000 7f03345ff700 1 -- 172.21.15.2:0/1527320905 >> 172.21.15.2:0/1527320905 conn(0x7f032c0fb840 msgr2=0x7f032c0fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:57.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.031+0000 7f03345ff700 1 -- 172.21.15.2:0/1527320905 shutdown_connections 2023-11-16T16:49:57.033 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.031+0000 7f03345ff700 1 -- 172.21.15.2:0/1527320905 wait complete. 2023-11-16T16:49:57.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.032+0000 7f03345ff700 1 Processor -- start 2023-11-16T16:49:57.034 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.032+0000 7f03345ff700 1 -- start start 2023-11-16T16:49:57.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.032+0000 7f03345ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f032c073550 0x7f032c19c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:57.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.032+0000 7f03345ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f032c074b30 0x7f032c19cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:57.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.032+0000 7f03345ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f032c10c810 0x7f032c1a3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:57.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.033+0000 7f03345ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c1a4210 con 0x7f032c073550 2023-11-16T16:49:57.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.033+0000 7f03345ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c1a4350 con 0x7f032c10c810 2023-11-16T16:49:57.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.033+0000 7f03345ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c1a44e0 con 0x7f032c074b30 2023-11-16T16:49:57.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.033+0000 7f033239b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f032c073550 0x7f032c19c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:57.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.033+0000 7f0332b9c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f032c10c810 0x7f032c1a3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:57.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.033+0000 7f033239b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f032c073550 0x7f032c19c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60640/0 (socket says 172.21.15.2:60640) 2023-11-16T16:49:57.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.033+0000 7f0332b9c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f032c10c810 0x7f032c1a3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:59748/0 (socket says 172.21.15.2:59748) 2023-11-16T16:49:57.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.033+0000 7f033239b700 1 -- 172.21.15.2:0/2733167014 learned_addr learned my addr 172.21.15.2:0/2733167014 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:49:57.035 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.033+0000 7f0331b9a700 1 --2- 172.21.15.2:0/2733167014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f032c074b30 0x7f032c19cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:57.036 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.033+0000 7f0332b9c700 1 -- 172.21.15.2:0/2733167014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f032c074b30 msgr2=0x7f032c19cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:57.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.033+0000 7f0332b9c700 1 --2- 172.21.15.2:0/2733167014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f032c074b30 0x7f032c19cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.033+0000 7f0332b9c700 1 -- 172.21.15.2:0/2733167014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f032c073550 msgr2=0x7f032c19c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:57.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.033+0000 7f0332b9c700 1 --2- 172.21.15.2:0/2733167014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f032c073550 0x7f032c19c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.033+0000 7f0332b9c700 1 -- 172.21.15.2:0/2733167014 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0328009520 con 0x7f032c10c810 2023-11-16T16:49:57.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.034+0000 7f0332b9c700 1 --2- 172.21.15.2:0/2733167014 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f032c10c810 0x7f032c1a3c40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0324009cd0 tx=0x7f032400c620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:57.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.034+0000 7f03237fe700 1 -- 172.21.15.2:0/2733167014 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0324017070 con 0x7f032c10c810 2023-11-16T16:49:57.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.034+0000 7f03237fe700 1 -- 172.21.15.2:0/2733167014 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f032400f040 con 0x7f032c10c810 2023-11-16T16:49:57.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.034+0000 7f03345ff700 1 -- 172.21.15.2:0/2733167014 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0324009d00 con 0x7f032c10c810 2023-11-16T16:49:57.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.034+0000 7f03237fe700 1 -- 172.21.15.2:0/2733167014 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032400e050 con 0x7f032c10c810 2023-11-16T16:49:57.037 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.034+0000 7f03345ff700 1 -- 172.21.15.2:0/2733167014 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f031c009520 con 0x7f032c10c810 2023-11-16T16:49:57.038 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.035+0000 7f03345ff700 1 -- 172.21.15.2:0/2733167014 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f032c102360 con 0x7f032c10c810 2023-11-16T16:49:57.040 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.039+0000 7f03237fe700 1 -- 172.21.15.2:0/2733167014 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 23) v1 ==== 135890+0+0 (secure 0 0 0) 0x7f032400cd10 con 0x7f032c10c810 2023-11-16T16:49:57.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.040+0000 7f03237fe700 1 --2- 172.21.15.2:0/2733167014 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f03180a13d0 0x7f03180a3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:49:57.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.040+0000 7f03237fe700 1 -- 172.21.15.2:0/2733167014 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f03240cc7f0 con 0x7f032c10c810 2023-11-16T16:49:57.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.040+0000 7f033239b700 1 --2- 172.21.15.2:0/2733167014 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f03180a13d0 0x7f03180a3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:49:57.042 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.041+0000 7f033239b700 1 --2- 172.21.15.2:0/2733167014 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f03180a13d0 0x7f03180a3880 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0328009e90 tx=0x7f0328009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:49:57.045 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.044+0000 7f03237fe700 1 -- 172.21.15.2:0/2733167014 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0324095720 con 0x7f032c10c810 2023-11-16T16:49:57.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:56 smithi155 ceph-mon[131607]: [16/Nov/2023:16:49:55] ENGINE Bus STARTING 2023-11-16T16:49:57.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:56 smithi155 ceph-mon[131607]: [16/Nov/2023:16:49:55] ENGINE Serving on https://172.21.15.2:7150 2023-11-16T16:49:57.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:56 smithi155 ceph-mon[131607]: [16/Nov/2023:16:49:55] ENGINE Serving on http://172.21.15.2:8765 2023-11-16T16:49:57.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:56 smithi155 ceph-mon[131607]: [16/Nov/2023:16:49:55] ENGINE Bus STARTED 2023-11-16T16:49:57.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:56 smithi155 ceph-mon[131607]: mgrmap e22: x(active, since 1.01993s), standbys: z, y 2023-11-16T16:49:57.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.332+0000 7f03345ff700 1 -- 172.21.15.2:0/2733167014 --> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f032c060f30 con 0x7f03180a13d0 2023-11-16T16:49:57.341 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.339+0000 7f03237fe700 1 -- 172.21.15.2:0/2733167014 <== mgr.15372 v2:172.21.15.2:6800/1352003968 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f032c060f30 con 0x7f03180a13d0 2023-11-16T16:49:57.341 INFO:teuthology.orchestra.run.smithi002.stdout:/ 1M 6M3h 2023-11-16T16:49:57.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.342+0000 7f03345ff700 1 -- 172.21.15.2:0/2733167014 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f03180a13d0 msgr2=0x7f03180a3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:57.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.342+0000 7f03345ff700 1 --2- 172.21.15.2:0/2733167014 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f03180a13d0 0x7f03180a3880 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0328009e90 tx=0x7f0328009260 comp rx=0 tx=0).stop 2023-11-16T16:49:57.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.342+0000 7f03345ff700 1 -- 172.21.15.2:0/2733167014 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f032c10c810 msgr2=0x7f032c1a3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:49:57.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.342+0000 7f03345ff700 1 --2- 172.21.15.2:0/2733167014 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f032c10c810 0x7f032c1a3c40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0324009cd0 tx=0x7f032400c620 comp rx=0 tx=0).stop 2023-11-16T16:49:57.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.342+0000 7f03345ff700 1 -- 172.21.15.2:0/2733167014 shutdown_connections 2023-11-16T16:49:57.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.342+0000 7f03345ff700 1 --2- 172.21.15.2:0/2733167014 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f03180a13d0 0x7f03180a3880 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.342+0000 7f03345ff700 1 --2- 172.21.15.2:0/2733167014 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f032c10c810 0x7f032c1a3c40 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.342+0000 7f03345ff700 1 --2- 172.21.15.2:0/2733167014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f032c073550 0x7f032c19c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.342+0000 7f03345ff700 1 --2- 172.21.15.2:0/2733167014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f032c074b30 0x7f032c19cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:49:57.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.342+0000 7f03345ff700 1 -- 172.21.15.2:0/2733167014 >> 172.21.15.2:0/2733167014 conn(0x7f032c0fb840 msgr2=0x7f032c0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:49:57.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.342+0000 7f03345ff700 1 -- 172.21.15.2:0/2733167014 shutdown_connections 2023-11-16T16:49:57.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:49:57.342+0000 7f03345ff700 1 -- 172.21.15.2:0/2733167014 wait complete. 2023-11-16T16:49:57.362 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2023-11-16T16:49:57.451 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2023-11-16T16:49:57.461 INFO:teuthology.run_tasks:Running task background_exec... 2023-11-16T16:49:57.470 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T16:49:57.470 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2023-11-16T16:49:57.474 INFO:teuthology.run_tasks:Running task workunit... 2023-11-16T16:49:57.486 INFO:tasks.workunit:Pulling workunits from ref 5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:49:57.486 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-11-16T16:49:57.486 INFO:tasks.workunit:timeout=3h 2023-11-16T16:49:57.486 INFO:tasks.workunit:cleanup=True 2023-11-16T16:49:57.486 DEBUG:teuthology.orchestra.run.smithi002:> stat -- /home/ubuntu/cephtest/mnt.0 2023-11-16T16:49:57.537 INFO:teuthology.orchestra.run.smithi002.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-11-16T16:49:57.538 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-11-16T16:49:57.538 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 79h/121d Inode: 1099511628283 Links: 2 2023-11-16T16:49:57.538 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-16T16:49:57.538 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:cephfs_t:s0 2023-11-16T16:49:57.538 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2023-11-16 16:47:54.430561400 +0000 2023-11-16T16:49:57.538 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2023-11-16 16:47:54.430561400 +0000 2023-11-16T16:49:57.538 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2023-11-16 16:49:43.674534754 +0000 2023-11-16T16:49:57.538 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2023-11-16 16:47:54.430561400 +0000 2023-11-16T16:49:57.539 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-11-16T16:49:57.539 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-11-16T16:49:57.620 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 5e4a743ad7fd1422ebd6c4dc950a21a303279207 2023-11-16T16:49:57.668 INFO:tasks.workunit.client.0.smithi002.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-11-16T16:49:57.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:57 smithi002 ceph-mon[128712]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:57.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:57 smithi002 ceph-mon[128712]: pgmap v3: 129 pgs: 129 active+clean; 1.0 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:49:57.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:57 smithi002 ceph-mon[128712]: from='client.25435 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:57.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:57 smithi002 ceph-mon[128712]: pgmap v4: 129 pgs: 129 active+clean; 1.0 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:49:57.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:57 smithi002 ceph-mon[128712]: mgrmap e23: x(active, since 2s), standbys: z, y 2023-11-16T16:49:58.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:57 smithi141 ceph-mon[131184]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:58.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:57 smithi141 ceph-mon[131184]: pgmap v3: 129 pgs: 129 active+clean; 1.0 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:49:58.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:57 smithi141 ceph-mon[131184]: from='client.25435 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:58.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:57 smithi141 ceph-mon[131184]: pgmap v4: 129 pgs: 129 active+clean; 1.0 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:49:58.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:57 smithi141 ceph-mon[131184]: mgrmap e23: x(active, since 2s), standbys: z, y 2023-11-16T16:49:58.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:57 smithi155 ceph-mon[131607]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:58.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:57 smithi155 ceph-mon[131607]: pgmap v3: 129 pgs: 129 active+clean; 1.0 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:49:58.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:57 smithi155 ceph-mon[131607]: from='client.25435 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:58.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:57 smithi155 ceph-mon[131607]: pgmap v4: 129 pgs: 129 active+clean; 1.0 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:49:58.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:57 smithi155 ceph-mon[131607]: mgrmap e23: x(active, since 2s), standbys: z, y 2023-11-16T16:49:58.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:58 smithi002 ceph-mon[128712]: from='client.25441 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:58.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:58 smithi002 ceph-mon[128712]: from='client.25475 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:58.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:58 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:49:58.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:58 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:49:58.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:58 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:49:58.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:58 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:49:59.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:58 smithi141 ceph-mon[131184]: from='client.25441 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:59.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:58 smithi141 ceph-mon[131184]: from='client.25475 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:59.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:58 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:49:59.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:58 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:49:59.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:58 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:49:59.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:58 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:49:59.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:58 smithi155 ceph-mon[131607]: from='client.25441 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:59.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:58 smithi155 ceph-mon[131607]: from='client.25475 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:49:59.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:58 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:49:59.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:58 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:49:59.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:58 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:49:59.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:58 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:49:59.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:59 smithi002 ceph-mon[128712]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:49:59.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:59 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:49:59.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:59 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:49:59.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:59 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:49:59.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:59 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:49:59.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:59 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:49:59.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:59 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:49:59.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:59 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:49:59.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:59 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:49:59.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:59 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:49:59.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:59 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:49:59.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:59 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:49:59.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:59 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:49:59.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:59 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:49:59.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:49:59 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:59 smithi141 ceph-mon[131184]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:50:00.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:59 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:00.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:59 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:00.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:59 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:00.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:59 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:00.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:59 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:59 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:00.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:59 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:59 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:00.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:59 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:59 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:59 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:59 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:59 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:49:59 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:59 smithi155 ceph-mon[131607]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:50:00.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:59 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:00.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:59 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:00.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:59 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:00.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:59 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:00.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:59 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:59 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:00.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:59 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:59 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:00.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:59 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:59 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:59 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:59 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:59 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:49:59 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:00.653 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:50:00 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:50:00.287+0000 7fa592ef7700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:50:00.653 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:50:00 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:50:00.287+0000 7fa592ef7700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:50:00.653 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:50:00 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:50:00.287+0000 7fa592ef7700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:50:00.653 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:50:00 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:50:00.287+0000 7fa592ef7700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:50:00.653 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 16:50:00 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T16:50:00.287+0000 7fa592ef7700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-16T16:50:00.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:00 smithi002 ceph-mon[128712]: Adjusting osd_memory_target on smithi141 to 196.2M 2023-11-16T16:50:00.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:00 smithi002 ceph-mon[128712]: Unable to set osd_memory_target on smithi141 to 205752422: error parsing value: Value '205752422' is below minimum 939524096 2023-11-16T16:50:00.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:00 smithi002 ceph-mon[128712]: Adjusting osd_memory_target on smithi155 to 196.2M 2023-11-16T16:50:00.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:00 smithi002 ceph-mon[128712]: Unable to set osd_memory_target on smithi155 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-11-16T16:50:00.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:00 smithi002 ceph-mon[128712]: overall HEALTH_OK 2023-11-16T16:50:01.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:00 smithi141 ceph-mon[131184]: Adjusting osd_memory_target on smithi141 to 196.2M 2023-11-16T16:50:01.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:00 smithi141 ceph-mon[131184]: Unable to set osd_memory_target on smithi141 to 205752422: error parsing value: Value '205752422' is below minimum 939524096 2023-11-16T16:50:01.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:00 smithi141 ceph-mon[131184]: Adjusting osd_memory_target on smithi155 to 196.2M 2023-11-16T16:50:01.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:00 smithi141 ceph-mon[131184]: Unable to set osd_memory_target on smithi155 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-11-16T16:50:01.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:00 smithi141 ceph-mon[131184]: overall HEALTH_OK 2023-11-16T16:50:01.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:00 smithi155 ceph-mon[131607]: Adjusting osd_memory_target on smithi141 to 196.2M 2023-11-16T16:50:01.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:00 smithi155 ceph-mon[131607]: Unable to set osd_memory_target on smithi141 to 205752422: error parsing value: Value '205752422' is below minimum 939524096 2023-11-16T16:50:01.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:00 smithi155 ceph-mon[131607]: Adjusting osd_memory_target on smithi155 to 196.2M 2023-11-16T16:50:01.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:00 smithi155 ceph-mon[131607]: Unable to set osd_memory_target on smithi155 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-11-16T16:50:01.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:00 smithi155 ceph-mon[131607]: overall HEALTH_OK 2023-11-16T16:50:01.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:01 smithi002 ceph-mon[128712]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:50:01.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:01 smithi002 ceph-mon[128712]: mgrmap e24: x(active, since 6s), standbys: z, y 2023-11-16T16:50:01.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:01 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:01.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:01 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:01.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:01 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:01.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:01 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:01.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:01 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:01.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:01 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:01.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:01 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:50:01.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:01 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:50:02.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:01 smithi141 ceph-mon[131184]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:50:02.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:01 smithi141 ceph-mon[131184]: mgrmap e24: x(active, since 6s), standbys: z, y 2023-11-16T16:50:02.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:01 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:02.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:01 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:02.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:01 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:02.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:01 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:02.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:01 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:02.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:01 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:02.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:01 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:50:02.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:01 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:50:02.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:01 smithi155 ceph-mon[131607]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T16:50:02.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:01 smithi155 ceph-mon[131607]: mgrmap e24: x(active, since 6s), standbys: z, y 2023-11-16T16:50:02.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:01 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:02.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:01 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:02.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:01 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:02.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:01 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:02.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:01 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:02.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:01 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T16:50:02.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:01 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:50:02.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:01 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:50:03.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:02 smithi141 ceph-mon[131184]: Adjusting osd_memory_target on smithi002 to 223.0M 2023-11-16T16:50:03.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:02 smithi141 ceph-mon[131184]: Unable to set osd_memory_target on smithi002 to 233877504: error parsing value: Value '233877504' is below minimum 939524096 2023-11-16T16:50:03.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:02 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:50:03.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:02 smithi141 ceph-mon[131184]: Updating smithi002:/etc/ceph/ceph.conf 2023-11-16T16:50:03.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:02 smithi141 ceph-mon[131184]: Updating smithi141:/etc/ceph/ceph.conf 2023-11-16T16:50:03.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:02 smithi141 ceph-mon[131184]: Updating smithi155:/etc/ceph/ceph.conf 2023-11-16T16:50:03.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:02 smithi141 ceph-mon[131184]: Updating smithi141:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:50:03.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:02 smithi141 ceph-mon[131184]: Updating smithi155:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:50:03.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:02 smithi141 ceph-mon[131184]: Updating smithi141:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:50:03.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:02 smithi141 ceph-mon[131184]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:50:03.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:02 smithi141 ceph-mon[131184]: Updating smithi002:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:50:03.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:02 smithi141 ceph-mon[131184]: Updating smithi141:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:50:03.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:02 smithi141 ceph-mon[131184]: Updating smithi155:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:50:03.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:02 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:03.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:02 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:03.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:02 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:03.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:02 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:03.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:02 smithi155 ceph-mon[131607]: Adjusting osd_memory_target on smithi002 to 223.0M 2023-11-16T16:50:03.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:02 smithi155 ceph-mon[131607]: Unable to set osd_memory_target on smithi002 to 233877504: error parsing value: Value '233877504' is below minimum 939524096 2023-11-16T16:50:03.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:02 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:50:03.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:02 smithi155 ceph-mon[131607]: Updating smithi002:/etc/ceph/ceph.conf 2023-11-16T16:50:03.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:02 smithi155 ceph-mon[131607]: Updating smithi141:/etc/ceph/ceph.conf 2023-11-16T16:50:03.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:02 smithi155 ceph-mon[131607]: Updating smithi155:/etc/ceph/ceph.conf 2023-11-16T16:50:03.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:02 smithi155 ceph-mon[131607]: Updating smithi141:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:50:03.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:02 smithi155 ceph-mon[131607]: Updating smithi155:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:50:03.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:02 smithi155 ceph-mon[131607]: Updating smithi141:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:50:03.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:02 smithi155 ceph-mon[131607]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:50:03.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:02 smithi155 ceph-mon[131607]: Updating smithi002:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:50:03.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:02 smithi155 ceph-mon[131607]: Updating smithi141:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:50:03.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:02 smithi155 ceph-mon[131607]: Updating smithi155:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:50:03.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:02 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:03.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:02 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:03.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:02 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:03.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:02 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:03.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:02 smithi002 ceph-mon[128712]: Adjusting osd_memory_target on smithi002 to 223.0M 2023-11-16T16:50:03.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:02 smithi002 ceph-mon[128712]: Unable to set osd_memory_target on smithi002 to 233877504: error parsing value: Value '233877504' is below minimum 939524096 2023-11-16T16:50:03.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:02 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:50:03.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:02 smithi002 ceph-mon[128712]: Updating smithi002:/etc/ceph/ceph.conf 2023-11-16T16:50:03.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:02 smithi002 ceph-mon[128712]: Updating smithi141:/etc/ceph/ceph.conf 2023-11-16T16:50:03.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:02 smithi002 ceph-mon[128712]: Updating smithi155:/etc/ceph/ceph.conf 2023-11-16T16:50:03.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:02 smithi002 ceph-mon[128712]: Updating smithi141:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:50:03.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:02 smithi002 ceph-mon[128712]: Updating smithi155:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:50:03.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:02 smithi002 ceph-mon[128712]: Updating smithi141:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:50:03.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:02 smithi002 ceph-mon[128712]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:50:03.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:02 smithi002 ceph-mon[128712]: Updating smithi002:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.conf 2023-11-16T16:50:03.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:02 smithi002 ceph-mon[128712]: Updating smithi141:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:50:03.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:02 smithi002 ceph-mon[128712]: Updating smithi155:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:50:03.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:02 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:03.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:02 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:03.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:02 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:03.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:02 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:03 smithi141 ceph-mon[131184]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:50:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:03 smithi141 ceph-mon[131184]: Updating smithi002:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:50:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:03 smithi141 ceph-mon[131184]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 170 KiB/s wr, 73 op/s 2023-11-16T16:50:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:03 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:03 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:03 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:03 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:50:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:03 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:50:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:03 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:50:04.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:03 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:04.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:03 smithi155 ceph-mon[131607]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:50:04.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:03 smithi155 ceph-mon[131607]: Updating smithi002:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:50:04.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:03 smithi155 ceph-mon[131607]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 170 KiB/s wr, 73 op/s 2023-11-16T16:50:04.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:03 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:04.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:03 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:04.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:03 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:04.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:03 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:50:04.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:03 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:50:04.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:03 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:50:04.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:03 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:04.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:03 smithi002 ceph-mon[128712]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2023-11-16T16:50:04.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:03 smithi002 ceph-mon[128712]: Updating smithi002:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/config/ceph.client.admin.keyring 2023-11-16T16:50:04.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:03 smithi002 ceph-mon[128712]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 170 KiB/s wr, 73 op/s 2023-11-16T16:50:04.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:03 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:04.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:03 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:04.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:03 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:04.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:03 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:50:04.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:03 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:50:04.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:03 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:50:04.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:03 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:50:05.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:04 smithi141 ceph-mon[131184]: Service mds.all has no related daemon already placed. 2023-11-16T16:50:05.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:04 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:50:05.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:04 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:50:05.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:04 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:50:05.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:04 smithi141 ceph-mon[131184]: Service mds.all has no related daemon already placed. 2023-11-16T16:50:05.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:04 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:50:05.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:04 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:50:05.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:04 smithi155 ceph-mon[131607]: Service mds.all has no related daemon already placed. 2023-11-16T16:50:05.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:04 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:50:05.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:04 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:50:05.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:04 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:50:05.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:04 smithi155 ceph-mon[131607]: Service mds.all has no related daemon already placed. 2023-11-16T16:50:05.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:04 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:50:05.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:04 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:50:05.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:04 smithi002 ceph-mon[128712]: Service mds.all has no related daemon already placed. 2023-11-16T16:50:05.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:04 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:50:05.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:04 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:50:05.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:04 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:50:05.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:04 smithi002 ceph-mon[128712]: Service mds.all has no related daemon already placed. 2023-11-16T16:50:05.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:04 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:50:05.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:04 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:50:06.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:05 smithi141 ceph-mon[131184]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 132 KiB/s wr, 56 op/s 2023-11-16T16:50:06.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:05 smithi155 ceph-mon[131607]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 132 KiB/s wr, 56 op/s 2023-11-16T16:50:06.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:05 smithi002 ceph-mon[128712]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 132 KiB/s wr, 56 op/s 2023-11-16T16:50:07.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:06 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:50:07.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:06 smithi141 ceph-mon[131184]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 109 KiB/s wr, 49 op/s 2023-11-16T16:50:07.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:06 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:50:07.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:06 smithi155 ceph-mon[131607]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 109 KiB/s wr, 49 op/s 2023-11-16T16:50:07.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:06 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:50:07.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:06 smithi002 ceph-mon[128712]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 109 KiB/s wr, 49 op/s 2023-11-16T16:50:09.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:08 smithi141 ceph-mon[131184]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 100 KiB/s wr, 45 op/s 2023-11-16T16:50:09.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:08 smithi155 ceph-mon[131607]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 100 KiB/s wr, 45 op/s 2023-11-16T16:50:09.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:08 smithi002 ceph-mon[128712]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 100 KiB/s wr, 45 op/s 2023-11-16T16:50:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.740+0000 7f366f3ff700 1 -- 172.21.15.2:0/3286610109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3668100c50 msgr2=0x7f3668101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.740+0000 7f366f3ff700 1 --2- 172.21.15.2:0/3286610109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3668100c50 0x7f3668101020 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f36640099c0 tx=0x7f3664009cd0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.741+0000 7f366f3ff700 1 -- 172.21.15.2:0/3286610109 shutdown_connections 2023-11-16T16:50:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.741+0000 7f366f3ff700 1 --2- 172.21.15.2:0/3286610109 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3668105560 0x7f366810d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.741+0000 7f366f3ff700 1 --2- 172.21.15.2:0/3286610109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3668100c50 0x7f3668101020 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.741+0000 7f366f3ff700 1 --2- 172.21.15.2:0/3286610109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36681015f0 0x7f3668104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.741+0000 7f366f3ff700 1 -- 172.21.15.2:0/3286610109 >> 172.21.15.2:0/3286610109 conn(0x7f3668077550 msgr2=0x7f3668077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.741+0000 7f366f3ff700 1 -- 172.21.15.2:0/3286610109 shutdown_connections 2023-11-16T16:50:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.741+0000 7f366f3ff700 1 -- 172.21.15.2:0/3286610109 wait complete. 2023-11-16T16:50:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.742+0000 7f366f3ff700 1 Processor -- start 2023-11-16T16:50:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.742+0000 7f366f3ff700 1 -- start start 2023-11-16T16:50:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.743+0000 7f366f3ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3668100c50 0x7f3668198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.743+0000 7f366f3ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36681015f0 0x7f3668198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.743+0000 7f366f3ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3668105560 0x7f366819cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.743+0000 7f366f3ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366819d5f0 con 0x7f36681015f0 2023-11-16T16:50:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.743+0000 7f366f3ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366819d760 con 0x7f3668100c50 2023-11-16T16:50:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.743+0000 7f366f3ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366819d920 con 0x7f3668105560 2023-11-16T16:50:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.743+0000 7f366c99a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36681015f0 0x7f3668198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.743+0000 7f366d19b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3668100c50 0x7f3668198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.743+0000 7f366c99a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36681015f0 0x7f3668198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51556/0 (socket says 172.21.15.2:51556) 2023-11-16T16:50:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.743+0000 7f366c99a700 1 -- 172.21.15.2:0/109867465 learned_addr learned my addr 172.21.15.2:0/109867465 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.743+0000 7f366d99c700 1 --2- 172.21.15.2:0/109867465 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3668105560 0x7f366819cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.743+0000 7f366c99a700 1 -- 172.21.15.2:0/109867465 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3668105560 msgr2=0x7f366819cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.743+0000 7f366c99a700 1 --2- 172.21.15.2:0/109867465 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3668105560 0x7f366819cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.743+0000 7f366c99a700 1 -- 172.21.15.2:0/109867465 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3668100c50 msgr2=0x7f3668198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.744+0000 7f366c99a700 1 --2- 172.21.15.2:0/109867465 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3668100c50 0x7f3668198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.744+0000 7f366c99a700 1 -- 172.21.15.2:0/109867465 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36640095f0 con 0x7f36681015f0 2023-11-16T16:50:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.744+0000 7f366c99a700 1 --2- 172.21.15.2:0/109867465 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36681015f0 0x7f3668198b50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f365800e940 tx=0x7f365800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.744+0000 7f365e7fc700 1 -- 172.21.15.2:0/109867465 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3658009730 con 0x7f36681015f0 2023-11-16T16:50:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.744+0000 7f365e7fc700 1 -- 172.21.15.2:0/109867465 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3658010430 con 0x7f36681015f0 2023-11-16T16:50:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.744+0000 7f366f3ff700 1 -- 172.21.15.2:0/109867465 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36681103b0 con 0x7f36681015f0 2023-11-16T16:50:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.744+0000 7f365e7fc700 1 -- 172.21.15.2:0/109867465 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365800cea0 con 0x7f36681015f0 2023-11-16T16:50:10.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.744+0000 7f366f3ff700 1 -- 172.21.15.2:0/109867465 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3668110220 con 0x7f36681015f0 2023-11-16T16:50:10.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.747+0000 7f365e7fc700 1 -- 172.21.15.2:0/109867465 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7f3658010810 con 0x7f36681015f0 2023-11-16T16:50:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.748+0000 7f3682c8ab80 1 Processor -- start 2023-11-16T16:50:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.749+0000 7f3682c8ab80 1 -- start start 2023-11-16T16:50:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.749+0000 7f365e7fc700 1 --2- 172.21.15.2:0/109867465 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f36540a1460 0x7f36540a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.749+0000 7f3682c8ab80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757aa966b0 0x55757ac401e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.749+0000 7f3682c8ab80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40720 0x55757ac3a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.749+0000 7f365e7fc700 1 -- 172.21.15.2:0/109867465 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f3658014070 con 0x7f36681015f0 2023-11-16T16:50:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.749+0000 7f3682c8ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757ac40b90 0x55757ac3ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.749+0000 7f3682c8ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55757ab40000 con 0x55757ac40b90 2023-11-16T16:50:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.749+0000 7f3682c8ab80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55757ab92840 con 0x55757aa966b0 2023-11-16T16:50:10.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.749+0000 7f3682c8ab80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55757ab8bcd0 con 0x55757ac40720 2023-11-16T16:50:10.753 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.749+0000 7f366d99c700 1 --2- 172.21.15.2:0/109867465 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f36540a1460 0x7f36540a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.749+0000 7f366c99a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40720 0x55757ac3a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.749+0000 7f366c99a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40720 0x55757ac3a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:43090/0 (socket says 172.21.15.2:43090) 2023-11-16T16:50:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.749+0000 7f366d19b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757aa966b0 0x55757ac401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.749+0000 7f366c99a700 1 -- 172.21.15.2:0/3729258758 learned_addr learned my addr 172.21.15.2:0/3729258758 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:10.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.750+0000 7f366d99c700 1 --2- 172.21.15.2:0/3729258758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757ac40b90 0x55757ac3ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.750+0000 7f366c99a700 1 -- 172.21.15.2:0/3729258758 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757aa966b0 msgr2=0x55757ac401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.750+0000 7f366c99a700 1 --2- 172.21.15.2:0/3729258758 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757aa966b0 0x55757ac401e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.750+0000 7f366c99a700 1 -- 172.21.15.2:0/3729258758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757ac40b90 msgr2=0x55757ac3ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.750+0000 7f366c99a700 1 --2- 172.21.15.2:0/3729258758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757ac40b90 0x55757ac3ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.750+0000 7f366c99a700 1 -- 172.21.15.2:0/3729258758 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55757aa0d5d0 con 0x55757ac40720 2023-11-16T16:50:10.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.750+0000 7f366d99c700 1 --2- 172.21.15.2:0/3729258758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757ac40b90 0x55757ac3ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:50:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.750+0000 7f366c99a700 1 --2- 172.21.15.2:0/3729258758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40720 0x55757ac3a560 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3658099ea0 tx=0x7f36580998f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea401a4f0401ca41 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.750+0000 7f366d99c700 1 --2- 172.21.15.2:0/109867465 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f36540a1460 0x7f36540a3910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x55757ac40f60 tx=0x7f3660009920 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.750+0000 7f36537fe700 1 -- 172.21.15.2:0/3729258758 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365809df40 con 0x55757ac40720 2023-11-16T16:50:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.750+0000 7f36537fe700 1 -- 172.21.15.2:0/3729258758 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f36580a2360 con 0x55757ac40720 2023-11-16T16:50:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.750+0000 7f36537fe700 1 -- 172.21.15.2:0/3729258758 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36580a2130 con 0x55757ac40720 2023-11-16T16:50:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.751+0000 7f3682c8ab80 1 -- 172.21.15.2:0/3729258758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40720 msgr2=0x55757ac3a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.751+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/3729258758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40720 0x55757ac3a560 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3658099ea0 tx=0x7f36580998f0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.751+0000 7f3682c8ab80 1 -- 172.21.15.2:0/3729258758 shutdown_connections 2023-11-16T16:50:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.751+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/3729258758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757ac40b90 0x55757ac3ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.751+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/3729258758 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757aa966b0 0x55757ac401e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.751+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/3729258758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40720 0x55757ac3a560 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.751+0000 7f3682c8ab80 1 -- 172.21.15.2:0/3729258758 >> 172.21.15.2:0/3729258758 conn(0x55757ac2ff50 msgr2=0x55757ac35270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.751+0000 7f3682c8ab80 1 -- 172.21.15.2:0/3729258758 shutdown_connections 2023-11-16T16:50:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.751+0000 7f3682c8ab80 1 -- 172.21.15.2:0/3729258758 wait complete. 2023-11-16T16:50:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.752+0000 7f3682c8ab80 1 -- start start 2023-11-16T16:50:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.752+0000 7f3682c8ab80 1 Processor -- start 2023-11-16T16:50:10.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.752+0000 7f3682c8ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757aa966b0 0x55757ac46aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:10.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.752+0000 7f3682c8ab80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40720 0x55757ac43d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:10.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.752+0000 7f3682c8ab80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757ac40b90 0x55757ac44460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:10.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.752+0000 7f3682c8ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55757ab2d880 con 0x55757aa966b0 2023-11-16T16:50:10.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.752+0000 7f3682c8ab80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55757ab2d520 con 0x55757ac40b90 2023-11-16T16:50:10.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.752+0000 7f3682c8ab80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55757aa0f8c0 con 0x55757ac40720 2023-11-16T16:50:10.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.753+0000 7f3682c8ab80 10 client.0 _ll_register_callbacks cb 0x55757ab4b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:10.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.753+0000 7f366d19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757aa966b0 0x55757ac46aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:10.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.753+0000 7f366d19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757aa966b0 0x55757ac46aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51588/0 (socket says 172.21.15.2:51588) 2023-11-16T16:50:10.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.753+0000 7f366d19b700 1 -- 172.21.15.2:0/2834625977 learned_addr learned my addr 172.21.15.2:0/2834625977 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.753+0000 7f366d99c700 1 --2- 172.21.15.2:0/2834625977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757ac40b90 0x55757ac44460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.753+0000 7f366c99a700 1 --2- 172.21.15.2:0/2834625977 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40720 0x55757ac43d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.753+0000 7f366d19b700 1 -- 172.21.15.2:0/2834625977 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40720 msgr2=0x55757ac43d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.753+0000 7f366d19b700 1 --2- 172.21.15.2:0/2834625977 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40720 0x55757ac43d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.753+0000 7f366d19b700 1 -- 172.21.15.2:0/2834625977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757ac40b90 msgr2=0x55757ac44460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.753+0000 7f366d19b700 1 --2- 172.21.15.2:0/2834625977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757ac40b90 0x55757ac44460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.753+0000 7f366d19b700 1 -- 172.21.15.2:0/2834625977 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3668110220 con 0x55757aa966b0 2023-11-16T16:50:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.753+0000 7f366d19b700 1 --2- 172.21.15.2:0/2834625977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757aa966b0 0x55757ac46aa0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f36640058e0 tx=0x7f3664009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501876a9abcc6240 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.753+0000 7f3650ff9700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:50:10.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.754+0000 7f3650ff9700 1 -- 172.21.15.2:0/2834625977 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3664005590 con 0x55757aa966b0 2023-11-16T16:50:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.754+0000 7f3650ff9700 1 -- 172.21.15.2:0/2834625977 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f366400b790 con 0x55757aa966b0 2023-11-16T16:50:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.754+0000 7f3682c8ab80 1 -- 172.21.15.2:0/2834625977 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55757ac2d710 con 0x55757aa966b0 2023-11-16T16:50:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.754+0000 7f3650ff9700 1 -- 172.21.15.2:0/2834625977 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36640056f0 con 0x55757aa966b0 2023-11-16T16:50:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.754+0000 7f3682c8ab80 10 client.15462 fetch_fsmap learned FSMap version 27 2023-11-16T16:50:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.754+0000 7f3682c8ab80 1 -- 172.21.15.2:0/2834625977 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55757ab52130 con 0x55757aa966b0 2023-11-16T16:50:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.754+0000 7f3650ff9700 1 -- 172.21.15.2:0/2834625977 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55757ac2d710 con 0x55757aa966b0 2023-11-16T16:50:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.755+0000 7f3650ff9700 1 -- 172.21.15.2:0/2834625977 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 27) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f366400fb00 con 0x55757aa966b0 2023-11-16T16:50:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.755+0000 7f3682c8ab80 10 client.15462 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T16:50:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.755+0000 7f3682c8ab80 10 client.15462 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.755+0000 7f3682c8ab80 20 client.15462 populate_metadata read hostname 'smithi002' 2023-11-16T16:50:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.755+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/2834625977 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55757ac53ef0 0x55757ac56300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:10.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.755+0000 7f3682c8ab80 4 client.15462 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:50:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.755+0000 7f3682c8ab80 1 -- 172.21.15.2:0/2834625977 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55757ab4e040 con 0x55757ac53ef0 2023-11-16T16:50:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.755+0000 7f366c99a700 1 --2- 172.21.15.2:0/2834625977 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55757ac53ef0 0x55757ac56300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.756+0000 7f366c99a700 1 --2- 172.21.15.2:0/2834625977 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55757ac53ef0 0x55757ac56300 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7646ffe405796579 server_cookie=35cea6384ccf05d in_seq=0 out_seq=0 2023-11-16T16:50:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.756+0000 7f3650ff9700 10 client.15462 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:50:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.757+0000 7f3650ff9700 1 -- 172.21.15.2:0/2834625977 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55757ab4e040 con 0x55757ac53ef0 2023-11-16T16:50:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.757+0000 7f3650ff9700 10 client.15462 handle_command_reply: tid=0 2023-11-16T16:50:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.757+0000 7f3682c8ab80 1 client.15462 shutdown 2023-11-16T16:50:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.757+0000 7f3682c8ab80 1 -- 172.21.15.2:0/2834625977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757aa966b0 msgr2=0x55757ac46aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.757+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/2834625977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757aa966b0 0x55757ac46aa0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f36640058e0 tx=0x7f3664009290 comp rx=0 tx=0).stop 2023-11-16T16:50:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.757+0000 7f3682c8ab80 1 -- 172.21.15.2:0/2834625977 shutdown_connections 2023-11-16T16:50:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.757+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/2834625977 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757ac40b90 0x55757ac44460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.757+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/2834625977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757aa966b0 0x55757ac46aa0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.757+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/2834625977 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55757ac53ef0 0x55757ac56300 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.757+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/2834625977 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40720 0x55757ac43d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.757+0000 7f3650ff9700 0 client.15462 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:50:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.757+0000 7f3682c8ab80 1 -- 172.21.15.2:0/2834625977 >> 172.21.15.2:0/2834625977 conn(0x55757ac2ff50 msgr2=0x55757ac347e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.757+0000 7f3682c8ab80 1 -- 172.21.15.2:0/2834625977 shutdown_connections 2023-11-16T16:50:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.757+0000 7f3682c8ab80 1 -- 172.21.15.2:0/2834625977 wait complete. 2023-11-16T16:50:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.757+0000 7f3682c8ab80 20 client.15462 trim_cache size 0 max 16384 2023-11-16T16:50:10.775 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.773+0000 7f3682c8ab80 1 Processor -- start 2023-11-16T16:50:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.773+0000 7f3682c8ab80 1 -- start start 2023-11-16T16:50:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.773+0000 7f3682c8ab80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757aa966b0 0x55757ac4b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.773+0000 7f3682c8ab80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40720 0x55757ac46640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.773+0000 7f3682c8ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757ac40b90 0x55757ac3a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.773+0000 7f3682c8ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55757ac44ca0 con 0x55757ac40b90 2023-11-16T16:50:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.773+0000 7f3682c8ab80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55757ac36a00 con 0x55757aa966b0 2023-11-16T16:50:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.773+0000 7f3682c8ab80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55757ac35fb0 con 0x55757ac40720 2023-11-16T16:50:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.773+0000 7f366d19b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757aa966b0 0x55757ac4b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.773+0000 7f366d99c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757ac40b90 0x55757ac3a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.773+0000 7f366d19b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757aa966b0 0x55757ac4b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:43236/0 (socket says 172.21.15.2:43236) 2023-11-16T16:50:10.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.773+0000 7f366d19b700 1 -- 172.21.15.2:0/1355222298 learned_addr learned my addr 172.21.15.2:0/1355222298 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:10.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.774+0000 7f366c99a700 1 --2- 172.21.15.2:0/1355222298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40720 0x55757ac46640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:10.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.774+0000 7f366d19b700 1 -- 172.21.15.2:0/1355222298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40720 msgr2=0x55757ac46640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:10.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.774+0000 7f366d19b700 1 --2- 172.21.15.2:0/1355222298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40720 0x55757ac46640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.774+0000 7f366d19b700 1 -- 172.21.15.2:0/1355222298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757ac40b90 msgr2=0x55757ac3a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:10.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.774+0000 7f366d19b700 1 --2- 172.21.15.2:0/1355222298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757ac40b90 0x55757ac3a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.774+0000 7f366d19b700 1 -- 172.21.15.2:0/1355222298 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55757ac316f0 con 0x55757aa966b0 2023-11-16T16:50:10.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.774+0000 7f366d19b700 1 --2- 172.21.15.2:0/1355222298 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757aa966b0 0x55757ac4b060 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3664017410 tx=0x7f3664009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43b964fb72452f44 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.774+0000 7f3653fff700 1 -- 172.21.15.2:0/1355222298 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36640165a0 con 0x55757aa966b0 2023-11-16T16:50:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.774+0000 7f3653fff700 1 -- 172.21.15.2:0/1355222298 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f366401e040 con 0x55757aa966b0 2023-11-16T16:50:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.774+0000 7f3653fff700 1 -- 172.21.15.2:0/1355222298 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366400b6b0 con 0x55757aa966b0 2023-11-16T16:50:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.775+0000 7f3682c8ab80 1 -- 172.21.15.2:0/1355222298 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757aa966b0 msgr2=0x55757ac4b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:10.780 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.775+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/1355222298 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757aa966b0 0x55757ac4b060 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3664017410 tx=0x7f3664009310 comp rx=0 tx=0).stop 2023-11-16T16:50:10.781 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.775+0000 7f3682c8ab80 1 -- 172.21.15.2:0/1355222298 shutdown_connections 2023-11-16T16:50:10.781 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.775+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/1355222298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757ac40b90 0x55757ac3a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.781 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.775+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/1355222298 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757aa966b0 0x55757ac4b060 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.781 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.775+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/1355222298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40720 0x55757ac46640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.781 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.775+0000 7f3682c8ab80 1 -- 172.21.15.2:0/1355222298 >> 172.21.15.2:0/1355222298 conn(0x55757ac2ff50 msgr2=0x55757ac502d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:10.781 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.775+0000 7f3682c8ab80 1 -- 172.21.15.2:0/1355222298 shutdown_connections 2023-11-16T16:50:10.781 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.775+0000 7f3682c8ab80 1 -- 172.21.15.2:0/1355222298 wait complete. 2023-11-16T16:50:10.781 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.775+0000 7f3682c8ab80 1 -- start start 2023-11-16T16:50:10.781 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.776+0000 7f3682c8ab80 1 Processor -- start 2023-11-16T16:50:10.781 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.776+0000 7f3682c8ab80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757aa966b0 0x55757ac4b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:10.781 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.776+0000 7f3682c8ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757ac40720 0x55757ac3c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:10.781 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.776+0000 7f3682c8ab80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40b90 0x55757ac3cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:10.782 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.776+0000 7f3682c8ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55757ac3d1e0 con 0x55757ac40720 2023-11-16T16:50:10.782 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.776+0000 7f3682c8ab80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55757ac3d350 con 0x55757aa966b0 2023-11-16T16:50:10.782 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.776+0000 7f3682c8ab80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55757ac3d510 con 0x55757ac40b90 2023-11-16T16:50:10.782 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.776+0000 7f3682c8ab80 10 client.0 _ll_register_callbacks cb 0x7f3664003ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:10.782 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.776+0000 7f366c99a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757ac40720 0x55757ac3c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:10.782 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.777+0000 7f366c99a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757ac40720 0x55757ac3c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51602/0 (socket says 172.21.15.2:51602) 2023-11-16T16:50:10.782 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.777+0000 7f366c99a700 1 -- 172.21.15.2:0/1272494455 learned_addr learned my addr 172.21.15.2:0/1272494455 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:10.782 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.777+0000 7f366d19b700 1 --2- 172.21.15.2:0/1272494455 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757aa966b0 0x55757ac4b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:10.782 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.777+0000 7f366d99c700 1 --2- 172.21.15.2:0/1272494455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40b90 0x55757ac3cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:10.782 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.777+0000 7f366c99a700 1 -- 172.21.15.2:0/1272494455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40b90 msgr2=0x55757ac3cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:10.782 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.777+0000 7f366c99a700 1 --2- 172.21.15.2:0/1272494455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40b90 0x55757ac3cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.783 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.777+0000 7f366c99a700 1 -- 172.21.15.2:0/1272494455 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757aa966b0 msgr2=0x55757ac4b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:10.783 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.777+0000 7f366c99a700 1 --2- 172.21.15.2:0/1272494455 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757aa966b0 0x55757ac4b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.783 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.777+0000 7f366c99a700 1 -- 172.21.15.2:0/1272494455 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55757ab52130 con 0x55757ac40720 2023-11-16T16:50:10.783 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.777+0000 7f366d19b700 1 --2- 172.21.15.2:0/1272494455 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757aa966b0 0x55757ac4b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:10.783 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.777+0000 7f366c99a700 1 --2- 172.21.15.2:0/1272494455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757ac40720 0x55757ac3c6f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3658098260 tx=0x7f365809a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2caf128b01192e50 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:10.783 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.777+0000 7f3650ff9700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:50:10.783 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.777+0000 7f3650ff9700 1 -- 172.21.15.2:0/1272494455 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36580b8220 con 0x55757ac40720 2023-11-16T16:50:10.783 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.778+0000 7f3650ff9700 1 -- 172.21.15.2:0/1272494455 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f3658099400 con 0x55757ac40720 2023-11-16T16:50:10.783 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.778+0000 7f3682c8ab80 1 -- 172.21.15.2:0/1272494455 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36580a2360 con 0x55757ac40720 2023-11-16T16:50:10.783 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.778+0000 7f3650ff9700 1 -- 172.21.15.2:0/1272494455 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36580995c0 con 0x55757ac40720 2023-11-16T16:50:10.783 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.778+0000 7f3682c8ab80 10 client.15468 fetch_fsmap learned FSMap version 27 2023-11-16T16:50:10.783 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.778+0000 7f3682c8ab80 1 -- 172.21.15.2:0/1272494455 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55757a9b9d80 con 0x55757ac40720 2023-11-16T16:50:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.778+0000 7f3650ff9700 1 -- 172.21.15.2:0/1272494455 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f365807c080 con 0x55757ac40720 2023-11-16T16:50:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.778+0000 7f3650ff9700 1 -- 172.21.15.2:0/1272494455 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 27) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f36580a1f60 con 0x55757ac40720 2023-11-16T16:50:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.778+0000 7f3682c8ab80 10 client.15468 fetch_fsmap finished waiting for FSMap version 27 2023-11-16T16:50:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.778+0000 7f3682c8ab80 10 client.15468 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.778+0000 7f3682c8ab80 20 client.15468 populate_metadata read hostname 'smithi002' 2023-11-16T16:50:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.778+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/1272494455 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55757aca7f30 0x55757acaa340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.778+0000 7f3682c8ab80 4 client.15468 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:50:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.778+0000 7f3682c8ab80 1 -- 172.21.15.2:0/1272494455 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55757ab5a600 con 0x55757aca7f30 2023-11-16T16:50:10.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.779+0000 7f366d19b700 1 --2- 172.21.15.2:0/1272494455 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55757aca7f30 0x55757acaa340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.779+0000 7f366d19b700 1 --2- 172.21.15.2:0/1272494455 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55757aca7f30 0x55757acaa340 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e43c5bd0dc71f804 server_cookie=d74461e9b7e278f1 in_seq=0 out_seq=0 2023-11-16T16:50:10.785 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.779+0000 7f3650ff9700 10 client.15468 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:50:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.790+0000 7f3650ff9700 1 -- 172.21.15.2:0/1272494455 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55757ab5a600 con 0x55757aca7f30 2023-11-16T16:50:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.790+0000 7f3650ff9700 10 client.15468 handle_command_reply: tid=0 2023-11-16T16:50:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.790+0000 7f3682c8ab80 1 client.15468 shutdown 2023-11-16T16:50:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.790+0000 7f3682c8ab80 1 -- 172.21.15.2:0/1272494455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757ac40720 msgr2=0x55757ac3c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.790+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/1272494455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757ac40720 0x55757ac3c6f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3658098260 tx=0x7f365809a490 comp rx=0 tx=0).stop 2023-11-16T16:50:10.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.790+0000 7f3682c8ab80 1 -- 172.21.15.2:0/1272494455 shutdown_connections 2023-11-16T16:50:10.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.790+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/1272494455 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55757aca7f30 0x55757acaa340 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.790+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/1272494455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55757ac40b90 0x55757ac3cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.790+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/1272494455 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55757aa966b0 0x55757ac4b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.790+0000 7f3682c8ab80 1 --2- 172.21.15.2:0/1272494455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55757ac40720 0x55757ac3c6f0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.790+0000 7f3650ff9700 0 client.15468 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:50:10.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.790+0000 7f3682c8ab80 1 -- 172.21.15.2:0/1272494455 >> 172.21.15.2:0/1272494455 conn(0x55757ac2ff50 msgr2=0x55757ac3edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:10.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.790+0000 7f3682c8ab80 1 -- 172.21.15.2:0/1272494455 shutdown_connections 2023-11-16T16:50:10.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.790+0000 7f3682c8ab80 1 -- 172.21.15.2:0/1272494455 wait complete. 2023-11-16T16:50:10.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.791+0000 7f3682c8ab80 20 client.15468 trim_cache size 0 max 16384 2023-11-16T16:50:10.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:50:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:50:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:50:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:50:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.791+0000 7f366f3ff700 1 -- 172.21.15.2:0/109867465 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f36540a1460 msgr2=0x7f36540a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.791+0000 7f366f3ff700 1 --2- 172.21.15.2:0/109867465 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f36540a1460 0x7f36540a3910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x55757ac40f60 tx=0x7f3660009920 comp rx=0 tx=0).stop 2023-11-16T16:50:10.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.791+0000 7f366f3ff700 1 -- 172.21.15.2:0/109867465 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36681015f0 msgr2=0x7f3668198b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:10.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.791+0000 7f366f3ff700 1 --2- 172.21.15.2:0/109867465 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36681015f0 0x7f3668198b50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f365800e940 tx=0x7f365800ec50 comp rx=0 tx=0).stop 2023-11-16T16:50:10.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.791+0000 7f366f3ff700 1 -- 172.21.15.2:0/109867465 shutdown_connections 2023-11-16T16:50:10.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.791+0000 7f366f3ff700 1 --2- 172.21.15.2:0/109867465 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f36540a1460 0x7f36540a3910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.791+0000 7f366f3ff700 1 --2- 172.21.15.2:0/109867465 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3668105560 0x7f366819cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.791+0000 7f366f3ff700 1 --2- 172.21.15.2:0/109867465 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3668100c50 0x7f3668198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.791+0000 7f366f3ff700 1 --2- 172.21.15.2:0/109867465 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36681015f0 0x7f3668198b50 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:10.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.791+0000 7f366f3ff700 1 -- 172.21.15.2:0/109867465 >> 172.21.15.2:0/109867465 conn(0x7f3668077550 msgr2=0x7f36680fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:10.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.791+0000 7f366f3ff700 1 -- 172.21.15.2:0/109867465 shutdown_connections 2023-11-16T16:50:10.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:10.791+0000 7f366f3ff700 1 -- 172.21.15.2:0/109867465 wait complete. 2023-11-16T16:50:11.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:10 smithi141 ceph-mon[131184]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 100 KiB/s wr, 45 op/s 2023-11-16T16:50:11.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:10 smithi155 ceph-mon[131607]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 100 KiB/s wr, 45 op/s 2023-11-16T16:50:11.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:10 smithi002 ceph-mon[128712]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 100 KiB/s wr, 45 op/s 2023-11-16T16:50:13.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:12 smithi141 ceph-mon[131184]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 100 KiB/s wr, 48 op/s 2023-11-16T16:50:13.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:12 smithi155 ceph-mon[131607]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 100 KiB/s wr, 48 op/s 2023-11-16T16:50:13.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:12 smithi002 ceph-mon[128712]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 100 KiB/s wr, 48 op/s 2023-11-16T16:50:15.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:15 smithi155 ceph-mon[131607]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2023-11-16T16:50:15.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:15 smithi002 ceph-mon[128712]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2023-11-16T16:50:16.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:15 smithi141 ceph-mon[131184]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2023-11-16T16:50:16.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:16 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:50:16.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:16 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:50:17.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:16 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:50:17.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:17 smithi155 ceph-mon[131607]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2023-11-16T16:50:17.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:17 smithi002 ceph-mon[128712]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2023-11-16T16:50:18.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:17 smithi141 ceph-mon[131184]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2023-11-16T16:50:19.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:18 smithi141 ceph-mon[131184]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2023-11-16T16:50:19.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:18 smithi155 ceph-mon[131607]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2023-11-16T16:50:19.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:18 smithi002 ceph-mon[128712]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2023-11-16T16:50:21.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:20 smithi141 ceph-mon[131184]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2023-11-16T16:50:21.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:20 smithi155 ceph-mon[131607]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2023-11-16T16:50:21.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:20 smithi002 ceph-mon[128712]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2023-11-16T16:50:23.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:22 smithi141 ceph-mon[131184]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.3 KiB/s wr, 10 op/s 2023-11-16T16:50:23.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:22 smithi155 ceph-mon[131607]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.3 KiB/s wr, 10 op/s 2023-11-16T16:50:23.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:22 smithi002 ceph-mon[128712]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.3 KiB/s wr, 10 op/s 2023-11-16T16:50:23.955 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.953+0000 7f70999ff700 1 -- 172.21.15.2:0/524157723 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70940746a0 msgr2=0x7f7094072d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:23.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.953+0000 7f70999ff700 1 --2- 172.21.15.2:0/524157723 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70940746a0 0x7f7094072d20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7084009990 tx=0x7f7084009ca0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.954+0000 7f70999ff700 1 -- 172.21.15.2:0/524157723 shutdown_connections 2023-11-16T16:50:23.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.954+0000 7f70999ff700 1 --2- 172.21.15.2:0/524157723 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f709410ce50 0x7f709410f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.954+0000 7f70999ff700 1 --2- 172.21.15.2:0/524157723 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70940746a0 0x7f7094072d20 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.954+0000 7f70999ff700 1 --2- 172.21.15.2:0/524157723 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7094074b00 0x7f7094073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.954+0000 7f70999ff700 1 -- 172.21.15.2:0/524157723 >> 172.21.15.2:0/524157723 conn(0x7f7094077550 msgr2=0x7f7094077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:23.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.954+0000 7f70999ff700 1 -- 172.21.15.2:0/524157723 shutdown_connections 2023-11-16T16:50:23.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.954+0000 7f70999ff700 1 -- 172.21.15.2:0/524157723 wait complete. 2023-11-16T16:50:23.957 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.955+0000 7f70999ff700 1 Processor -- start 2023-11-16T16:50:23.957 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.955+0000 7f70999ff700 1 -- start start 2023-11-16T16:50:23.959 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.956+0000 7f70999ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70940746a0 0x7f70941985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:23.959 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.956+0000 7f70999ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7094074b00 0x7f7094198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:23.959 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.956+0000 7f70999ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f709410ce50 0x7f709419cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:23.959 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.956+0000 7f70999ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f709419d5c0 con 0x7f709410ce50 2023-11-16T16:50:23.959 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.956+0000 7f70999ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f709419d730 con 0x7f70940746a0 2023-11-16T16:50:23.959 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.956+0000 7f70999ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f709419d8f0 con 0x7f7094074b00 2023-11-16T16:50:23.960 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.956+0000 7f70927fc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7094074b00 0x7f7094198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:23.960 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.956+0000 7f70937fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f709410ce50 0x7f709419cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:23.960 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.956+0000 7f70927fc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7094074b00 0x7f7094198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:35548/0 (socket says 172.21.15.2:35548) 2023-11-16T16:50:23.960 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.956+0000 7f70927fc700 1 -- 172.21.15.2:0/2889569807 learned_addr learned my addr 172.21.15.2:0/2889569807 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:23.960 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.956+0000 7f7092ffd700 1 --2- 172.21.15.2:0/2889569807 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70940746a0 0x7f70941985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:23.960 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.957+0000 7f70927fc700 1 -- 172.21.15.2:0/2889569807 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70940746a0 msgr2=0x7f70941985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:23.960 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.957+0000 7f70927fc700 1 --2- 172.21.15.2:0/2889569807 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70940746a0 0x7f70941985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.960 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.957+0000 7f70927fc700 1 -- 172.21.15.2:0/2889569807 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f709410ce50 msgr2=0x7f709419cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:23.960 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.957+0000 7f70927fc700 1 --2- 172.21.15.2:0/2889569807 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f709410ce50 0x7f709419cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.960 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.957+0000 7f70927fc700 1 -- 172.21.15.2:0/2889569807 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70840095f0 con 0x7f7094074b00 2023-11-16T16:50:23.960 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.957+0000 7f70937fe700 1 --2- 172.21.15.2:0/2889569807 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f709410ce50 0x7f709419cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:23.960 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.957+0000 7f70927fc700 1 --2- 172.21.15.2:0/2889569807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7094074b00 0x7f7094198b20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f708800e9a0 tx=0x7f708800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:23.960 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.957+0000 7f70989fd700 1 -- 172.21.15.2:0/2889569807 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7088009790 con 0x7f7094074b00 2023-11-16T16:50:23.961 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.957+0000 7f70989fd700 1 -- 172.21.15.2:0/2889569807 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7088010430 con 0x7f7094074b00 2023-11-16T16:50:23.961 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.957+0000 7f70999ff700 1 -- 172.21.15.2:0/2889569807 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7094112590 con 0x7f7094074b00 2023-11-16T16:50:23.961 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.957+0000 7f70989fd700 1 -- 172.21.15.2:0/2889569807 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7088018680 con 0x7f7094074b00 2023-11-16T16:50:23.961 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.957+0000 7f70999ff700 1 -- 172.21.15.2:0/2889569807 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7094112400 con 0x7f7094074b00 2023-11-16T16:50:23.962 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.961+0000 7f70989fd700 1 -- 172.21.15.2:0/2889569807 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7f70880105a0 con 0x7f7094074b00 2023-11-16T16:50:23.964 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.962+0000 7f70ad2adb80 1 Processor -- start 2023-11-16T16:50:23.965 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.962+0000 7f70ad2adb80 1 -- start start 2023-11-16T16:50:23.965 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.962+0000 7f70ad2adb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea869fd6b0 0x55ea86ba71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:23.965 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.962+0000 7f70ad2adb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea86ba7720 0x55ea86ba1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:23.965 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.962+0000 7f70ad2adb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7b90 0x55ea86ba1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:23.965 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.962+0000 7f70ad2adb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea86aa7000 con 0x55ea869fd6b0 2023-11-16T16:50:23.965 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.962+0000 7f70ad2adb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea86af9840 con 0x55ea86ba7b90 2023-11-16T16:50:23.965 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.962+0000 7f70ad2adb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea86af2cd0 con 0x55ea86ba7720 2023-11-16T16:50:23.965 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.962+0000 7f70989fd700 1 --2- 172.21.15.2:0/2889569807 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f707c0a13e0 0x7f707c0a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:23.965 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.963+0000 7f7092ffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea86ba7720 0x55ea86ba1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:23.965 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.963+0000 7f70989fd700 1 -- 172.21.15.2:0/2889569807 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f7088014070 con 0x7f7094074b00 2023-11-16T16:50:23.965 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.963+0000 7f7092ffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea86ba7720 0x55ea86ba1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:35554/0 (socket says 172.21.15.2:35554) 2023-11-16T16:50:23.966 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.963+0000 7f70937fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea869fd6b0 0x55ea86ba71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:23.966 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.963+0000 7f7092ffd700 1 -- 172.21.15.2:0/1367347619 learned_addr learned my addr 172.21.15.2:0/1367347619 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:23.966 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.963+0000 7f70927fc700 1 --2- 172.21.15.2:0/1367347619 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7b90 0x55ea86ba1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:23.966 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.963+0000 7f70937fe700 1 --2- 172.21.15.2:0/2889569807 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f707c0a13e0 0x7f707c0a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:23.966 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.963+0000 7f7092ffd700 1 -- 172.21.15.2:0/1367347619 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7b90 msgr2=0x55ea86ba1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:23.966 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.963+0000 7f7092ffd700 1 --2- 172.21.15.2:0/1367347619 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7b90 0x55ea86ba1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.966 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.963+0000 7f7092ffd700 1 -- 172.21.15.2:0/1367347619 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea869fd6b0 msgr2=0x55ea86ba71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:23.966 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.963+0000 7f7092ffd700 1 --2- 172.21.15.2:0/1367347619 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea869fd6b0 0x55ea86ba71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.966 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.963+0000 7f7092ffd700 1 -- 172.21.15.2:0/1367347619 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea869745d0 con 0x55ea86ba7720 2023-11-16T16:50:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.963+0000 7f70937fe700 1 --2- 172.21.15.2:0/2889569807 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f707c0a13e0 0x7f707c0a3890 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f708c009fd0 tx=0x7f708c009040 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.963+0000 7f7092ffd700 1 --2- 172.21.15.2:0/1367347619 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea86ba7720 0x55ea86ba1560 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7084005ea0 tx=0x7f7084005ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=319504a916768a02 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.964+0000 7f7079ffb700 1 -- 172.21.15.2:0/1367347619 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708401d070 con 0x55ea86ba7720 2023-11-16T16:50:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.964+0000 7f7079ffb700 1 -- 172.21.15.2:0/1367347619 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f708401e040 con 0x55ea86ba7720 2023-11-16T16:50:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.964+0000 7f7079ffb700 1 -- 172.21.15.2:0/1367347619 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7084004880 con 0x55ea86ba7720 2023-11-16T16:50:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.965+0000 7f70ad2adb80 1 -- 172.21.15.2:0/1367347619 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea86ba7720 msgr2=0x55ea86ba1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.965+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/1367347619 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea86ba7720 0x55ea86ba1560 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7084005ea0 tx=0x7f7084005ed0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.965+0000 7f70ad2adb80 1 -- 172.21.15.2:0/1367347619 shutdown_connections 2023-11-16T16:50:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.965+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/1367347619 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7b90 0x55ea86ba1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.965+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/1367347619 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea869fd6b0 0x55ea86ba71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.965+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/1367347619 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea86ba7720 0x55ea86ba1560 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.965+0000 7f70ad2adb80 1 -- 172.21.15.2:0/1367347619 >> 172.21.15.2:0/1367347619 conn(0x55ea86b96f50 msgr2=0x55ea86b9c270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.965+0000 7f70ad2adb80 1 -- 172.21.15.2:0/1367347619 shutdown_connections 2023-11-16T16:50:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.965+0000 7f70ad2adb80 1 -- 172.21.15.2:0/1367347619 wait complete. 2023-11-16T16:50:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.966+0000 7f70ad2adb80 1 -- start start 2023-11-16T16:50:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.966+0000 7f70ad2adb80 1 Processor -- start 2023-11-16T16:50:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.966+0000 7f70ad2adb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea869fd6b0 0x55ea86bb09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.966+0000 7f70ad2adb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7720 0x55ea86bb0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.966+0000 7f70ad2adb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea86ba7b90 0x55ea86baac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.966+0000 7f70ad2adb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea86a94880 con 0x55ea86ba7b90 2023-11-16T16:50:23.970 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.966+0000 7f70ad2adb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea86a94520 con 0x55ea86ba7720 2023-11-16T16:50:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.966+0000 7f70ad2adb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea869768c0 con 0x55ea869fd6b0 2023-11-16T16:50:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.966+0000 7f70ad2adb80 10 client.0 _ll_register_callbacks cb 0x55ea86ab2720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.967+0000 7f70937fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea86ba7b90 0x55ea86baac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.967+0000 7f70937fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea86ba7b90 0x55ea86baac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35640/0 (socket says 172.21.15.2:35640) 2023-11-16T16:50:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.967+0000 7f70927fc700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7720 0x55ea86bb0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.967+0000 7f70937fe700 1 -- 172.21.15.2:0/1349977011 learned_addr learned my addr 172.21.15.2:0/1349977011 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.967+0000 7f7092ffd700 1 --2- 172.21.15.2:0/1349977011 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea869fd6b0 0x55ea86bb09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.967+0000 7f70937fe700 1 -- 172.21.15.2:0/1349977011 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea869fd6b0 msgr2=0x55ea86bb09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.967+0000 7f70937fe700 1 --2- 172.21.15.2:0/1349977011 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea869fd6b0 0x55ea86bb09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.967+0000 7f70937fe700 1 -- 172.21.15.2:0/1349977011 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7720 msgr2=0x55ea86bb0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.967+0000 7f70937fe700 1 --2- 172.21.15.2:0/1349977011 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7720 0x55ea86bb0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.967+0000 7f70937fe700 1 -- 172.21.15.2:0/1349977011 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f709410fb60 con 0x55ea86ba7b90 2023-11-16T16:50:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.967+0000 7f70927fc700 1 --2- 172.21.15.2:0/1349977011 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7720 0x55ea86bb0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.967+0000 7f7092ffd700 1 --2- 172.21.15.2:0/1349977011 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea869fd6b0 0x55ea86bb09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.967+0000 7f70937fe700 1 --2- 172.21.15.2:0/1349977011 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea86ba7b90 0x55ea86baac60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f708c009550 tx=0x7f708c009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e278f5855f5a18 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.967+0000 7f7072ffd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:50:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.967+0000 7f7072ffd700 1 -- 172.21.15.2:0/1349977011 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708c029070 con 0x55ea86ba7b90 2023-11-16T16:50:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.968+0000 7f70ad2adb80 1 -- 172.21.15.2:0/1349977011 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ea86b94710 con 0x55ea86ba7b90 2023-11-16T16:50:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.968+0000 7f7072ffd700 1 -- 172.21.15.2:0/1349977011 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f708c004660 con 0x55ea86ba7b90 2023-11-16T16:50:23.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.968+0000 7f7072ffd700 1 -- 172.21.15.2:0/1349977011 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f708c003e50 con 0x55ea86ba7b90 2023-11-16T16:50:23.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.968+0000 7f70ad2adb80 10 client.15480 fetch_fsmap learned FSMap version 28 2023-11-16T16:50:23.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.968+0000 7f70ad2adb80 1 -- 172.21.15.2:0/1349977011 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ea86ab9130 con 0x55ea86ba7b90 2023-11-16T16:50:23.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.968+0000 7f7072ffd700 1 -- 172.21.15.2:0/1349977011 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ea86b94710 con 0x55ea86ba7b90 2023-11-16T16:50:23.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.968+0000 7f7072ffd700 1 -- 172.21.15.2:0/1349977011 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f708c02c020 con 0x55ea86ba7b90 2023-11-16T16:50:23.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.968+0000 7f70ad2adb80 10 client.15480 fetch_fsmap finished waiting for FSMap version 28 2023-11-16T16:50:23.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.968+0000 7f70ad2adb80 10 client.15480 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:23.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.968+0000 7f70ad2adb80 20 client.15480 populate_metadata read hostname 'smithi002' 2023-11-16T16:50:23.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.968+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/1349977011 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55ea86bbaf40 0x55ea86bbd350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:23.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.968+0000 7f70ad2adb80 4 client.15480 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:50:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.968+0000 7f70ad2adb80 1 -- 172.21.15.2:0/1349977011 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ea86ab5040 con 0x55ea86bbaf40 2023-11-16T16:50:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.969+0000 7f7092ffd700 1 --2- 172.21.15.2:0/1349977011 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55ea86bbaf40 0x55ea86bbd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.969+0000 7f7092ffd700 1 --2- 172.21.15.2:0/1349977011 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55ea86bbaf40 0x55ea86bbd350 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1af650eed69e0f65 server_cookie=6f44c77cdbf85b83 in_seq=0 out_seq=0 2023-11-16T16:50:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.969+0000 7f7072ffd700 10 client.15480 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:50:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.970+0000 7f7072ffd700 1 -- 172.21.15.2:0/1349977011 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ea86ab5040 con 0x55ea86bbaf40 2023-11-16T16:50:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.970+0000 7f7072ffd700 10 client.15480 handle_command_reply: tid=0 2023-11-16T16:50:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.970+0000 7f70ad2adb80 1 client.15480 shutdown 2023-11-16T16:50:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.970+0000 7f70ad2adb80 1 -- 172.21.15.2:0/1349977011 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea86ba7b90 msgr2=0x55ea86baac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.970+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/1349977011 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea86ba7b90 0x55ea86baac60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f708c009550 tx=0x7f708c009920 comp rx=0 tx=0).stop 2023-11-16T16:50:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.970+0000 7f70ad2adb80 1 -- 172.21.15.2:0/1349977011 shutdown_connections 2023-11-16T16:50:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.970+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/1349977011 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55ea86bbaf40 0x55ea86bbd350 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.970+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/1349977011 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea869fd6b0 0x55ea86bb09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.970+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/1349977011 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea86ba7b90 0x55ea86baac60 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.970+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/1349977011 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7720 0x55ea86bb0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.970+0000 7f7072ffd700 0 client.15480 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:50:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.970+0000 7f70ad2adb80 1 -- 172.21.15.2:0/1349977011 >> 172.21.15.2:0/1349977011 conn(0x55ea86b96f50 msgr2=0x55ea86b9b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.970+0000 7f70ad2adb80 1 -- 172.21.15.2:0/1349977011 shutdown_connections 2023-11-16T16:50:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.970+0000 7f70ad2adb80 1 -- 172.21.15.2:0/1349977011 wait complete. 2023-11-16T16:50:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.971+0000 7f70ad2adb80 20 client.15480 trim_cache size 0 max 16384 2023-11-16T16:50:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.986+0000 7f70ad2adb80 1 Processor -- start 2023-11-16T16:50:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.986+0000 7f70ad2adb80 1 -- start start 2023-11-16T16:50:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.986+0000 7f70ad2adb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea869fd6b0 0x55ea86ba87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.986+0000 7f70ad2adb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7720 0x55ea86b9bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.986+0000 7f70ad2adb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea86ba7b90 0x55ea86ba6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.986+0000 7f70ad2adb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea86ba9270 con 0x55ea869fd6b0 2023-11-16T16:50:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.986+0000 7f70ad2adb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea86ba5910 con 0x55ea86ba7720 2023-11-16T16:50:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.986+0000 7f70ad2adb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea86bb77f0 con 0x55ea86ba7b90 2023-11-16T16:50:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.987+0000 7f7092ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea869fd6b0 0x55ea86ba87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.987+0000 7f7092ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea869fd6b0 0x55ea86ba87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35648/0 (socket says 172.21.15.2:35648) 2023-11-16T16:50:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.987+0000 7f7092ffd700 1 -- 172.21.15.2:0/39649176 learned_addr learned my addr 172.21.15.2:0/39649176 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.987+0000 7f70927fc700 1 --2- 172.21.15.2:0/39649176 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7720 0x55ea86b9bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.987+0000 7f70937fe700 1 --2- 172.21.15.2:0/39649176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea86ba7b90 0x55ea86ba6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.987+0000 7f7092ffd700 1 -- 172.21.15.2:0/39649176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea86ba7b90 msgr2=0x55ea86ba6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.987+0000 7f7092ffd700 1 --2- 172.21.15.2:0/39649176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea86ba7b90 0x55ea86ba6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.987+0000 7f7092ffd700 1 -- 172.21.15.2:0/39649176 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7720 msgr2=0x55ea86b9bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.987+0000 7f7092ffd700 1 --2- 172.21.15.2:0/39649176 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7720 0x55ea86b9bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.987+0000 7f7092ffd700 1 -- 172.21.15.2:0/39649176 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea86ab8e80 con 0x55ea869fd6b0 2023-11-16T16:50:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.987+0000 7f70937fe700 1 --2- 172.21.15.2:0/39649176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea86ba7b90 0x55ea86ba6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.987+0000 7f7092ffd700 1 --2- 172.21.15.2:0/39649176 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea869fd6b0 0x55ea86ba87f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7084009990 tx=0x7f7084005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eedf816a6bd8af72 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.988+0000 7f707a7fc700 1 -- 172.21.15.2:0/39649176 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708401d040 con 0x55ea869fd6b0 2023-11-16T16:50:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.988+0000 7f707a7fc700 1 -- 172.21.15.2:0/39649176 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7084005950 con 0x55ea869fd6b0 2023-11-16T16:50:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.988+0000 7f707a7fc700 1 -- 172.21.15.2:0/39649176 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7084032c10 con 0x55ea869fd6b0 2023-11-16T16:50:23.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.988+0000 7f70ad2adb80 1 -- 172.21.15.2:0/39649176 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea869fd6b0 msgr2=0x55ea86ba87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.988+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/39649176 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea869fd6b0 0x55ea86ba87f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7084009990 tx=0x7f7084005fb0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.988+0000 7f70ad2adb80 1 -- 172.21.15.2:0/39649176 shutdown_connections 2023-11-16T16:50:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.988+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/39649176 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea86ba7b90 0x55ea86ba6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.988+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/39649176 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea869fd6b0 0x55ea86ba87f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.988+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/39649176 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7720 0x55ea86b9bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.988+0000 7f70ad2adb80 1 -- 172.21.15.2:0/39649176 >> 172.21.15.2:0/39649176 conn(0x55ea86b96f50 msgr2=0x55ea86ba5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.988+0000 7f70ad2adb80 1 -- 172.21.15.2:0/39649176 shutdown_connections 2023-11-16T16:50:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.988+0000 7f70ad2adb80 1 -- 172.21.15.2:0/39649176 wait complete. 2023-11-16T16:50:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.989+0000 7f70ad2adb80 1 -- start start 2023-11-16T16:50:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.989+0000 7f70ad2adb80 1 Processor -- start 2023-11-16T16:50:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.989+0000 7f70ad2adb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea869fd6b0 0x55ea86bae580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:23.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.989+0000 7f70ad2adb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea86ba7720 0x55ea86baeac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:23.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.989+0000 7f70ad2adb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7b90 0x55ea86baf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:23.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.989+0000 7f70ad2adb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea86c0de30 con 0x55ea86ba7720 2023-11-16T16:50:23.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.989+0000 7f70ad2adb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea86c0dfa0 con 0x55ea86ba7b90 2023-11-16T16:50:23.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.989+0000 7f70ad2adb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea86c0e160 con 0x55ea869fd6b0 2023-11-16T16:50:23.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.989+0000 7f70ad2adb80 10 client.0 _ll_register_callbacks cb 0x7f708c009270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:23.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.990+0000 7f70927fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea86ba7720 0x55ea86baeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:23.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.990+0000 7f70927fc700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea86ba7720 0x55ea86baeac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35654/0 (socket says 172.21.15.2:35654) 2023-11-16T16:50:23.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.990+0000 7f70937fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7b90 0x55ea86baf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:23.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.990+0000 7f70927fc700 1 -- 172.21.15.2:0/3448428562 learned_addr learned my addr 172.21.15.2:0/3448428562 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:23.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.990+0000 7f7092ffd700 1 --2- 172.21.15.2:0/3448428562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea869fd6b0 0x55ea86bae580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:23.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.990+0000 7f70937fe700 1 -- 172.21.15.2:0/3448428562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea869fd6b0 msgr2=0x55ea86bae580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:23.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.990+0000 7f70937fe700 1 --2- 172.21.15.2:0/3448428562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea869fd6b0 0x55ea86bae580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.990+0000 7f70937fe700 1 -- 172.21.15.2:0/3448428562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea86ba7720 msgr2=0x55ea86baeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:23.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.990+0000 7f70937fe700 1 --2- 172.21.15.2:0/3448428562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea86ba7720 0x55ea86baeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.990+0000 7f70937fe700 1 -- 172.21.15.2:0/3448428562 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ea869745d0 con 0x55ea86ba7b90 2023-11-16T16:50:23.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.990+0000 7f70927fc700 1 --2- 172.21.15.2:0/3448428562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea86ba7720 0x55ea86baeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:23.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.990+0000 7f70937fe700 1 --2- 172.21.15.2:0/3448428562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7b90 0x55ea86baf000 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f708c009ec0 tx=0x7f708c01b530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1286fd55eea89184 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:23.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.990+0000 7f7072ffd700 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2023-11-16T16:50:23.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.991+0000 7f7072ffd700 1 -- 172.21.15.2:0/3448428562 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708c029070 con 0x55ea86ba7b90 2023-11-16T16:50:23.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.991+0000 7f70ad2adb80 1 -- 172.21.15.2:0/3448428562 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f708401e040 con 0x55ea86ba7b90 2023-11-16T16:50:23.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.991+0000 7f7072ffd700 1 -- 172.21.15.2:0/3448428562 <== mon.2 v2:172.21.15.141:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f708c003e50 con 0x55ea86ba7b90 2023-11-16T16:50:23.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.991+0000 7f7072ffd700 1 -- 172.21.15.2:0/3448428562 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f708c01b5a0 con 0x55ea86ba7b90 2023-11-16T16:50:23.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.991+0000 7f70ad2adb80 10 client.25517 fetch_fsmap learned FSMap version 28 2023-11-16T16:50:23.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.991+0000 7f70ad2adb80 1 -- 172.21.15.2:0/3448428562 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ea86909c00 con 0x55ea86ba7b90 2023-11-16T16:50:23.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.991+0000 7f7072ffd700 1 -- 172.21.15.2:0/3448428562 <== mon.2 v2:172.21.15.141:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f708401e040 con 0x55ea86ba7b90 2023-11-16T16:50:23.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.991+0000 7f7072ffd700 1 -- 172.21.15.2:0/3448428562 <== mon.2 v2:172.21.15.141:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f708c02c070 con 0x55ea86ba7b90 2023-11-16T16:50:23.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.992+0000 7f70ad2adb80 10 client.25517 fetch_fsmap finished waiting for FSMap version 28 2023-11-16T16:50:23.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.992+0000 7f70ad2adb80 10 client.25517 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:23.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.992+0000 7f70ad2adb80 20 client.25517 populate_metadata read hostname 'smithi002' 2023-11-16T16:50:23.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.992+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/3448428562 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55ea86c0f7e0 0x55ea86c11bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:23.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.992+0000 7f70ad2adb80 4 client.25517 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:50:23.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.992+0000 7f70ad2adb80 1 -- 172.21.15.2:0/3448428562 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ea86ac1600 con 0x55ea86c0f7e0 2023-11-16T16:50:23.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.992+0000 7f7092ffd700 1 --2- 172.21.15.2:0/3448428562 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55ea86c0f7e0 0x55ea86c11bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:23.998 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.992+0000 7f7072ffd700 10 client.25517 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:50:23.998 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.992+0000 7f7092ffd700 1 --2- 172.21.15.2:0/3448428562 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55ea86c0f7e0 0x55ea86c11bf0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fc42e91b28ba572 server_cookie=878e85459d676d2e in_seq=0 out_seq=0 2023-11-16T16:50:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.996+0000 7f7072ffd700 1 -- 172.21.15.2:0/3448428562 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ea86ac1600 con 0x55ea86c0f7e0 2023-11-16T16:50:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.996+0000 7f7072ffd700 10 client.25517 handle_command_reply: tid=0 2023-11-16T16:50:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.996+0000 7f70ad2adb80 1 client.25517 shutdown 2023-11-16T16:50:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.996+0000 7f70ad2adb80 1 -- 172.21.15.2:0/3448428562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7b90 msgr2=0x55ea86baf000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.996+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/3448428562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7b90 0x55ea86baf000 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f708c009ec0 tx=0x7f708c01b530 comp rx=0 tx=0).stop 2023-11-16T16:50:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.997+0000 7f70ad2adb80 1 -- 172.21.15.2:0/3448428562 shutdown_connections 2023-11-16T16:50:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.997+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/3448428562 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55ea86c0f7e0 0x55ea86c11bf0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.997+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/3448428562 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ea869fd6b0 0x55ea86bae580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.997+0000 7f7072ffd700 0 client.25517 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:50:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.997+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/3448428562 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55ea86ba7b90 0x55ea86baf000 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.997+0000 7f70ad2adb80 1 --2- 172.21.15.2:0/3448428562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ea86ba7720 0x55ea86baeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:23.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.997+0000 7f70ad2adb80 1 -- 172.21.15.2:0/3448428562 >> 172.21.15.2:0/3448428562 conn(0x55ea86b96f50 msgr2=0x55ea86ba5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:24.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.997+0000 7f70ad2adb80 1 -- 172.21.15.2:0/3448428562 shutdown_connections 2023-11-16T16:50:24.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.997+0000 7f70ad2adb80 1 -- 172.21.15.2:0/3448428562 wait complete. 2023-11-16T16:50:24.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.997+0000 7f70ad2adb80 20 client.25517 trim_cache size 0 max 16384 2023-11-16T16:50:24.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:50:24.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:50:24.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:50:24.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:50:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.997+0000 7f70999ff700 1 -- 172.21.15.2:0/2889569807 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f707c0a13e0 msgr2=0x7f707c0a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.997+0000 7f70999ff700 1 --2- 172.21.15.2:0/2889569807 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f707c0a13e0 0x7f707c0a3890 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f708c009fd0 tx=0x7f708c009040 comp rx=0 tx=0).stop 2023-11-16T16:50:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.997+0000 7f70999ff700 1 -- 172.21.15.2:0/2889569807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7094074b00 msgr2=0x7f7094198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.997+0000 7f70999ff700 1 --2- 172.21.15.2:0/2889569807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7094074b00 0x7f7094198b20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f708800e9a0 tx=0x7f708800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.998+0000 7f70999ff700 1 -- 172.21.15.2:0/2889569807 shutdown_connections 2023-11-16T16:50:24.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.998+0000 7f70999ff700 1 --2- 172.21.15.2:0/2889569807 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f707c0a13e0 0x7f707c0a3890 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.998+0000 7f70999ff700 1 --2- 172.21.15.2:0/2889569807 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f709410ce50 0x7f709419cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.998+0000 7f70999ff700 1 --2- 172.21.15.2:0/2889569807 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f70940746a0 0x7f70941985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.998+0000 7f70999ff700 1 --2- 172.21.15.2:0/2889569807 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7094074b00 0x7f7094198b20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:24.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.998+0000 7f70999ff700 1 -- 172.21.15.2:0/2889569807 >> 172.21.15.2:0/2889569807 conn(0x7f7094077550 msgr2=0x7f7094104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:24.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.998+0000 7f70999ff700 1 -- 172.21.15.2:0/2889569807 shutdown_connections 2023-11-16T16:50:24.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:23.998+0000 7f70999ff700 1 -- 172.21.15.2:0/2889569807 wait complete. 2023-11-16T16:50:25.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:25 smithi155 ceph-mon[131607]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-11-16T16:50:25.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:25 smithi002 ceph-mon[128712]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-11-16T16:50:26.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:25 smithi141 ceph-mon[131184]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-11-16T16:50:27.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:26 smithi141 ceph-mon[131184]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2023-11-16T16:50:27.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:26 smithi155 ceph-mon[131607]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2023-11-16T16:50:27.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:26 smithi002 ceph-mon[128712]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2023-11-16T16:50:29.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:28 smithi141 ceph-mon[131184]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2023-11-16T16:50:29.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:28 smithi155 ceph-mon[131607]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2023-11-16T16:50:29.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:28 smithi002 ceph-mon[128712]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2023-11-16T16:50:31.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:30 smithi141 ceph-mon[131184]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2023-11-16T16:50:31.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:30 smithi155 ceph-mon[131607]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2023-11-16T16:50:31.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:30 smithi002 ceph-mon[128712]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2023-11-16T16:50:33.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:32 smithi141 ceph-mon[131184]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2023-11-16T16:50:33.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:32 smithi155 ceph-mon[131607]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2023-11-16T16:50:33.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:32 smithi002 ceph-mon[128712]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2023-11-16T16:50:35.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:35 smithi155 ceph-mon[131607]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2023-11-16T16:50:35.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:35 smithi002 ceph-mon[128712]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2023-11-16T16:50:36.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:35 smithi141 ceph-mon[131184]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2023-11-16T16:50:37.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:36 smithi141 ceph-mon[131184]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1023 B/s wr, 8 op/s 2023-11-16T16:50:37.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:36 smithi155 ceph-mon[131607]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1023 B/s wr, 8 op/s 2023-11-16T16:50:37.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:36 smithi002 ceph-mon[128712]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1023 B/s wr, 8 op/s 2023-11-16T16:50:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.160+0000 7f7e7a9ff700 1 -- 172.21.15.2:0/3086915993 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e7410cb70 msgr2=0x7f7e7410ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.160+0000 7f7e7a9ff700 1 --2- 172.21.15.2:0/3086915993 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e7410cb70 0x7f7e7410ef50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00b0a0 tx=0x7f7e6c00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.160+0000 7f7e7a9ff700 1 -- 172.21.15.2:0/3086915993 shutdown_connections 2023-11-16T16:50:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.160+0000 7f7e7a9ff700 1 --2- 172.21.15.2:0/3086915993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e74072e30 0x7f7e74073200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.160+0000 7f7e7a9ff700 1 --2- 172.21.15.2:0/3086915993 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e7410cb70 0x7f7e7410ef50 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.160+0000 7f7e7a9ff700 1 --2- 172.21.15.2:0/3086915993 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e74073740 0x7f7e7410c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.160+0000 7f7e7a9ff700 1 -- 172.21.15.2:0/3086915993 >> 172.21.15.2:0/3086915993 conn(0x7f7e74078210 msgr2=0x7f7e74078610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.160+0000 7f7e7a9ff700 1 -- 172.21.15.2:0/3086915993 shutdown_connections 2023-11-16T16:50:37.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.160+0000 7f7e7a9ff700 1 -- 172.21.15.2:0/3086915993 wait complete. 2023-11-16T16:50:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.161+0000 7f7e7a9ff700 1 Processor -- start 2023-11-16T16:50:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.161+0000 7f7e7a9ff700 1 -- start start 2023-11-16T16:50:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.162+0000 7f7e7a9ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e74072e30 0x7f7e74198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.162+0000 7f7e7a9ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e74073740 0x7f7e74198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.162+0000 7f7e7a9ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e7410cb70 0x7f7e7419bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:37.164 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.162+0000 7f7e7a9ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7419c5f0 con 0x7f7e74073740 2023-11-16T16:50:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.162+0000 7f7e7a9ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7419c760 con 0x7f7e7410cb70 2023-11-16T16:50:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.162+0000 7f7e7a9ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7419c920 con 0x7f7e74072e30 2023-11-16T16:50:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.162+0000 7f7e737fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e74073740 0x7f7e74198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.162+0000 7f7e73fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e74072e30 0x7f7e74198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.162+0000 7f7e73fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e74072e30 0x7f7e74198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:55940/0 (socket says 172.21.15.2:55940) 2023-11-16T16:50:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.162+0000 7f7e73fff700 1 -- 172.21.15.2:0/2604790223 learned_addr learned my addr 172.21.15.2:0/2604790223 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.162+0000 7f7e78f9c700 1 --2- 172.21.15.2:0/2604790223 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e7410cb70 0x7f7e7419bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.162+0000 7f7e73fff700 1 -- 172.21.15.2:0/2604790223 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e7410cb70 msgr2=0x7f7e7419bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.162+0000 7f7e73fff700 1 --2- 172.21.15.2:0/2604790223 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e7410cb70 0x7f7e7419bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.162+0000 7f7e73fff700 1 -- 172.21.15.2:0/2604790223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e74073740 msgr2=0x7f7e74198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.162+0000 7f7e73fff700 1 --2- 172.21.15.2:0/2604790223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e74073740 0x7f7e74198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.162+0000 7f7e73fff700 1 -- 172.21.15.2:0/2604790223 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e6c009d00 con 0x7f7e74072e30 2023-11-16T16:50:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.162+0000 7f7e737fe700 1 --2- 172.21.15.2:0/2604790223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e74073740 0x7f7e74198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:37.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.163+0000 7f7e73fff700 1 --2- 172.21.15.2:0/2604790223 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e74072e30 0x7f7e74198700 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400e9a0 tx=0x7f7e6400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:37.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.163+0000 7f7e717fa700 1 -- 172.21.15.2:0/2604790223 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64009790 con 0x7f7e74072e30 2023-11-16T16:50:37.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.163+0000 7f7e7a9ff700 1 -- 172.21.15.2:0/2604790223 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e741124f0 con 0x7f7e74072e30 2023-11-16T16:50:37.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.163+0000 7f7e717fa700 1 -- 172.21.15.2:0/2604790223 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e64010430 con 0x7f7e74072e30 2023-11-16T16:50:37.167 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.163+0000 7f7e717fa700 1 -- 172.21.15.2:0/2604790223 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64018680 con 0x7f7e74072e30 2023-11-16T16:50:37.167 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.163+0000 7f7e7a9ff700 1 -- 172.21.15.2:0/2604790223 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e74112360 con 0x7f7e74072e30 2023-11-16T16:50:37.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.167+0000 7f7e717fa700 1 -- 172.21.15.2:0/2604790223 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7f7e640105a0 con 0x7f7e74072e30 2023-11-16T16:50:37.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.168+0000 7f7e8e3eeb80 1 Processor -- start 2023-11-16T16:50:37.170 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.168+0000 7f7e8e3eeb80 1 -- start start 2023-11-16T16:50:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.168+0000 7f7e8e3eeb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b7d76b0 0x559e9b9811e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.168+0000 7f7e8e3eeb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b981720 0x559e9b97b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.168+0000 7f7e8e3eeb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b981b90 0x559e9b97bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.169+0000 7f7e8e3eeb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e9b881000 con 0x559e9b981b90 2023-11-16T16:50:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.169+0000 7f7e8e3eeb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e9b8d3840 con 0x559e9b981720 2023-11-16T16:50:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.169+0000 7f7e8e3eeb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e9b8cccd0 con 0x559e9b7d76b0 2023-11-16T16:50:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.169+0000 7f7e717fa700 1 --2- 172.21.15.2:0/2604790223 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f7e5c0a13e0 0x7f7e5c0a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.169+0000 7f7e717fa700 1 -- 172.21.15.2:0/2604790223 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f7e64014070 con 0x7f7e74072e30 2023-11-16T16:50:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.169+0000 7f7e737fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b981b90 0x559e9b97bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.169+0000 7f7e73fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b981720 0x559e9b97b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.169+0000 7f7e73fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b981720 0x559e9b97b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:56092/0 (socket says 172.21.15.2:56092) 2023-11-16T16:50:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.169+0000 7f7e73fff700 1 -- 172.21.15.2:0/864367281 learned_addr learned my addr 172.21.15.2:0/864367281 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.169+0000 7f7e737fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b981b90 0x559e9b97bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40568/0 (socket says 172.21.15.2:40568) 2023-11-16T16:50:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.169+0000 7f7e78f9c700 1 --2- 172.21.15.2:0/2604790223 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f7e5c0a13e0 0x7f7e5c0a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.169+0000 7f7e78f9c700 1 --2- 172.21.15.2:0/864367281 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b7d76b0 0x559e9b9811e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.169+0000 7f7e73fff700 1 -- 172.21.15.2:0/864367281 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b7d76b0 msgr2=0x559e9b9811e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.170+0000 7f7e73fff700 1 --2- 172.21.15.2:0/864367281 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b7d76b0 0x559e9b9811e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.170+0000 7f7e73fff700 1 -- 172.21.15.2:0/864367281 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b981b90 msgr2=0x559e9b97bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:37.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.170+0000 7f7e73fff700 1 --2- 172.21.15.2:0/864367281 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b981b90 0x559e9b97bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.170+0000 7f7e73fff700 1 -- 172.21.15.2:0/864367281 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e9b74e5d0 con 0x559e9b981720 2023-11-16T16:50:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.170+0000 7f7e73fff700 1 --2- 172.21.15.2:0/864367281 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b981720 0x559e9b97b560 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400e970 tx=0x7f7e64099970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d872297cfc58088 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.170+0000 7f7e78f9c700 1 --2- 172.21.15.2:0/2604790223 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f7e5c0a13e0 0x7f7e5c0a3890 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00b780 tx=0x7f7e6c0099e0 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.170+0000 7f7e5a7fc700 1 -- 172.21.15.2:0/864367281 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6409e040 con 0x559e9b981720 2023-11-16T16:50:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.170+0000 7f7e5a7fc700 1 -- 172.21.15.2:0/864367281 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e640a2460 con 0x559e9b981720 2023-11-16T16:50:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.170+0000 7f7e5a7fc700 1 -- 172.21.15.2:0/864367281 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64099700 con 0x559e9b981720 2023-11-16T16:50:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.171+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/864367281 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b981720 msgr2=0x559e9b97b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.171+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/864367281 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b981720 0x559e9b97b560 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400e970 tx=0x7f7e64099970 comp rx=0 tx=0).stop 2023-11-16T16:50:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.171+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/864367281 shutdown_connections 2023-11-16T16:50:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.171+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/864367281 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b7d76b0 0x559e9b9811e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.171+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/864367281 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b981b90 0x559e9b97bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.171+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/864367281 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b981720 0x559e9b97b560 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.171+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/864367281 >> 172.21.15.2:0/864367281 conn(0x559e9b970f50 msgr2=0x559e9b976270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.171+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/864367281 shutdown_connections 2023-11-16T16:50:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.171+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/864367281 wait complete. 2023-11-16T16:50:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.172+0000 7f7e8e3eeb80 1 -- start start 2023-11-16T16:50:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.172+0000 7f7e8e3eeb80 1 Processor -- start 2023-11-16T16:50:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.173+0000 7f7e8e3eeb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b7d76b0 0x559e9b98a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.173+0000 7f7e8e3eeb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b981720 0x559e9b98af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.173+0000 7f7e8e3eeb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b981b90 0x559e9b984c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.173+0000 7f7e8e3eeb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e9b86e880 con 0x559e9b981b90 2023-11-16T16:50:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.173+0000 7f7e8e3eeb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e9b86e520 con 0x559e9b981720 2023-11-16T16:50:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.173+0000 7f7e8e3eeb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e9b7508c0 con 0x559e9b7d76b0 2023-11-16T16:50:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.173+0000 7f7e8e3eeb80 10 client.0 _ll_register_callbacks cb 0x559e9b88c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.173+0000 7f7e73fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b7d76b0 0x559e9b98a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.173+0000 7f7e73fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b7d76b0 0x559e9b98a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:55956/0 (socket says 172.21.15.2:55956) 2023-11-16T16:50:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.173+0000 7f7e73fff700 1 -- 172.21.15.2:0/2038170001 learned_addr learned my addr 172.21.15.2:0/2038170001 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.173+0000 7f7e737fe700 1 --2- 172.21.15.2:0/2038170001 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b981720 0x559e9b98af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.173+0000 7f7e78f9c700 1 --2- 172.21.15.2:0/2038170001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b981b90 0x559e9b984c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.174+0000 7f7e73fff700 1 -- 172.21.15.2:0/2038170001 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b981720 msgr2=0x559e9b98af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.174+0000 7f7e73fff700 1 --2- 172.21.15.2:0/2038170001 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b981720 0x559e9b98af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.174+0000 7f7e73fff700 1 -- 172.21.15.2:0/2038170001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b981b90 msgr2=0x559e9b984c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.174+0000 7f7e73fff700 1 --2- 172.21.15.2:0/2038170001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b981b90 0x559e9b984c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.174+0000 7f7e73fff700 1 -- 172.21.15.2:0/2038170001 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e74112360 con 0x559e9b7d76b0 2023-11-16T16:50:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.174+0000 7f7e78f9c700 1 --2- 172.21.15.2:0/2038170001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b981b90 0x559e9b984c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.174+0000 7f7e73fff700 1 --2- 172.21.15.2:0/2038170001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b7d76b0 0x559e9b98a9f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7e64005830 tx=0x7f7e64010ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2a506cb9630ddc2 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.174+0000 7f7e537fe700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:50:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.174+0000 7f7e537fe700 1 -- 172.21.15.2:0/2038170001 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e640a21e0 con 0x559e9b7d76b0 2023-11-16T16:50:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.174+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/2038170001 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559e9b96e710 con 0x559e9b7d76b0 2023-11-16T16:50:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.174+0000 7f7e537fe700 1 -- 172.21.15.2:0/2038170001 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f7e640b8310 con 0x559e9b7d76b0 2023-11-16T16:50:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.175+0000 7f7e537fe700 1 -- 172.21.15.2:0/2038170001 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e640cd140 con 0x559e9b7d76b0 2023-11-16T16:50:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.175+0000 7f7e8e3eeb80 10 client.25501 fetch_fsmap learned FSMap version 28 2023-11-16T16:50:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.175+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/2038170001 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559e9b893130 con 0x559e9b7d76b0 2023-11-16T16:50:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.175+0000 7f7e537fe700 1 -- 172.21.15.2:0/2038170001 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e64063020 con 0x559e9b7d76b0 2023-11-16T16:50:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.175+0000 7f7e537fe700 1 -- 172.21.15.2:0/2038170001 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7e640ca080 con 0x559e9b7d76b0 2023-11-16T16:50:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.175+0000 7f7e8e3eeb80 10 client.25501 fetch_fsmap finished waiting for FSMap version 28 2023-11-16T16:50:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.175+0000 7f7e8e3eeb80 10 client.25501 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.175+0000 7f7e8e3eeb80 20 client.25501 populate_metadata read hostname 'smithi002' 2023-11-16T16:50:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.175+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/2038170001 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x559e9b994be0 0x559e9b996fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.175+0000 7f7e8e3eeb80 4 client.25501 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:50:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.175+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/2038170001 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559e9b88f040 con 0x559e9b994be0 2023-11-16T16:50:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.176+0000 7f7e737fe700 1 --2- 172.21.15.2:0/2038170001 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x559e9b994be0 0x559e9b996fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.176+0000 7f7e737fe700 1 --2- 172.21.15.2:0/2038170001 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x559e9b994be0 0x559e9b996fc0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a5fb80bbad9c496 server_cookie=e6d83bc0e1188611 in_seq=0 out_seq=0 2023-11-16T16:50:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.176+0000 7f7e537fe700 10 client.25501 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:50:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.177+0000 7f7e537fe700 1 -- 172.21.15.2:0/2038170001 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x559e9b88f040 con 0x559e9b994be0 2023-11-16T16:50:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.177+0000 7f7e537fe700 10 client.25501 handle_command_reply: tid=0 2023-11-16T16:50:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.177+0000 7f7e8e3eeb80 1 client.25501 shutdown 2023-11-16T16:50:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.177+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/2038170001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b7d76b0 msgr2=0x559e9b98a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.177+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/2038170001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b7d76b0 0x559e9b98a9f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7e64005830 tx=0x7f7e64010ab0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.177+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/2038170001 shutdown_connections 2023-11-16T16:50:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.177+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/2038170001 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x559e9b994be0 0x559e9b996fc0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.177+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/2038170001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b7d76b0 0x559e9b98a9f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.177+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/2038170001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b981b90 0x559e9b984c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.177+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/2038170001 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b981720 0x559e9b98af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.177+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/2038170001 >> 172.21.15.2:0/2038170001 conn(0x559e9b970f50 msgr2=0x559e9b974f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.177+0000 7f7e537fe700 0 client.25501 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:50:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.177+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/2038170001 shutdown_connections 2023-11-16T16:50:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.178+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/2038170001 wait complete. 2023-11-16T16:50:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.178+0000 7f7e8e3eeb80 20 client.25501 trim_cache size 0 max 16384 2023-11-16T16:50:37.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.193+0000 7f7e8e3eeb80 1 Processor -- start 2023-11-16T16:50:37.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.193+0000 7f7e8e3eeb80 1 -- start start 2023-11-16T16:50:37.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.194+0000 7f7e8e3eeb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b7d76b0 0x559e9b9826a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:37.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.194+0000 7f7e8e3eeb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b981720 0x559e9b975f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:37.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.194+0000 7f7e8e3eeb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b981b90 0x559e9b97dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:37.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.194+0000 7f7e8e3eeb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e9b972910 con 0x559e9b7d76b0 2023-11-16T16:50:37.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.194+0000 7f7e8e3eeb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e9b982be0 con 0x559e9b981720 2023-11-16T16:50:37.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.194+0000 7f7e8e3eeb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e9b980d50 con 0x559e9b981b90 2023-11-16T16:50:37.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.194+0000 7f7e73fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b7d76b0 0x559e9b9826a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:37.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.194+0000 7f7e73fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b7d76b0 0x559e9b9826a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40596/0 (socket says 172.21.15.2:40596) 2023-11-16T16:50:37.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.194+0000 7f7e73fff700 1 -- 172.21.15.2:0/2109253408 learned_addr learned my addr 172.21.15.2:0/2109253408 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:37.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.194+0000 7f7e78f9c700 1 --2- 172.21.15.2:0/2109253408 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b981b90 0x559e9b97dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:37.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.194+0000 7f7e737fe700 1 --2- 172.21.15.2:0/2109253408 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b981720 0x559e9b975f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:37.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.194+0000 7f7e73fff700 1 -- 172.21.15.2:0/2109253408 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b981b90 msgr2=0x559e9b97dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:37.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.194+0000 7f7e73fff700 1 --2- 172.21.15.2:0/2109253408 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b981b90 0x559e9b97dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.195+0000 7f7e73fff700 1 -- 172.21.15.2:0/2109253408 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b981720 msgr2=0x559e9b975f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:37.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.195+0000 7f7e73fff700 1 --2- 172.21.15.2:0/2109253408 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b981720 0x559e9b975f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.195+0000 7f7e73fff700 1 -- 172.21.15.2:0/2109253408 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e9b892e80 con 0x559e9b7d76b0 2023-11-16T16:50:37.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.195+0000 7f7e73fff700 1 --2- 172.21.15.2:0/2109253408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b7d76b0 0x559e9b9826a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400c2d0 tx=0x7f7e640ca870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12de17251a0af4b3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:37.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.195+0000 7f7e5affd700 1 -- 172.21.15.2:0/2109253408 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64099970 con 0x559e9b7d76b0 2023-11-16T16:50:37.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.195+0000 7f7e5affd700 1 -- 172.21.15.2:0/2109253408 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e64063070 con 0x559e9b7d76b0 2023-11-16T16:50:37.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.195+0000 7f7e5affd700 1 -- 172.21.15.2:0/2109253408 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e640cd3a0 con 0x559e9b7d76b0 2023-11-16T16:50:37.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.196+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/2109253408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b7d76b0 msgr2=0x559e9b9826a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.196+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/2109253408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b7d76b0 0x559e9b9826a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400c2d0 tx=0x7f7e640ca870 comp rx=0 tx=0).stop 2023-11-16T16:50:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.196+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/2109253408 shutdown_connections 2023-11-16T16:50:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.196+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/2109253408 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b981b90 0x559e9b97dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.196+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/2109253408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b7d76b0 0x559e9b9826a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.196+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/2109253408 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b981720 0x559e9b975f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.196+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/2109253408 >> 172.21.15.2:0/2109253408 conn(0x559e9b970f50 msgr2=0x559e9b981f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.196+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/2109253408 shutdown_connections 2023-11-16T16:50:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.196+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/2109253408 wait complete. 2023-11-16T16:50:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.196+0000 7f7e8e3eeb80 1 -- start start 2023-11-16T16:50:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.197+0000 7f7e8e3eeb80 1 Processor -- start 2023-11-16T16:50:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.197+0000 7f7e8e3eeb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b7d76b0 0x559e9b98ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.197+0000 7f7e8e3eeb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b981720 0x559e9b988100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:37.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.197+0000 7f7e8e3eeb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b981b90 0x559e9b988680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:37.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.197+0000 7f7e8e3eeb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e9b988dc0 con 0x559e9b981b90 2023-11-16T16:50:37.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.197+0000 7f7e8e3eeb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e9b988f30 con 0x559e9b7d76b0 2023-11-16T16:50:37.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.197+0000 7f7e8e3eeb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e9b9e7a30 con 0x559e9b981720 2023-11-16T16:50:37.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.197+0000 7f7e8e3eeb80 10 client.0 _ll_register_callbacks cb 0x7f7e68009bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:37.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.198+0000 7f7e737fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b981720 0x559e9b988100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:37.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.198+0000 7f7e73fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b7d76b0 0x559e9b98ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:37.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.198+0000 7f7e78f9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b981b90 0x559e9b988680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:37.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.198+0000 7f7e737fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b981720 0x559e9b988100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:55976/0 (socket says 172.21.15.2:55976) 2023-11-16T16:50:37.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.198+0000 7f7e737fe700 1 -- 172.21.15.2:0/1307242107 learned_addr learned my addr 172.21.15.2:0/1307242107 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:37.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.198+0000 7f7e78f9c700 1 -- 172.21.15.2:0/1307242107 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b981720 msgr2=0x559e9b988100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:37.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.198+0000 7f7e78f9c700 1 --2- 172.21.15.2:0/1307242107 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b981720 0x559e9b988100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.198+0000 7f7e78f9c700 1 -- 172.21.15.2:0/1307242107 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b7d76b0 msgr2=0x559e9b98ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:37.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.198+0000 7f7e78f9c700 1 --2- 172.21.15.2:0/1307242107 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b7d76b0 0x559e9b98ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.198+0000 7f7e78f9c700 1 -- 172.21.15.2:0/1307242107 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559e9b893130 con 0x559e9b981b90 2023-11-16T16:50:37.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.198+0000 7f7e73fff700 1 --2- 172.21.15.2:0/1307242107 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b7d76b0 0x559e9b98ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:37.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.198+0000 7f7e737fe700 1 --2- 172.21.15.2:0/1307242107 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b981720 0x559e9b988100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:37.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.198+0000 7f7e78f9c700 1 --2- 172.21.15.2:0/1307242107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b981b90 0x559e9b988680 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c0062a0 tx=0x7f7e6c00b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6a8bda085e4df2 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:37.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.198+0000 7f7e537fe700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:50:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.199+0000 7f7e537fe700 1 -- 172.21.15.2:0/1307242107 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c025040 con 0x559e9b981b90 2023-11-16T16:50:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.199+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/1307242107 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7e640a2460 con 0x559e9b981b90 2023-11-16T16:50:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.199+0000 7f7e537fe700 1 -- 172.21.15.2:0/1307242107 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f7e6c020450 con 0x559e9b981b90 2023-11-16T16:50:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.199+0000 7f7e537fe700 1 -- 172.21.15.2:0/1307242107 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e6c00e040 con 0x559e9b981b90 2023-11-16T16:50:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.199+0000 7f7e8e3eeb80 10 client.15504 fetch_fsmap learned FSMap version 28 2023-11-16T16:50:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.199+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/1307242107 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559e9b6e3c00 con 0x559e9b981b90 2023-11-16T16:50:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.199+0000 7f7e537fe700 1 -- 172.21.15.2:0/1307242107 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e640a2460 con 0x559e9b981b90 2023-11-16T16:50:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.199+0000 7f7e537fe700 1 -- 172.21.15.2:0/1307242107 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7e6c005380 con 0x559e9b981b90 2023-11-16T16:50:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.199+0000 7f7e8e3eeb80 10 client.15504 fetch_fsmap finished waiting for FSMap version 28 2023-11-16T16:50:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.200+0000 7f7e8e3eeb80 10 client.15504 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.200+0000 7f7e8e3eeb80 20 client.15504 populate_metadata read hostname 'smithi002' 2023-11-16T16:50:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.200+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/1307242107 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x559e9b9e9040 0x559e9b9eb450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.200+0000 7f7e8e3eeb80 4 client.15504 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:50:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.200+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/1307242107 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559e9b89b600 con 0x559e9b9e9040 2023-11-16T16:50:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.200+0000 7f7e73fff700 1 --2- 172.21.15.2:0/1307242107 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x559e9b9e9040 0x559e9b9eb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.201+0000 7f7e537fe700 10 client.15504 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:50:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.201+0000 7f7e73fff700 1 --2- 172.21.15.2:0/1307242107 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x559e9b9e9040 0x559e9b9eb450 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63a9c04cac7aed7 server_cookie=1e6b208a1c9ea864 in_seq=0 out_seq=0 2023-11-16T16:50:37.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.204+0000 7f7e537fe700 1 -- 172.21.15.2:0/1307242107 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559e9b89b600 con 0x559e9b9e9040 2023-11-16T16:50:37.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.204+0000 7f7e537fe700 10 client.15504 handle_command_reply: tid=0 2023-11-16T16:50:37.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.204+0000 7f7e8e3eeb80 1 client.15504 shutdown 2023-11-16T16:50:37.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.205+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/1307242107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b981b90 msgr2=0x559e9b988680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:37.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.205+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/1307242107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b981b90 0x559e9b988680 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c0062a0 tx=0x7f7e6c00b3b0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.205+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/1307242107 shutdown_connections 2023-11-16T16:50:37.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.205+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/1307242107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559e9b981b90 0x559e9b988680 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.205+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/1307242107 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559e9b7d76b0 0x559e9b98ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.205+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/1307242107 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x559e9b9e9040 0x559e9b9eb450 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.205+0000 7f7e8e3eeb80 1 --2- 172.21.15.2:0/1307242107 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559e9b981720 0x559e9b988100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.205+0000 7f7e537fe700 0 client.15504 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:50:37.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.205+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/1307242107 >> 172.21.15.2:0/1307242107 conn(0x559e9b970f50 msgr2=0x559e9b9805d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:37.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.205+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/1307242107 shutdown_connections 2023-11-16T16:50:37.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.205+0000 7f7e8e3eeb80 1 -- 172.21.15.2:0/1307242107 wait complete. 2023-11-16T16:50:37.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.205+0000 7f7e8e3eeb80 20 client.15504 trim_cache size 0 max 16384 2023-11-16T16:50:37.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:50:37.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:50:37.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:50:37.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:50:37.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.206+0000 7f7e7a9ff700 1 -- 172.21.15.2:0/2604790223 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f7e5c0a13e0 msgr2=0x7f7e5c0a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:37.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.206+0000 7f7e7a9ff700 1 --2- 172.21.15.2:0/2604790223 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f7e5c0a13e0 0x7f7e5c0a3890 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c00b780 tx=0x7f7e6c0099e0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.206+0000 7f7e7a9ff700 1 -- 172.21.15.2:0/2604790223 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e74072e30 msgr2=0x7f7e74198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:37.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.206+0000 7f7e7a9ff700 1 --2- 172.21.15.2:0/2604790223 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e74072e30 0x7f7e74198700 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400e9a0 tx=0x7f7e6400ecb0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.206+0000 7f7e7a9ff700 1 -- 172.21.15.2:0/2604790223 shutdown_connections 2023-11-16T16:50:37.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.206+0000 7f7e7a9ff700 1 --2- 172.21.15.2:0/2604790223 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f7e5c0a13e0 0x7f7e5c0a3890 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.206+0000 7f7e7a9ff700 1 --2- 172.21.15.2:0/2604790223 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e74072e30 0x7f7e74198700 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.206+0000 7f7e7a9ff700 1 --2- 172.21.15.2:0/2604790223 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7e7410cb70 0x7f7e7419bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.206+0000 7f7e7a9ff700 1 --2- 172.21.15.2:0/2604790223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e74073740 0x7f7e74198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:37.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.206+0000 7f7e7a9ff700 1 -- 172.21.15.2:0/2604790223 >> 172.21.15.2:0/2604790223 conn(0x7f7e74078210 msgr2=0x7f7e741066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:37.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.206+0000 7f7e7a9ff700 1 -- 172.21.15.2:0/2604790223 shutdown_connections 2023-11-16T16:50:37.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:37.206+0000 7f7e7a9ff700 1 -- 172.21.15.2:0/2604790223 wait complete. 2023-11-16T16:50:39.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:38 smithi141 ceph-mon[131184]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-11-16T16:50:39.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:38 smithi155 ceph-mon[131607]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-11-16T16:50:39.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:38 smithi002 ceph-mon[128712]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-11-16T16:50:41.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:40 smithi141 ceph-mon[131184]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2023-11-16T16:50:41.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:40 smithi155 ceph-mon[131607]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2023-11-16T16:50:41.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:40 smithi002 ceph-mon[128712]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2023-11-16T16:50:43.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:42 smithi141 ceph-mon[131184]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-11-16T16:50:43.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:42 smithi155 ceph-mon[131607]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-11-16T16:50:43.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:42 smithi002 ceph-mon[128712]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-11-16T16:50:45.869 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:45 smithi155 ceph-mon[131607]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2023-11-16T16:50:45.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:45 smithi002 ceph-mon[128712]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2023-11-16T16:50:46.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:45 smithi141 ceph-mon[131184]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2023-11-16T16:50:47.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:46 smithi141 ceph-mon[131184]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2023-11-16T16:50:47.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:46 smithi155 ceph-mon[131607]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2023-11-16T16:50:47.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:46 smithi002 ceph-mon[128712]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2023-11-16T16:50:49.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:48 smithi141 ceph-mon[131184]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-11-16T16:50:49.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:48 smithi155 ceph-mon[131607]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-11-16T16:50:49.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:48 smithi002 ceph-mon[128712]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-11-16T16:50:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.445+0000 7f5e469ff700 1 -- 172.21.15.2:0/3315129363 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e400746a0 msgr2=0x7f5e40072d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.445+0000 7f5e469ff700 1 --2- 172.21.15.2:0/3315129363 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e400746a0 0x7f5e40072d20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5e30009990 tx=0x7f5e30009ca0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.446+0000 7f5e469ff700 1 -- 172.21.15.2:0/3315129363 shutdown_connections 2023-11-16T16:50:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.446+0000 7f5e469ff700 1 --2- 172.21.15.2:0/3315129363 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e4010ce50 0x7f5e4010f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.446+0000 7f5e469ff700 1 --2- 172.21.15.2:0/3315129363 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e400746a0 0x7f5e40072d20 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.446+0000 7f5e469ff700 1 --2- 172.21.15.2:0/3315129363 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e40074b00 0x7f5e40073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.446+0000 7f5e469ff700 1 -- 172.21.15.2:0/3315129363 >> 172.21.15.2:0/3315129363 conn(0x7f5e40077550 msgr2=0x7f5e40077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.446+0000 7f5e469ff700 1 -- 172.21.15.2:0/3315129363 shutdown_connections 2023-11-16T16:50:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.446+0000 7f5e469ff700 1 -- 172.21.15.2:0/3315129363 wait complete. 2023-11-16T16:50:50.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.447+0000 7f5e469ff700 1 Processor -- start 2023-11-16T16:50:50.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.448+0000 7f5e469ff700 1 -- start start 2023-11-16T16:50:50.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.448+0000 7f5e469ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e400746a0 0x7f5e401985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.448+0000 7f5e469ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e40074b00 0x7f5e40198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.448+0000 7f5e469ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e4010ce50 0x7f5e4019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.448+0000 7f5e469ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4019d5c0 con 0x7f5e40074b00 2023-11-16T16:50:50.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.448+0000 7f5e469ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4019d730 con 0x7f5e400746a0 2023-11-16T16:50:50.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.448+0000 7f5e469ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4019d8f0 con 0x7f5e4010ce50 2023-11-16T16:50:50.450 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.449+0000 7f5e3ffff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e400746a0 0x7f5e401985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.450 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.449+0000 7f5e3ffff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e400746a0 0x7f5e401985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:58946/0 (socket says 172.21.15.2:58946) 2023-11-16T16:50:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.449+0000 7f5e3ffff700 1 -- 172.21.15.2:0/4014410918 learned_addr learned my addr 172.21.15.2:0/4014410918 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.449+0000 7f5e3f7fe700 1 --2- 172.21.15.2:0/4014410918 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e40074b00 0x7f5e40198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.449+0000 7f5e44f9c700 1 --2- 172.21.15.2:0/4014410918 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e4010ce50 0x7f5e4019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.449+0000 7f5e3ffff700 1 -- 172.21.15.2:0/4014410918 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e4010ce50 msgr2=0x7f5e4019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.449+0000 7f5e3ffff700 1 --2- 172.21.15.2:0/4014410918 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e4010ce50 0x7f5e4019cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.449+0000 7f5e3ffff700 1 -- 172.21.15.2:0/4014410918 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e40074b00 msgr2=0x7f5e40198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.449+0000 7f5e3ffff700 1 --2- 172.21.15.2:0/4014410918 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e40074b00 0x7f5e40198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.449+0000 7f5e3ffff700 1 -- 172.21.15.2:0/4014410918 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e300095f0 con 0x7f5e400746a0 2023-11-16T16:50:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.449+0000 7f5e44f9c700 1 --2- 172.21.15.2:0/4014410918 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e4010ce50 0x7f5e4019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.449+0000 7f5e3f7fe700 1 --2- 172.21.15.2:0/4014410918 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e40074b00 0x7f5e40198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:50.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.449+0000 7f5e3ffff700 1 --2- 172.21.15.2:0/4014410918 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e400746a0 0x7f5e401985e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5e30005910 tx=0x7f5e3000b610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:50.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.450+0000 7f5e3d7fa700 1 -- 172.21.15.2:0/4014410918 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3001d070 con 0x7f5e400746a0 2023-11-16T16:50:50.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.450+0000 7f5e3d7fa700 1 -- 172.21.15.2:0/4014410918 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e3000fb80 con 0x7f5e400746a0 2023-11-16T16:50:50.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.450+0000 7f5e3d7fa700 1 -- 172.21.15.2:0/4014410918 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e30017650 con 0x7f5e400746a0 2023-11-16T16:50:50.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.450+0000 7f5e469ff700 1 -- 172.21.15.2:0/4014410918 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e40112590 con 0x7f5e400746a0 2023-11-16T16:50:50.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.450+0000 7f5e469ff700 1 -- 172.21.15.2:0/4014410918 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e40112400 con 0x7f5e400746a0 2023-11-16T16:50:50.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.455+0000 7f5e3d7fa700 1 -- 172.21.15.2:0/4014410918 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7f5e30022470 con 0x7f5e400746a0 2023-11-16T16:50:50.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.457+0000 7f5e5a321b80 1 Processor -- start 2023-11-16T16:50:50.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.457+0000 7f5e5a321b80 1 -- start start 2023-11-16T16:50:50.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.458+0000 7f5e5a321b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81ac86b0 0x563a81c721e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.458+0000 7f5e5a321b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72720 0x563a81c6c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.458+0000 7f5e5a321b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81c72b90 0x563a81c6cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.458+0000 7f5e5a321b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a81b72000 con 0x563a81ac86b0 2023-11-16T16:50:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.458+0000 7f5e5a321b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a81bc4840 con 0x563a81c72b90 2023-11-16T16:50:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.458+0000 7f5e5a321b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a81bbdcd0 con 0x563a81c72720 2023-11-16T16:50:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.458+0000 7f5e3d7fa700 1 --2- 172.21.15.2:0/4014410918 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f5e280a13e0 0x7f5e280a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.458+0000 7f5e3ffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72720 0x563a81c6c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.459+0000 7f5e3ffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72720 0x563a81c6c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:49218/0 (socket says 172.21.15.2:49218) 2023-11-16T16:50:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.459+0000 7f5e3ffff700 1 -- 172.21.15.2:0/66945377 learned_addr learned my addr 172.21.15.2:0/66945377 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:50.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.459+0000 7f5e3f7fe700 1 --2- 172.21.15.2:0/66945377 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81c72b90 0x563a81c6cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.459+0000 7f5e44f9c700 1 --2- 172.21.15.2:0/66945377 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81ac86b0 0x563a81c721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.459+0000 7f5e3ffff700 1 -- 172.21.15.2:0/66945377 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81c72b90 msgr2=0x563a81c6cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:50.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.459+0000 7f5e3ffff700 1 --2- 172.21.15.2:0/66945377 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81c72b90 0x563a81c6cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.459+0000 7f5e3ffff700 1 -- 172.21.15.2:0/66945377 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81ac86b0 msgr2=0x563a81c721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:50.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.459+0000 7f5e3ffff700 1 --2- 172.21.15.2:0/66945377 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81ac86b0 0x563a81c721e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.459+0000 7f5e3ffff700 1 -- 172.21.15.2:0/66945377 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a81a3f5d0 con 0x563a81c72720 2023-11-16T16:50:50.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.459+0000 7f5e44f9c700 1 --2- 172.21.15.2:0/66945377 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81ac86b0 0x563a81c721e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:50:50.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.459+0000 7f5e3ffff700 1 --2- 172.21.15.2:0/66945377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72720 0x563a81c6c560 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5e3009f7b0 tx=0x7f5e3009f8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=544c6cdaf9cce55f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:50.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.459+0000 7f5e3d7fa700 1 -- 172.21.15.2:0/4014410918 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f5e300cea80 con 0x7f5e400746a0 2023-11-16T16:50:50.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.460+0000 7f5e44f9c700 1 --2- 172.21.15.2:0/4014410918 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f5e280a13e0 0x7f5e280a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.460+0000 7f5e267fc700 1 -- 172.21.15.2:0/66945377 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e300d1ea0 con 0x563a81c72720 2023-11-16T16:50:50.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.460+0000 7f5e267fc700 1 -- 172.21.15.2:0/66945377 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e30065190 con 0x563a81c72720 2023-11-16T16:50:50.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.460+0000 7f5e267fc700 1 -- 172.21.15.2:0/66945377 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3006ed00 con 0x563a81c72720 2023-11-16T16:50:50.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.461+0000 7f5e5a321b80 1 -- 172.21.15.2:0/66945377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72720 msgr2=0x563a81c6c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.461+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/66945377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72720 0x563a81c6c560 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5e3009f7b0 tx=0x7f5e3009f8d0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.461+0000 7f5e44f9c700 1 --2- 172.21.15.2:0/4014410918 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f5e280a13e0 0x7f5e280a3890 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5e40069fd0 tx=0x7f5e3800b040 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:50.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.461+0000 7f5e5a321b80 1 -- 172.21.15.2:0/66945377 shutdown_connections 2023-11-16T16:50:50.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.461+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/66945377 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81c72b90 0x563a81c6cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.461+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/66945377 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81ac86b0 0x563a81c721e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.461+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/66945377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72720 0x563a81c6c560 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.461+0000 7f5e5a321b80 1 -- 172.21.15.2:0/66945377 >> 172.21.15.2:0/66945377 conn(0x563a81c61f50 msgr2=0x563a81c67270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:50.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.461+0000 7f5e5a321b80 1 -- 172.21.15.2:0/66945377 shutdown_connections 2023-11-16T16:50:50.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.462+0000 7f5e5a321b80 1 -- 172.21.15.2:0/66945377 wait complete. 2023-11-16T16:50:50.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.462+0000 7f5e5a321b80 1 -- start start 2023-11-16T16:50:50.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.463+0000 7f5e5a321b80 1 Processor -- start 2023-11-16T16:50:50.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.463+0000 7f5e5a321b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81ac86b0 0x563a81c7b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.463+0000 7f5e5a321b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72720 0x563a81c7bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.463+0000 7f5e5a321b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81c72b90 0x563a81c75920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.463+0000 7f5e5a321b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a81b5f880 con 0x563a81ac86b0 2023-11-16T16:50:50.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.463+0000 7f5e5a321b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a81b5f520 con 0x563a81c72b90 2023-11-16T16:50:50.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.463+0000 7f5e5a321b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a81a418c0 con 0x563a81c72720 2023-11-16T16:50:50.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.463+0000 7f5e5a321b80 10 client.0 _ll_register_callbacks cb 0x563a81b7d720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:50.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.464+0000 7f5e3ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81ac86b0 0x563a81c7b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.464+0000 7f5e44f9c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81c72b90 0x563a81c75920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.464+0000 7f5e3f7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72720 0x563a81c7bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.464+0000 7f5e44f9c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81c72b90 0x563a81c75920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:58964/0 (socket says 172.21.15.2:58964) 2023-11-16T16:50:50.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.464+0000 7f5e44f9c700 1 -- 172.21.15.2:0/1243561517 learned_addr learned my addr 172.21.15.2:0/1243561517 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:50.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.464+0000 7f5e44f9c700 1 -- 172.21.15.2:0/1243561517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72720 msgr2=0x563a81c7bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:50.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.464+0000 7f5e44f9c700 1 --2- 172.21.15.2:0/1243561517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72720 0x563a81c7bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.464+0000 7f5e44f9c700 1 -- 172.21.15.2:0/1243561517 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81ac86b0 msgr2=0x563a81c7b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:50.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.465+0000 7f5e44f9c700 1 --2- 172.21.15.2:0/1243561517 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81ac86b0 0x563a81c7b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.465+0000 7f5e44f9c700 1 -- 172.21.15.2:0/1243561517 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e40112400 con 0x563a81c72b90 2023-11-16T16:50:50.468 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.465+0000 7f5e3ffff700 1 --2- 172.21.15.2:0/1243561517 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81ac86b0 0x563a81c7b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:50.468 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.465+0000 7f5e44f9c700 1 --2- 172.21.15.2:0/1243561517 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81c72b90 0x563a81c75920 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5e38016c00 tx=0x7f5e38009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58431d60db70822c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:50.468 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.465+0000 7f5e1f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2023-11-16T16:50:50.468 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.465+0000 7f5e1f7fe700 1 -- 172.21.15.2:0/1243561517 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e38015a00 con 0x563a81c72b90 2023-11-16T16:50:50.468 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.465+0000 7f5e1f7fe700 1 -- 172.21.15.2:0/1243561517 <== mon.2 v2:172.21.15.141:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f5e38015b60 con 0x563a81c72b90 2023-11-16T16:50:50.468 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.466+0000 7f5e5a321b80 1 -- 172.21.15.2:0/1243561517 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563a81c5f710 con 0x563a81c72b90 2023-11-16T16:50:50.468 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.466+0000 7f5e1f7fe700 1 -- 172.21.15.2:0/1243561517 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e38015d20 con 0x563a81c72b90 2023-11-16T16:50:50.468 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.466+0000 7f5e5a321b80 10 client.25547 fetch_fsmap learned FSMap version 28 2023-11-16T16:50:50.468 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.466+0000 7f5e5a321b80 1 -- 172.21.15.2:0/1243561517 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563a81b84130 con 0x563a81c72b90 2023-11-16T16:50:50.468 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.467+0000 7f5e1f7fe700 1 -- 172.21.15.2:0/1243561517 <== mon.2 v2:172.21.15.141:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563a81c5f710 con 0x563a81c72b90 2023-11-16T16:50:50.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.468+0000 7f5e1f7fe700 1 -- 172.21.15.2:0/1243561517 <== mon.2 v2:172.21.15.141:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5e380204c0 con 0x563a81c72b90 2023-11-16T16:50:50.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.468+0000 7f5e5a321b80 10 client.25547 fetch_fsmap finished waiting for FSMap version 28 2023-11-16T16:50:50.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.468+0000 7f5e5a321b80 10 client.25547 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:50.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.468+0000 7f5e5a321b80 20 client.25547 populate_metadata read hostname 'smithi002' 2023-11-16T16:50:50.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.468+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/1243561517 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x563a81c85bd0 0x563a81c87fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.468+0000 7f5e5a321b80 4 client.25547 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:50:50.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.468+0000 7f5e5a321b80 1 -- 172.21.15.2:0/1243561517 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563a81b80040 con 0x563a81c85bd0 2023-11-16T16:50:50.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.468+0000 7f5e3ffff700 1 --2- 172.21.15.2:0/1243561517 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x563a81c85bd0 0x563a81c87fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.469+0000 7f5e3ffff700 1 --2- 172.21.15.2:0/1243561517 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x563a81c85bd0 0x563a81c87fe0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc0c7a1bbe1b74c5 server_cookie=7d04f873ae94f604 in_seq=0 out_seq=0 2023-11-16T16:50:50.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.469+0000 7f5e1f7fe700 10 client.25547 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:50:50.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.470+0000 7f5e1f7fe700 1 -- 172.21.15.2:0/1243561517 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x563a81b80040 con 0x563a81c85bd0 2023-11-16T16:50:50.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.470+0000 7f5e1f7fe700 10 client.25547 handle_command_reply: tid=0 2023-11-16T16:50:50.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.470+0000 7f5e5a321b80 1 client.25547 shutdown 2023-11-16T16:50:50.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.470+0000 7f5e5a321b80 1 -- 172.21.15.2:0/1243561517 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81c72b90 msgr2=0x563a81c75920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.470+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/1243561517 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81c72b90 0x563a81c75920 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5e38016c00 tx=0x7f5e38009040 comp rx=0 tx=0).stop 2023-11-16T16:50:50.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.470+0000 7f5e5a321b80 1 -- 172.21.15.2:0/1243561517 shutdown_connections 2023-11-16T16:50:50.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.470+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/1243561517 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81c72b90 0x563a81c75920 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.470+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/1243561517 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81ac86b0 0x563a81c7b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.470+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/1243561517 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x563a81c85bd0 0x563a81c87fe0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.470+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/1243561517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72720 0x563a81c7bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.470+0000 7f5e5a321b80 1 -- 172.21.15.2:0/1243561517 >> 172.21.15.2:0/1243561517 conn(0x563a81c61f50 msgr2=0x563a81c667e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:50.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.471+0000 7f5e1f7fe700 0 client.25547 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:50:50.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.471+0000 7f5e5a321b80 1 -- 172.21.15.2:0/1243561517 shutdown_connections 2023-11-16T16:50:50.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.471+0000 7f5e5a321b80 1 -- 172.21.15.2:0/1243561517 wait complete. 2023-11-16T16:50:50.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.471+0000 7f5e5a321b80 20 client.25547 trim_cache size 0 max 16384 2023-11-16T16:50:50.502 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.501+0000 7f5e5a321b80 1 Processor -- start 2023-11-16T16:50:50.502 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.501+0000 7f5e5a321b80 1 -- start start 2023-11-16T16:50:50.502 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.501+0000 7f5e5a321b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81ac86b0 0x563a81c66f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.502 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.502+0000 7f5e5a321b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81c72720 0x563a81c6ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.502 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.502+0000 7f5e5a321b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72b90 0x563a81c6f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.502 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.502+0000 7f5e5a321b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a81c6d690 con 0x563a81c72720 2023-11-16T16:50:50.503 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.502+0000 7f5e5a321b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a81c681d0 con 0x563a81ac86b0 2023-11-16T16:50:50.503 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.502+0000 7f5e5a321b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a81c5f300 con 0x563a81c72b90 2023-11-16T16:50:50.504 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.502+0000 7f5e3f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81c72720 0x563a81c6ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.504 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.502+0000 7f5e3f7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81c72720 0x563a81c6ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53958/0 (socket says 172.21.15.2:53958) 2023-11-16T16:50:50.504 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.502+0000 7f5e3f7fe700 1 -- 172.21.15.2:0/452944355 learned_addr learned my addr 172.21.15.2:0/452944355 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:50.504 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.502+0000 7f5e3ffff700 1 --2- 172.21.15.2:0/452944355 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81ac86b0 0x563a81c66f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.502+0000 7f5e44f9c700 1 --2- 172.21.15.2:0/452944355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72b90 0x563a81c6f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.503+0000 7f5e3ffff700 1 -- 172.21.15.2:0/452944355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72b90 msgr2=0x563a81c6f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:50.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.503+0000 7f5e3ffff700 1 --2- 172.21.15.2:0/452944355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72b90 0x563a81c6f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.503+0000 7f5e3ffff700 1 -- 172.21.15.2:0/452944355 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81c72720 msgr2=0x563a81c6ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:50.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.503+0000 7f5e3ffff700 1 --2- 172.21.15.2:0/452944355 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81c72720 0x563a81c6ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.503+0000 7f5e3ffff700 1 -- 172.21.15.2:0/452944355 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a81c636f0 con 0x563a81ac86b0 2023-11-16T16:50:50.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.503+0000 7f5e3f7fe700 1 --2- 172.21.15.2:0/452944355 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81c72720 0x563a81c6ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:50:50.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.503+0000 7f5e3ffff700 1 --2- 172.21.15.2:0/452944355 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81ac86b0 0x563a81c66f60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5e30064ef0 tx=0x7f5e30005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70bf3ec8f46abcff server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:50.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.503+0000 7f5e26ffd700 1 -- 172.21.15.2:0/452944355 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e300a3980 con 0x563a81ac86b0 2023-11-16T16:50:50.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.504+0000 7f5e26ffd700 1 -- 172.21.15.2:0/452944355 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e3009b5c0 con 0x563a81ac86b0 2023-11-16T16:50:50.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.504+0000 7f5e26ffd700 1 -- 172.21.15.2:0/452944355 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3009f9c0 con 0x563a81ac86b0 2023-11-16T16:50:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.505+0000 7f5e5a321b80 1 -- 172.21.15.2:0/452944355 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81ac86b0 msgr2=0x563a81c66f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.505+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/452944355 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81ac86b0 0x563a81c66f60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5e30064ef0 tx=0x7f5e30005fd0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.506 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.505+0000 7f5e5a321b80 1 -- 172.21.15.2:0/452944355 shutdown_connections 2023-11-16T16:50:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.505+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/452944355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72b90 0x563a81c6f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.505+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/452944355 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81ac86b0 0x563a81c66f60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.505+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/452944355 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81c72720 0x563a81c6ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.505+0000 7f5e5a321b80 1 -- 172.21.15.2:0/452944355 >> 172.21.15.2:0/452944355 conn(0x563a81c61f50 msgr2=0x563a81c762b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.505+0000 7f5e5a321b80 1 -- 172.21.15.2:0/452944355 shutdown_connections 2023-11-16T16:50:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.505+0000 7f5e5a321b80 1 -- 172.21.15.2:0/452944355 wait complete. 2023-11-16T16:50:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.505+0000 7f5e5a321b80 1 -- start start 2023-11-16T16:50:50.507 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.506+0000 7f5e5a321b80 1 Processor -- start 2023-11-16T16:50:50.508 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.506+0000 7f5e5a321b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81ac86b0 0x563a81c79240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.506+0000 7f5e5a321b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72720 0x563a81c79780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.506+0000 7f5e5a321b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81c72b90 0x563a81c79cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.507+0000 7f5e5a321b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a81c7bf40 con 0x563a81c72b90 2023-11-16T16:50:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.507+0000 7f5e5a321b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a81cd8a70 con 0x563a81ac86b0 2023-11-16T16:50:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.507+0000 7f5e5a321b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a81cd8bb0 con 0x563a81c72720 2023-11-16T16:50:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.507+0000 7f5e5a321b80 10 client.0 _ll_register_callbacks cb 0x7f5e380167b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:50:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.507+0000 7f5e3ffff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81ac86b0 0x563a81c79240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.507+0000 7f5e44f9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81c72b90 0x563a81c79cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.507+0000 7f5e3ffff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81ac86b0 0x563a81c79240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:58986/0 (socket says 172.21.15.2:58986) 2023-11-16T16:50:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.507+0000 7f5e3ffff700 1 -- 172.21.15.2:0/3163946342 learned_addr learned my addr 172.21.15.2:0/3163946342 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:50:50.509 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.507+0000 7f5e3f7fe700 1 --2- 172.21.15.2:0/3163946342 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72720 0x563a81c79780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.507+0000 7f5e3ffff700 1 -- 172.21.15.2:0/3163946342 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72720 msgr2=0x563a81c79780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.507+0000 7f5e3ffff700 1 --2- 172.21.15.2:0/3163946342 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72720 0x563a81c79780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.507+0000 7f5e3ffff700 1 -- 172.21.15.2:0/3163946342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81c72b90 msgr2=0x563a81c79cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:50:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.507+0000 7f5e3ffff700 1 --2- 172.21.15.2:0/3163946342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81c72b90 0x563a81c79cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.508+0000 7f5e3ffff700 1 -- 172.21.15.2:0/3163946342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e34004290 con 0x563a81ac86b0 2023-11-16T16:50:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.508+0000 7f5e3ffff700 1 --2- 172.21.15.2:0/3163946342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81ac86b0 0x563a81c79240 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5e30097ac0 tx=0x7f5e30009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2c156647b1b1ab0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:50:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.508+0000 7f5e1f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2023-11-16T16:50:50.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.508+0000 7f5e1f7fe700 1 -- 172.21.15.2:0/3163946342 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e300a3980 con 0x563a81ac86b0 2023-11-16T16:50:50.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.508+0000 7f5e1f7fe700 1 -- 172.21.15.2:0/3163946342 <== mon.2 v2:172.21.15.141:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f5e3000fcf0 con 0x563a81ac86b0 2023-11-16T16:50:50.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.508+0000 7f5e5a321b80 1 -- 172.21.15.2:0/3163946342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e30065190 con 0x563a81ac86b0 2023-11-16T16:50:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.509+0000 7f5e1f7fe700 1 -- 172.21.15.2:0/3163946342 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e3009f850 con 0x563a81ac86b0 2023-11-16T16:50:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.509+0000 7f5e5a321b80 10 client.25553 fetch_fsmap learned FSMap version 28 2023-11-16T16:50:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.509+0000 7f5e5a321b80 1 -- 172.21.15.2:0/3163946342 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563a81a3f5d0 con 0x563a81ac86b0 2023-11-16T16:50:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.509+0000 7f5e1f7fe700 1 -- 172.21.15.2:0/3163946342 <== mon.2 v2:172.21.15.141:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e30065190 con 0x563a81ac86b0 2023-11-16T16:50:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.509+0000 7f5e1f7fe700 1 -- 172.21.15.2:0/3163946342 <== mon.2 v2:172.21.15.141:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5e3009b820 con 0x563a81ac86b0 2023-11-16T16:50:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.509+0000 7f5e5a321b80 10 client.25553 fetch_fsmap finished waiting for FSMap version 28 2023-11-16T16:50:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.510+0000 7f5e5a321b80 10 client.25553 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:50:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.510+0000 7f5e5a321b80 20 client.25553 populate_metadata read hostname 'smithi002' 2023-11-16T16:50:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.510+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/3163946342 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x563a81cd9da0 0x563a81cdc180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:50:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.510+0000 7f5e5a321b80 4 client.25553 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:50:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.510+0000 7f5e5a321b80 1 -- 172.21.15.2:0/3163946342 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563a81b8c600 con 0x563a81cd9da0 2023-11-16T16:50:50.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.510+0000 7f5e3f7fe700 1 --2- 172.21.15.2:0/3163946342 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x563a81cd9da0 0x563a81cdc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:50:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.511+0000 7f5e3f7fe700 1 --2- 172.21.15.2:0/3163946342 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x563a81cd9da0 0x563a81cdc180 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfde131a39fb3939 server_cookie=25358856a63e55f8 in_seq=0 out_seq=0 2023-11-16T16:50:50.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.511+0000 7f5e1f7fe700 10 client.25553 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:50:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.518+0000 7f5e1f7fe700 1 -- 172.21.15.2:0/3163946342 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563a81b8c600 con 0x563a81cd9da0 2023-11-16T16:50:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.518+0000 7f5e1f7fe700 10 client.25553 handle_command_reply: tid=0 2023-11-16T16:50:50.519 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.518+0000 7f5e5a321b80 1 client.25553 shutdown 2023-11-16T16:50:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.519+0000 7f5e5a321b80 1 -- 172.21.15.2:0/3163946342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81ac86b0 msgr2=0x563a81c79240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.519+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/3163946342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81ac86b0 0x563a81c79240 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5e30097ac0 tx=0x7f5e30009310 comp rx=0 tx=0).stop 2023-11-16T16:50:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.519+0000 7f5e5a321b80 1 -- 172.21.15.2:0/3163946342 shutdown_connections 2023-11-16T16:50:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.519+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/3163946342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563a81c72b90 0x563a81c79cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.519+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/3163946342 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563a81ac86b0 0x563a81c79240 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.519+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/3163946342 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x563a81cd9da0 0x563a81cdc180 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.519+0000 7f5e5a321b80 1 --2- 172.21.15.2:0/3163946342 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563a81c72720 0x563a81c79780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.519+0000 7f5e1f7fe700 0 client.25553 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:50:50.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.519+0000 7f5e5a321b80 1 -- 172.21.15.2:0/3163946342 >> 172.21.15.2:0/3163946342 conn(0x563a81c61f50 msgr2=0x563a81c751c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:50.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.519+0000 7f5e5a321b80 1 -- 172.21.15.2:0/3163946342 shutdown_connections 2023-11-16T16:50:50.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.519+0000 7f5e5a321b80 1 -- 172.21.15.2:0/3163946342 wait complete. 2023-11-16T16:50:50.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.519+0000 7f5e5a321b80 20 client.25553 trim_cache size 0 max 16384 2023-11-16T16:50:50.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:50:50.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:50:50.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:50:50.522 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:50:50.522 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.520+0000 7f5e469ff700 1 -- 172.21.15.2:0/4014410918 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f5e280a13e0 msgr2=0x7f5e280a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.522 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.520+0000 7f5e469ff700 1 --2- 172.21.15.2:0/4014410918 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f5e280a13e0 0x7f5e280a3890 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5e40069fd0 tx=0x7f5e3800b040 comp rx=0 tx=0).stop 2023-11-16T16:50:50.522 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.520+0000 7f5e469ff700 1 -- 172.21.15.2:0/4014410918 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e400746a0 msgr2=0x7f5e401985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:50:50.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.520+0000 7f5e469ff700 1 --2- 172.21.15.2:0/4014410918 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e400746a0 0x7f5e401985e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5e30005910 tx=0x7f5e3000b610 comp rx=0 tx=0).stop 2023-11-16T16:50:50.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.520+0000 7f5e469ff700 1 -- 172.21.15.2:0/4014410918 shutdown_connections 2023-11-16T16:50:50.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.520+0000 7f5e469ff700 1 --2- 172.21.15.2:0/4014410918 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f5e280a13e0 0x7f5e280a3890 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.520+0000 7f5e469ff700 1 --2- 172.21.15.2:0/4014410918 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e4010ce50 0x7f5e4019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.520+0000 7f5e469ff700 1 --2- 172.21.15.2:0/4014410918 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e400746a0 0x7f5e401985e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.520+0000 7f5e469ff700 1 --2- 172.21.15.2:0/4014410918 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e40074b00 0x7f5e40198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:50:50.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.520+0000 7f5e469ff700 1 -- 172.21.15.2:0/4014410918 >> 172.21.15.2:0/4014410918 conn(0x7f5e40077550 msgr2=0x7f5e40104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:50:50.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.521+0000 7f5e469ff700 1 -- 172.21.15.2:0/4014410918 shutdown_connections 2023-11-16T16:50:50.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:50:50.521+0000 7f5e469ff700 1 -- 172.21.15.2:0/4014410918 wait complete. 2023-11-16T16:50:51.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:50 smithi141 ceph-mon[131184]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2023-11-16T16:50:51.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:50 smithi155 ceph-mon[131607]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2023-11-16T16:50:51.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:50 smithi002 ceph-mon[128712]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2023-11-16T16:50:53.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:52 smithi141 ceph-mon[131184]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2023-11-16T16:50:53.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:52 smithi155 ceph-mon[131607]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2023-11-16T16:50:53.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:52 smithi002 ceph-mon[128712]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2023-11-16T16:50:55.871 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:55 smithi155 ceph-mon[131607]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2023-11-16T16:50:55.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:55 smithi002 ceph-mon[128712]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2023-11-16T16:50:56.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:55 smithi141 ceph-mon[131184]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2023-11-16T16:50:58.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:50:57 smithi141 ceph-mon[131184]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2023-11-16T16:50:58.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:50:57 smithi155 ceph-mon[131607]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2023-11-16T16:50:58.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:50:57 smithi002 ceph-mon[128712]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2023-11-16T16:51:00.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:00 smithi155 ceph-mon[131607]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-11-16T16:51:00.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:00 smithi002 ceph-mon[128712]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-11-16T16:51:01.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:00 smithi141 ceph-mon[131184]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-11-16T16:51:02.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:02 smithi155 ceph-mon[131607]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 13 KiB/s wr, 10 op/s 2023-11-16T16:51:02.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:02 smithi002 ceph-mon[128712]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 13 KiB/s wr, 10 op/s 2023-11-16T16:51:03.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:02 smithi141 ceph-mon[131184]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 13 KiB/s wr, 10 op/s 2023-11-16T16:51:03.772 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.770+0000 7fa3209ff700 1 -- 172.21.15.2:0/713051484 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa31c1015f0 msgr2=0x7fa31c104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:03.772 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.770+0000 7fa3209ff700 1 --2- 172.21.15.2:0/713051484 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa31c1015f0 0x7fa31c104e80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa310009920 tx=0x7fa310009c30 comp rx=0 tx=0).stop 2023-11-16T16:51:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.770+0000 7fa3209ff700 1 -- 172.21.15.2:0/713051484 shutdown_connections 2023-11-16T16:51:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.770+0000 7fa3209ff700 1 --2- 172.21.15.2:0/713051484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa31c105560 0x7fa31c10d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.770+0000 7fa3209ff700 1 --2- 172.21.15.2:0/713051484 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa31c100c50 0x7fa31c101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.770+0000 7fa3209ff700 1 --2- 172.21.15.2:0/713051484 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa31c1015f0 0x7fa31c104e80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.770+0000 7fa3209ff700 1 -- 172.21.15.2:0/713051484 >> 172.21.15.2:0/713051484 conn(0x7fa31c077550 msgr2=0x7fa31c077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.771+0000 7fa3209ff700 1 -- 172.21.15.2:0/713051484 shutdown_connections 2023-11-16T16:51:03.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.771+0000 7fa3209ff700 1 -- 172.21.15.2:0/713051484 wait complete. 2023-11-16T16:51:03.774 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.772+0000 7fa3209ff700 1 Processor -- start 2023-11-16T16:51:03.774 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.773+0000 7fa3209ff700 1 -- start start 2023-11-16T16:51:03.775 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.773+0000 7fa3209ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa31c100c50 0x7fa31c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:03.775 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.773+0000 7fa3209ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa31c1015f0 0x7fa31c198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:03.775 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.774+0000 7fa3209ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa31c105560 0x7fa31c19bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.774+0000 7fa3209ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c19c5a0 con 0x7fa31c105560 2023-11-16T16:51:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.774+0000 7fa3209ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c19c710 con 0x7fa31c100c50 2023-11-16T16:51:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.774+0000 7fa3209ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c19c8d0 con 0x7fa31c1015f0 2023-11-16T16:51:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.774+0000 7fa31a59c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa31c100c50 0x7fa31c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.774+0000 7fa319d9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa31c1015f0 0x7fa31c198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.774+0000 7fa319d9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa31c1015f0 0x7fa31c198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:35784/0 (socket says 172.21.15.2:35784) 2023-11-16T16:51:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.774+0000 7fa319d9b700 1 -- 172.21.15.2:0/3288742296 learned_addr learned my addr 172.21.15.2:0/3288742296 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.774+0000 7fa31ad9d700 1 --2- 172.21.15.2:0/3288742296 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa31c105560 0x7fa31c19bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.774+0000 7fa319d9b700 1 -- 172.21.15.2:0/3288742296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa31c100c50 msgr2=0x7fa31c1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.775+0000 7fa319d9b700 1 --2- 172.21.15.2:0/3288742296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa31c100c50 0x7fa31c1986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.775+0000 7fa319d9b700 1 -- 172.21.15.2:0/3288742296 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa31c105560 msgr2=0x7fa31c19bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.775+0000 7fa319d9b700 1 --2- 172.21.15.2:0/3288742296 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa31c105560 0x7fa31c19bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.775+0000 7fa319d9b700 1 -- 172.21.15.2:0/3288742296 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa310009520 con 0x7fa31c1015f0 2023-11-16T16:51:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.775+0000 7fa31ad9d700 1 --2- 172.21.15.2:0/3288742296 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa31c105560 0x7fa31c19bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:51:03.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.775+0000 7fa319d9b700 1 --2- 172.21.15.2:0/3288742296 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa31c1015f0 0x7fa31c198bf0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa310009230 tx=0x7fa31000fa20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:03.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.776+0000 7fa30b7fe700 1 -- 172.21.15.2:0/3288742296 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31001c070 con 0x7fa31c1015f0 2023-11-16T16:51:03.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.776+0000 7fa3209ff700 1 -- 172.21.15.2:0/3288742296 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa31c110370 con 0x7fa31c1015f0 2023-11-16T16:51:03.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.776+0000 7fa3209ff700 1 -- 172.21.15.2:0/3288742296 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa31c1101e0 con 0x7fa31c1015f0 2023-11-16T16:51:03.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.776+0000 7fa30b7fe700 1 -- 172.21.15.2:0/3288742296 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa31000bb20 con 0x7fa31c1015f0 2023-11-16T16:51:03.779 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.776+0000 7fa30b7fe700 1 -- 172.21.15.2:0/3288742296 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa310004930 con 0x7fa31c1015f0 2023-11-16T16:51:03.783 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.781+0000 7fa30b7fe700 1 -- 172.21.15.2:0/3288742296 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7fa310020070 con 0x7fa31c1015f0 2023-11-16T16:51:03.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.782+0000 7fa334270b80 1 Processor -- start 2023-11-16T16:51:03.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.783+0000 7fa334270b80 1 -- start start 2023-11-16T16:51:03.785 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.784+0000 7fa334270b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e4226b0 0x55e05e5cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.784+0000 7fa30b7fe700 1 --2- 172.21.15.2:0/3288742296 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fa3040a13e0 0x7fa3040a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.784+0000 7fa334270b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5cc720 0x55e05e5c6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.784+0000 7fa334270b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e5ccb90 0x55e05e5c6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.784+0000 7fa334270b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05e4cc000 con 0x55e05e4226b0 2023-11-16T16:51:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.784+0000 7fa334270b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05e51e840 con 0x55e05e5cc720 2023-11-16T16:51:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.784+0000 7fa334270b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05e517cd0 con 0x55e05e5ccb90 2023-11-16T16:51:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.784+0000 7fa30b7fe700 1 -- 172.21.15.2:0/3288742296 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fa3100cdf70 con 0x7fa31c1015f0 2023-11-16T16:51:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.785+0000 7fa31ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e4226b0 0x55e05e5cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.785+0000 7fa31ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e4226b0 0x55e05e5cc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41306/0 (socket says 172.21.15.2:41306) 2023-11-16T16:51:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.785+0000 7fa31ad9d700 1 -- 172.21.15.2:0/932064430 learned_addr learned my addr 172.21.15.2:0/932064430 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.785+0000 7fa31a59c700 1 --2- 172.21.15.2:0/3288742296 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fa3040a13e0 0x7fa3040a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.785+0000 7fa31ad9d700 1 -- 172.21.15.2:0/932064430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e5ccb90 msgr2=0x55e05e5c6b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T16:51:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.785+0000 7fa31ad9d700 1 --2- 172.21.15.2:0/932064430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e5ccb90 0x55e05e5c6b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.785+0000 7fa31ad9d700 1 -- 172.21.15.2:0/932064430 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5cc720 msgr2=0x55e05e5c6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.785+0000 7fa31ad9d700 1 --2- 172.21.15.2:0/932064430 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5cc720 0x55e05e5c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.785+0000 7fa31ad9d700 1 -- 172.21.15.2:0/932064430 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e05e3995d0 con 0x55e05e4226b0 2023-11-16T16:51:03.788 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.786+0000 7fa31ad9d700 1 --2- 172.21.15.2:0/932064430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e4226b0 0x55e05e5cc1e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa314007740 tx=0x7fa3140099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a081325f0221de4a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:03.788 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.787+0000 7fa308ff9700 1 -- 172.21.15.2:0/932064430 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3140155f0 con 0x55e05e4226b0 2023-11-16T16:51:03.788 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.787+0000 7fa308ff9700 1 -- 172.21.15.2:0/932064430 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa314011040 con 0x55e05e4226b0 2023-11-16T16:51:03.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.787+0000 7fa308ff9700 1 -- 172.21.15.2:0/932064430 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa314014a20 con 0x55e05e4226b0 2023-11-16T16:51:03.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.788+0000 7fa334270b80 1 -- 172.21.15.2:0/932064430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e4226b0 msgr2=0x55e05e5cc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:03.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.788+0000 7fa31a59c700 1 --2- 172.21.15.2:0/3288742296 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fa3040a13e0 0x7fa3040a3890 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa30c009ca0 tx=0x7fa30c009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.788+0000 7fa334270b80 1 --2- 172.21.15.2:0/932064430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e4226b0 0x55e05e5cc1e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa314007740 tx=0x7fa3140099e0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.789+0000 7fa334270b80 1 -- 172.21.15.2:0/932064430 shutdown_connections 2023-11-16T16:51:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.789+0000 7fa334270b80 1 --2- 172.21.15.2:0/932064430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e5ccb90 0x55e05e5c6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.789+0000 7fa334270b80 1 --2- 172.21.15.2:0/932064430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e4226b0 0x55e05e5cc1e0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.789+0000 7fa334270b80 1 --2- 172.21.15.2:0/932064430 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5cc720 0x55e05e5c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.789+0000 7fa334270b80 1 -- 172.21.15.2:0/932064430 >> 172.21.15.2:0/932064430 conn(0x55e05e5bbf50 msgr2=0x55e05e5c1270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.789+0000 7fa334270b80 1 -- 172.21.15.2:0/932064430 shutdown_connections 2023-11-16T16:51:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.789+0000 7fa334270b80 1 -- 172.21.15.2:0/932064430 wait complete. 2023-11-16T16:51:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.790+0000 7fa334270b80 1 -- start start 2023-11-16T16:51:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.790+0000 7fa334270b80 1 Processor -- start 2023-11-16T16:51:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.791+0000 7fa334270b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e4226b0 0x55e05e5cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.791+0000 7fa334270b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e5cc720 0x55e05e5d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.791+0000 7fa334270b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5ccb90 0x55e05e5d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.791+0000 7fa334270b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05e4b9880 con 0x55e05e5cc720 2023-11-16T16:51:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.791+0000 7fa334270b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05e4b9520 con 0x55e05e5ccb90 2023-11-16T16:51:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.791+0000 7fa334270b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05e39b8c0 con 0x55e05e4226b0 2023-11-16T16:51:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.791+0000 7fa334270b80 10 client.0 _ll_register_callbacks cb 0x55e05e4d7720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.791+0000 7fa31a59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e4226b0 0x55e05e5cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.791+0000 7fa31a59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e4226b0 0x55e05e5cfda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:35790/0 (socket says 172.21.15.2:35790) 2023-11-16T16:51:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.791+0000 7fa31a59c700 1 -- 172.21.15.2:0/3013022773 learned_addr learned my addr 172.21.15.2:0/3013022773 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.792+0000 7fa31ad9d700 1 --2- 172.21.15.2:0/3013022773 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5ccb90 0x55e05e5d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.792+0000 7fa319d9b700 1 --2- 172.21.15.2:0/3013022773 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e5cc720 0x55e05e5d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.792+0000 7fa31a59c700 1 -- 172.21.15.2:0/3013022773 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5ccb90 msgr2=0x55e05e5d0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.792+0000 7fa31a59c700 1 --2- 172.21.15.2:0/3013022773 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5ccb90 0x55e05e5d0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.792+0000 7fa31a59c700 1 -- 172.21.15.2:0/3013022773 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e5cc720 msgr2=0x55e05e5d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.792+0000 7fa31a59c700 1 --2- 172.21.15.2:0/3013022773 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e5cc720 0x55e05e5d02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.792+0000 7fa31a59c700 1 -- 172.21.15.2:0/3013022773 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3140091a0 con 0x55e05e4226b0 2023-11-16T16:51:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.792+0000 7fa319d9b700 1 --2- 172.21.15.2:0/3013022773 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e5cc720 0x55e05e5d02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:51:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.792+0000 7fa31a59c700 1 --2- 172.21.15.2:0/3013022773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e4226b0 0x55e05e5cfda0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa30c0042e0 tx=0x7fa30c0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f6a9f61e4630345 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.792+0000 7fa301ffb700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:51:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.793+0000 7fa301ffb700 1 -- 172.21.15.2:0/3013022773 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c003a90 con 0x55e05e4226b0 2023-11-16T16:51:03.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.793+0000 7fa301ffb700 1 -- 172.21.15.2:0/3013022773 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fa30c018070 con 0x55e05e4226b0 2023-11-16T16:51:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.793+0000 7fa334270b80 1 -- 172.21.15.2:0/3013022773 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e05e5b9710 con 0x55e05e4226b0 2023-11-16T16:51:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.794+0000 7fa301ffb700 1 -- 172.21.15.2:0/3013022773 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa30c003bf0 con 0x55e05e4226b0 2023-11-16T16:51:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.794+0000 7fa334270b80 10 client.25531 fetch_fsmap learned FSMap version 28 2023-11-16T16:51:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.794+0000 7fa334270b80 1 -- 172.21.15.2:0/3013022773 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e05e4de130 con 0x55e05e4226b0 2023-11-16T16:51:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.794+0000 7fa301ffb700 1 -- 172.21.15.2:0/3013022773 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa30c02d030 con 0x55e05e4226b0 2023-11-16T16:51:03.797 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.796+0000 7fa301ffb700 1 -- 172.21.15.2:0/3013022773 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa30c027060 con 0x55e05e4226b0 2023-11-16T16:51:03.797 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.796+0000 7fa334270b80 10 client.25531 fetch_fsmap finished waiting for FSMap version 28 2023-11-16T16:51:03.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.796+0000 7fa334270b80 10 client.25531 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:03.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.796+0000 7fa334270b80 20 client.25531 populate_metadata read hostname 'smithi002' 2023-11-16T16:51:03.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.796+0000 7fa334270b80 1 --2- 172.21.15.2:0/3013022773 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55e05e5e8900 0x55e05e5ead10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:03.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.796+0000 7fa334270b80 4 client.25531 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:51:03.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.796+0000 7fa334270b80 1 -- 172.21.15.2:0/3013022773 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e05e4da040 con 0x55e05e5e8900 2023-11-16T16:51:03.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.796+0000 7fa319d9b700 1 --2- 172.21.15.2:0/3013022773 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55e05e5e8900 0x55e05e5ead10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:03.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.797+0000 7fa319d9b700 1 --2- 172.21.15.2:0/3013022773 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55e05e5e8900 0x55e05e5ead10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65ba1fdb261bf6e5 server_cookie=a4a567303ba019f3 in_seq=0 out_seq=0 2023-11-16T16:51:03.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.797+0000 7fa301ffb700 10 client.25531 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:51:03.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.797+0000 7fa301ffb700 1 -- 172.21.15.2:0/3013022773 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55e05e4da040 con 0x55e05e5e8900 2023-11-16T16:51:03.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.798+0000 7fa301ffb700 10 client.25531 handle_command_reply: tid=0 2023-11-16T16:51:03.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.798+0000 7fa334270b80 1 client.25531 shutdown 2023-11-16T16:51:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.798+0000 7fa334270b80 1 -- 172.21.15.2:0/3013022773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e4226b0 msgr2=0x55e05e5cfda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.798+0000 7fa334270b80 1 --2- 172.21.15.2:0/3013022773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e4226b0 0x55e05e5cfda0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa30c0042e0 tx=0x7fa30c0047c0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.798+0000 7fa334270b80 1 -- 172.21.15.2:0/3013022773 shutdown_connections 2023-11-16T16:51:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.798+0000 7fa334270b80 1 --2- 172.21.15.2:0/3013022773 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55e05e5e8900 0x55e05e5ead10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.798+0000 7fa334270b80 1 --2- 172.21.15.2:0/3013022773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e4226b0 0x55e05e5cfda0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.798+0000 7fa334270b80 1 --2- 172.21.15.2:0/3013022773 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5ccb90 0x55e05e5d0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.798+0000 7fa334270b80 1 --2- 172.21.15.2:0/3013022773 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e5cc720 0x55e05e5d02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.798+0000 7fa334270b80 1 -- 172.21.15.2:0/3013022773 >> 172.21.15.2:0/3013022773 conn(0x55e05e5bbf50 msgr2=0x55e05e5c07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:03.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.798+0000 7fa301ffb700 0 client.25531 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:51:03.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.799+0000 7fa334270b80 1 -- 172.21.15.2:0/3013022773 shutdown_connections 2023-11-16T16:51:03.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.799+0000 7fa334270b80 1 -- 172.21.15.2:0/3013022773 wait complete. 2023-11-16T16:51:03.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.799+0000 7fa334270b80 20 client.25531 trim_cache size 0 max 16384 2023-11-16T16:51:03.834 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.832+0000 7fa334270b80 1 Processor -- start 2023-11-16T16:51:03.834 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.832+0000 7fa334270b80 1 -- start start 2023-11-16T16:51:03.834 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.832+0000 7fa334270b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e4226b0 0x55e05e5c0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.832+0000 7fa334270b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e5cc720 0x55e05e5c6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.832+0000 7fa334270b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5ccb90 0x55e05e5c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.832+0000 7fa334270b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05e5bdff0 con 0x55e05e4226b0 2023-11-16T16:51:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.832+0000 7fa334270b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05e5d2e30 con 0x55e05e5ccb90 2023-11-16T16:51:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.832+0000 7fa334270b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05e5c23e0 con 0x55e05e5cc720 2023-11-16T16:51:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.833+0000 7fa31ad9d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5ccb90 0x55e05e5c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.833+0000 7fa31ad9d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5ccb90 0x55e05e5c6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:57500/0 (socket says 172.21.15.2:57500) 2023-11-16T16:51:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.833+0000 7fa31ad9d700 1 -- 172.21.15.2:0/1899234981 learned_addr learned my addr 172.21.15.2:0/1899234981 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.833+0000 7fa319d9b700 1 --2- 172.21.15.2:0/1899234981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e5cc720 0x55e05e5c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:03.836 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.833+0000 7fa31a59c700 1 --2- 172.21.15.2:0/1899234981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e4226b0 0x55e05e5c0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.833+0000 7fa31ad9d700 1 -- 172.21.15.2:0/1899234981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e5cc720 msgr2=0x55e05e5c6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.833+0000 7fa31ad9d700 1 --2- 172.21.15.2:0/1899234981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e5cc720 0x55e05e5c6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.833+0000 7fa31ad9d700 1 -- 172.21.15.2:0/1899234981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e4226b0 msgr2=0x55e05e5c0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.834+0000 7fa31ad9d700 1 --2- 172.21.15.2:0/1899234981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e4226b0 0x55e05e5c0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.834+0000 7fa31ad9d700 1 -- 172.21.15.2:0/1899234981 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e05e5bd6f0 con 0x55e05e5ccb90 2023-11-16T16:51:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.834+0000 7fa31a59c700 1 --2- 172.21.15.2:0/1899234981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e4226b0 0x55e05e5c0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:51:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.834+0000 7fa31ad9d700 1 --2- 172.21.15.2:0/1899234981 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5ccb90 0x55e05e5c6aa0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa314007740 tx=0x7fa3140144c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d111b4be7ea89e8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.835+0000 7fa3097fa700 1 -- 172.21.15.2:0/1899234981 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa314014870 con 0x55e05e5ccb90 2023-11-16T16:51:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.835+0000 7fa3097fa700 1 -- 172.21.15.2:0/1899234981 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa314014e90 con 0x55e05e5ccb90 2023-11-16T16:51:03.837 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.835+0000 7fa3097fa700 1 -- 172.21.15.2:0/1899234981 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31401ee40 con 0x55e05e5ccb90 2023-11-16T16:51:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.836+0000 7fa334270b80 1 -- 172.21.15.2:0/1899234981 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5ccb90 msgr2=0x55e05e5c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.836+0000 7fa334270b80 1 --2- 172.21.15.2:0/1899234981 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5ccb90 0x55e05e5c6aa0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa314007740 tx=0x7fa3140144c0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.838 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.836+0000 7fa334270b80 1 -- 172.21.15.2:0/1899234981 shutdown_connections 2023-11-16T16:51:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.836+0000 7fa334270b80 1 --2- 172.21.15.2:0/1899234981 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5ccb90 0x55e05e5c6aa0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.836+0000 7fa334270b80 1 --2- 172.21.15.2:0/1899234981 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e4226b0 0x55e05e5c0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.836+0000 7fa334270b80 1 --2- 172.21.15.2:0/1899234981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e5cc720 0x55e05e5c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.836+0000 7fa334270b80 1 -- 172.21.15.2:0/1899234981 >> 172.21.15.2:0/1899234981 conn(0x55e05e5bbf50 msgr2=0x55e05e5c9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.836+0000 7fa334270b80 1 -- 172.21.15.2:0/1899234981 shutdown_connections 2023-11-16T16:51:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.836+0000 7fa334270b80 1 -- 172.21.15.2:0/1899234981 wait complete. 2023-11-16T16:51:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.837+0000 7fa334270b80 1 -- start start 2023-11-16T16:51:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.837+0000 7fa334270b80 1 Processor -- start 2023-11-16T16:51:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.838+0000 7fa334270b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e4226b0 0x55e05e5dd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.838+0000 7fa334270b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5cc720 0x55e05e5dde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.838+0000 7fa334270b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e5ccb90 0x55e05e5de3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.838+0000 7fa334270b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05e63b8a0 con 0x55e05e5ccb90 2023-11-16T16:51:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.838+0000 7fa334270b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05e63ba10 con 0x55e05e5cc720 2023-11-16T16:51:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.838+0000 7fa334270b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e05e63bbd0 con 0x55e05e4226b0 2023-11-16T16:51:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.838+0000 7fa334270b80 10 client.0 _ll_register_callbacks cb 0x7fa30c011d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.839+0000 7fa31a59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e4226b0 0x55e05e5dd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.839+0000 7fa31a59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e4226b0 0x55e05e5dd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:35812/0 (socket says 172.21.15.2:35812) 2023-11-16T16:51:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.839+0000 7fa31a59c700 1 -- 172.21.15.2:0/2058125192 learned_addr learned my addr 172.21.15.2:0/2058125192 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.839+0000 7fa319d9b700 1 --2- 172.21.15.2:0/2058125192 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5cc720 0x55e05e5dde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.839+0000 7fa31ad9d700 1 --2- 172.21.15.2:0/2058125192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e5ccb90 0x55e05e5de3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.840+0000 7fa319d9b700 1 -- 172.21.15.2:0/2058125192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e4226b0 msgr2=0x55e05e5dd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.840+0000 7fa319d9b700 1 --2- 172.21.15.2:0/2058125192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e4226b0 0x55e05e5dd920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.840+0000 7fa319d9b700 1 -- 172.21.15.2:0/2058125192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e5ccb90 msgr2=0x55e05e5de3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.840+0000 7fa319d9b700 1 --2- 172.21.15.2:0/2058125192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e5ccb90 0x55e05e5de3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.840+0000 7fa319d9b700 1 -- 172.21.15.2:0/2058125192 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e05e3995d0 con 0x55e05e5cc720 2023-11-16T16:51:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.840+0000 7fa31ad9d700 1 --2- 172.21.15.2:0/2058125192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e5ccb90 0x55e05e5de3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.840+0000 7fa319d9b700 1 --2- 172.21.15.2:0/2058125192 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5cc720 0x55e05e5dde60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa3100cb170 tx=0x7fa310009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8fcfb954de1b931 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.840+0000 7fa301ffb700 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2023-11-16T16:51:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.841+0000 7fa301ffb700 1 -- 172.21.15.2:0/2058125192 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31009ad10 con 0x55e05e5cc720 2023-11-16T16:51:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.841+0000 7fa301ffb700 1 -- 172.21.15.2:0/2058125192 <== mon.2 v2:172.21.15.141:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fa31009ae70 con 0x55e05e5cc720 2023-11-16T16:51:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.841+0000 7fa334270b80 1 -- 172.21.15.2:0/2058125192 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa314011040 con 0x55e05e5cc720 2023-11-16T16:51:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.842+0000 7fa301ffb700 1 -- 172.21.15.2:0/2058125192 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa310099fa0 con 0x55e05e5cc720 2023-11-16T16:51:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.842+0000 7fa334270b80 10 client.25568 fetch_fsmap learned FSMap version 28 2023-11-16T16:51:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.842+0000 7fa334270b80 1 -- 172.21.15.2:0/2058125192 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e05e345d80 con 0x55e05e5cc720 2023-11-16T16:51:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.842+0000 7fa301ffb700 1 -- 172.21.15.2:0/2058125192 <== mon.2 v2:172.21.15.141:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa314011040 con 0x55e05e5cc720 2023-11-16T16:51:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.842+0000 7fa301ffb700 1 -- 172.21.15.2:0/2058125192 <== mon.2 v2:172.21.15.141:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa3100c7070 con 0x55e05e5cc720 2023-11-16T16:51:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.842+0000 7fa334270b80 10 client.25568 fetch_fsmap finished waiting for FSMap version 28 2023-11-16T16:51:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.843+0000 7fa334270b80 10 client.25568 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.843+0000 7fa334270b80 20 client.25568 populate_metadata read hostname 'smithi002' 2023-11-16T16:51:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.843+0000 7fa334270b80 1 --2- 172.21.15.2:0/2058125192 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55e05e63cf60 0x55e05e63f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.843+0000 7fa334270b80 4 client.25568 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:51:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.843+0000 7fa334270b80 1 -- 172.21.15.2:0/2058125192 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e05e4e6600 con 0x55e05e63cf60 2023-11-16T16:51:03.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.843+0000 7fa31a59c700 1 --2- 172.21.15.2:0/2058125192 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55e05e63cf60 0x55e05e63f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.844+0000 7fa31a59c700 1 --2- 172.21.15.2:0/2058125192 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55e05e63cf60 0x55e05e63f370 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad494125138582a2 server_cookie=741fbb474e037887 in_seq=0 out_seq=0 2023-11-16T16:51:03.846 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:03.844+0000 7fa301ffb700 10 client.25568 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:51:03.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:03 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:51:03.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:03 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:51:04.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:03 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:51:04.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:04 smithi155 ceph-mon[131607]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 18 op/s 2023-11-16T16:51:04.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:04 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:51:04.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:04 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:51:04.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:04 smithi002 ceph-mon[128712]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 18 op/s 2023-11-16T16:51:04.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:04 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:51:04.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:04 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:51:05.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:04 smithi141 ceph-mon[131184]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 18 op/s 2023-11-16T16:51:05.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:04 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:51:05.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:04 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:51:05.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.314+0000 7fa301ffb700 1 -- 172.21.15.2:0/2058125192 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e05e4e6600 con 0x55e05e63cf60 2023-11-16T16:51:05.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.314+0000 7fa301ffb700 10 client.25568 handle_command_reply: tid=0 2023-11-16T16:51:05.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.314+0000 7fa334270b80 1 client.25568 shutdown 2023-11-16T16:51:05.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.315+0000 7fa334270b80 1 -- 172.21.15.2:0/2058125192 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5cc720 msgr2=0x55e05e5dde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.315+0000 7fa334270b80 1 --2- 172.21.15.2:0/2058125192 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5cc720 0x55e05e5dde60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa3100cb170 tx=0x7fa310009260 comp rx=0 tx=0).stop 2023-11-16T16:51:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.315+0000 7fa334270b80 1 -- 172.21.15.2:0/2058125192 shutdown_connections 2023-11-16T16:51:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.315+0000 7fa334270b80 1 --2- 172.21.15.2:0/2058125192 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55e05e63cf60 0x55e05e63f370 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.315+0000 7fa334270b80 1 --2- 172.21.15.2:0/2058125192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e05e4226b0 0x55e05e5dd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.315+0000 7fa301ffb700 0 client.25568 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:51:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.315+0000 7fa334270b80 1 --2- 172.21.15.2:0/2058125192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e05e5ccb90 0x55e05e5de3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.315+0000 7fa334270b80 1 --2- 172.21.15.2:0/2058125192 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e05e5cc720 0x55e05e5dde60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.315+0000 7fa334270b80 1 -- 172.21.15.2:0/2058125192 >> 172.21.15.2:0/2058125192 conn(0x55e05e5bbf50 msgr2=0x55e05e5ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.315+0000 7fa334270b80 1 -- 172.21.15.2:0/2058125192 shutdown_connections 2023-11-16T16:51:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.315+0000 7fa334270b80 1 -- 172.21.15.2:0/2058125192 wait complete. 2023-11-16T16:51:05.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.316+0000 7fa334270b80 20 client.25568 trim_cache size 0 max 16384 2023-11-16T16:51:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:51:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:51:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:51:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:51:05.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.316+0000 7fa3209ff700 1 -- 172.21.15.2:0/3288742296 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fa3040a13e0 msgr2=0x7fa3040a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.316+0000 7fa3209ff700 1 --2- 172.21.15.2:0/3288742296 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fa3040a13e0 0x7fa3040a3890 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa30c009ca0 tx=0x7fa30c009260 comp rx=0 tx=0).stop 2023-11-16T16:51:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.317+0000 7fa3209ff700 1 -- 172.21.15.2:0/3288742296 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa31c1015f0 msgr2=0x7fa31c198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.317+0000 7fa3209ff700 1 --2- 172.21.15.2:0/3288742296 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa31c1015f0 0x7fa31c198bf0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa310009230 tx=0x7fa31000fa20 comp rx=0 tx=0).stop 2023-11-16T16:51:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.317+0000 7fa3209ff700 1 -- 172.21.15.2:0/3288742296 shutdown_connections 2023-11-16T16:51:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.317+0000 7fa3209ff700 1 --2- 172.21.15.2:0/3288742296 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fa3040a13e0 0x7fa3040a3890 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.317+0000 7fa3209ff700 1 --2- 172.21.15.2:0/3288742296 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa31c105560 0x7fa31c19bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.317+0000 7fa3209ff700 1 --2- 172.21.15.2:0/3288742296 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fa31c100c50 0x7fa31c1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.317+0000 7fa3209ff700 1 --2- 172.21.15.2:0/3288742296 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa31c1015f0 0x7fa31c198bf0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.317+0000 7fa3209ff700 1 -- 172.21.15.2:0/3288742296 >> 172.21.15.2:0/3288742296 conn(0x7fa31c077550 msgr2=0x7fa31c0ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.317+0000 7fa3209ff700 1 -- 172.21.15.2:0/3288742296 shutdown_connections 2023-11-16T16:51:05.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:05.317+0000 7fa3209ff700 1 -- 172.21.15.2:0/3288742296 wait complete. 2023-11-16T16:51:06.119 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: *** Caught signal (Segmentation fault) ** 2023-11-16T16:51:06.119 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: in thread 7f84802a5700 thread_name:md_log_replay 2023-11-16T16:51:06.119 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev) 2023-11-16T16:51:06.119 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8490e54cf0] 2023-11-16T16:51:06.119 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 2: /usr/bin/ceph-mds(+0x4c1cff) [0x559f78d11cff] 2023-11-16T16:51:06.119 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x559f78d13f33] 2023-11-16T16:51:06.119 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 4: (MDLog::_replay_thread()+0x66d) [0x559f78d7fb9d] 2023-11-16T16:51:06.120 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 5: (MDLog::ReplayThread::entry()+0x11) [0x559f78a2f2c1] 2023-11-16T16:51:06.120 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f8490e4a1ca] 2023-11-16T16:51:06.120 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 7: clone() 2023-11-16T16:51:06.120 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 2023-11-16T16:51:05.752+0000 7f84802a5700 -1 *** Caught signal (Segmentation fault) ** 2023-11-16T16:51:06.120 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: in thread 7f84802a5700 thread_name:md_log_replay 2023-11-16T16:51:06.120 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 2023-11-16T16:51:06.120 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev) 2023-11-16T16:51:06.120 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8490e54cf0] 2023-11-16T16:51:06.120 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 2: /usr/bin/ceph-mds(+0x4c1cff) [0x559f78d11cff] 2023-11-16T16:51:06.120 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x559f78d13f33] 2023-11-16T16:51:06.120 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 4: (MDLog::_replay_thread()+0x66d) [0x559f78d7fb9d] 2023-11-16T16:51:06.120 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 5: (MDLog::ReplayThread::entry()+0x11) [0x559f78a2f2c1] 2023-11-16T16:51:06.120 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f8490e4a1ca] 2023-11-16T16:51:06.120 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 7: clone() 2023-11-16T16:51:06.121 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-16T16:51:06.121 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 2023-11-16T16:51:06.121 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 0> 2023-11-16T16:51:05.752+0000 7f84802a5700 -1 *** Caught signal (Segmentation fault) ** 2023-11-16T16:51:06.121 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: in thread 7f84802a5700 thread_name:md_log_replay 2023-11-16T16:51:06.121 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 2023-11-16T16:51:06.121 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev) 2023-11-16T16:51:06.121 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8490e54cf0] 2023-11-16T16:51:06.121 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 2: /usr/bin/ceph-mds(+0x4c1cff) [0x559f78d11cff] 2023-11-16T16:51:06.121 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x559f78d13f33] 2023-11-16T16:51:06.121 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 4: (MDLog::_replay_thread()+0x66d) [0x559f78d7fb9d] 2023-11-16T16:51:06.121 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 5: (MDLog::ReplayThread::entry()+0x11) [0x559f78a2f2c1] 2023-11-16T16:51:06.121 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f8490e4a1ca] 2023-11-16T16:51:06.121 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 7: clone() 2023-11-16T16:51:06.121 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-16T16:51:06.121 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 2023-11-16T16:51:06.122 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 0> 2023-11-16T16:51:05.752+0000 7f84802a5700 -1 *** Caught signal (Segmentation fault) ** 2023-11-16T16:51:06.122 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: in thread 7f84802a5700 thread_name:md_log_replay 2023-11-16T16:51:06.122 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 2023-11-16T16:51:06.122 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: ceph version 18.0.0-3905-g5e4a743a (5e4a743ad7fd1422ebd6c4dc950a21a303279207) reef (dev) 2023-11-16T16:51:06.122 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8490e54cf0] 2023-11-16T16:51:06.122 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 2: /usr/bin/ceph-mds(+0x4c1cff) [0x559f78d11cff] 2023-11-16T16:51:06.122 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x559f78d13f33] 2023-11-16T16:51:06.122 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 4: (MDLog::_replay_thread()+0x66d) [0x559f78d7fb9d] 2023-11-16T16:51:06.122 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 5: (MDLog::ReplayThread::entry()+0x11) [0x559f78a2f2c1] 2023-11-16T16:51:06.122 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f8490e4a1ca] 2023-11-16T16:51:06.122 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 7: clone() 2023-11-16T16:51:06.122 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-16T16:51:06.122 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[157320]: 2023-11-16T16:51:06.341 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:06 smithi155 systemd[1]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.c.service: Main process exited, code=exited, status=139/n/a 2023-11-16T16:51:06.542 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:06 smithi155 systemd[1]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.c.service: Failed with result 'exit-code'. 2023-11-16T16:51:06.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:06 smithi155 ceph-mon[131607]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2023-11-16T16:51:06.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:06 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:51:06.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:06 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:51:06.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:06 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:51:06.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:06 smithi002 ceph-mon[128712]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2023-11-16T16:51:06.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:06 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:51:06.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:06 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:51:06.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:06 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:51:07.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:06 smithi141 ceph-mon[131184]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2023-11-16T16:51:07.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:06 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:51:07.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:06 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:51:07.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:06 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:51:07.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:07 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:51:07.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:07 smithi155 ceph-mon[131607]: Service mds.all has no related daemon already placed. 2023-11-16T16:51:07.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:07 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:51:07.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:07 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:51:07.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:07 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:51:07.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:07 smithi002 ceph-mon[128712]: Service mds.all has no related daemon already placed. 2023-11-16T16:51:07.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:07 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:51:07.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:07 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:51:08.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:07 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:51:08.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:07 smithi141 ceph-mon[131184]: Service mds.all has no related daemon already placed. 2023-11-16T16:51:08.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:07 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:51:08.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:07 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:51:08.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:08 smithi155 ceph-mon[131607]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2023-11-16T16:51:08.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:08 smithi002 ceph-mon[128712]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2023-11-16T16:51:09.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:08 smithi141 ceph-mon[131184]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2023-11-16T16:51:10.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:10 smithi155 ceph-mon[131607]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2023-11-16T16:51:10.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:10 smithi002 ceph-mon[128712]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2023-11-16T16:51:11.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:10 smithi141 ceph-mon[131184]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2023-11-16T16:51:11.405 INFO:tasks.workunit.client.0.smithi002.stderr:Updating files: 62% (7509/12074) Updating files: 63% (7607/12074) Updating files: 64% (7728/12074) Updating files: 65% (7849/12074) Updating files: 66% (7969/12074) Updating files: 67% (8090/12074) Updating files: 68% (8211/12074) Updating files: 69% (8332/12074) Updating files: 70% (8452/12074) Updating files: 71% (8573/12074) Updating files: 72% (8694/12074) Updating files: 73% (8815/12074) Updating files: 74% (8935/12074) Updating files: 75% (9056/12074) Updating files: 76% (9177/12074) Updating files: 77% (9297/12074) Updating files: 78% (9418/12074) Updating files: 79% (9539/12074) Updating files: 80% (9660/12074) Updating files: 81% (9780/12074) Updating files: 82% (9901/12074) Updating files: 83% (10022/12074) Updating files: 84% (10143/12074) Updating files: 85% (10263/12074) Updating files: 86% (10384/12074) Updating files: 87% (10505/12074) Updating files: 88% (10626/12074) Updating files: 89% (10746/12074) Updating files: 90% (10867/12074) Updating files: 91% (10988/12074) Updating files: 92% (11109/12074) Updating files: 93% (11229/12074) Updating files: 94% (11350/12074) Updating files: 95% (11471/12074) Updating files: 96% (11592/12074) Updating files: 97% (11712/12074) Updating files: 98% (11833/12074) Updating files: 99% (11954/12074) Updating files: 100% (12074/12074) Updating files: 100% (12074/12074), done. 2023-11-16T16:51:12.415 INFO:tasks.workunit.client.0.smithi002.stderr:Note: switching to '5e4a743ad7fd1422ebd6c4dc950a21a303279207'. 2023-11-16T16:51:12.415 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-11-16T16:51:12.415 INFO:tasks.workunit.client.0.smithi002.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-11-16T16:51:12.415 INFO:tasks.workunit.client.0.smithi002.stderr:changes and commit them, and you can discard any commits you make in this 2023-11-16T16:51:12.415 INFO:tasks.workunit.client.0.smithi002.stderr:state without impacting any branches by switching back to a branch. 2023-11-16T16:51:12.416 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-11-16T16:51:12.416 INFO:tasks.workunit.client.0.smithi002.stderr:If you want to create a new branch to retain commits you create, you may 2023-11-16T16:51:12.416 INFO:tasks.workunit.client.0.smithi002.stderr:do so (now or later) by using -c with the switch command. Example: 2023-11-16T16:51:12.416 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-11-16T16:51:12.416 INFO:tasks.workunit.client.0.smithi002.stderr: git switch -c 2023-11-16T16:51:12.416 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-11-16T16:51:12.416 INFO:tasks.workunit.client.0.smithi002.stderr:Or undo this operation with: 2023-11-16T16:51:12.416 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-11-16T16:51:12.416 INFO:tasks.workunit.client.0.smithi002.stderr: git switch - 2023-11-16T16:51:12.416 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-11-16T16:51:12.416 INFO:tasks.workunit.client.0.smithi002.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-11-16T16:51:12.416 INFO:tasks.workunit.client.0.smithi002.stderr: 2023-11-16T16:51:12.416 INFO:tasks.workunit.client.0.smithi002.stderr:HEAD is now at 5e4a743ad7f doc/cephfs: Update docs to match remove functionality and respective vxattrs 2023-11-16T16:51:12.434 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-11-16T16:51:12.534 INFO:tasks.workunit.client.0.smithi002.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-11-16T16:51:12.539 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-16T16:51:12.539 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-11-16T16:51:12.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:12 smithi155 ceph-mon[131607]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2023-11-16T16:51:12.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:12 smithi002 ceph-mon[128712]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2023-11-16T16:51:13.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:12 smithi141 ceph-mon[131184]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2023-11-16T16:51:13.140 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-11-16T16:51:13.228 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-11-16T16:51:13.295 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-11-16T16:51:13.299 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-16T16:51:13.299 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-11-16T16:51:13.364 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-11-16T16:51:13.371 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T16:51:13.371 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-11-16T16:51:13.415 INFO:tasks.workunit:Running workunits matching fs/misc on client.0... 2023-11-16T16:51:13.417 INFO:tasks.workunit:Running workunit fs/misc/acl.sh... 2023-11-16T16:51:13.417 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/acl.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/acl.sh 2023-11-16T16:51:13.512 INFO:tasks.workunit.client.0.smithi002.stderr:+ set -e 2023-11-16T16:51:13.512 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir -p testdir 2023-11-16T16:51:13.516 INFO:tasks.workunit.client.0.smithi002.stderr:+ cd testdir 2023-11-16T16:51:13.517 INFO:tasks.workunit.client.0.smithi002.stderr:+ set +e 2023-11-16T16:51:13.517 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfacl -d -m u:nobody:rw . 2023-11-16T16:51:13.538 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2023-11-16T16:51:13.538 INFO:tasks.workunit.client.0.smithi002.stderr:+ set -e 2023-11-16T16:51:13.539 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=0 2023-11-16T16:51:13.539 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 0 -lt 100 ']' 2023-11-16T16:51:13.539 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 0 + 1 2023-11-16T16:51:13.541 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=1 2023-11-16T16:51:13.541 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:13.546 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:13.546 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:13.554 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:13.555 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:13.555 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:14.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:14 smithi155 ceph-mon[131607]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 12 KiB/s wr, 16 op/s 2023-11-16T16:51:14.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:14 smithi002 ceph-mon[128712]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 12 KiB/s wr, 16 op/s 2023-11-16T16:51:15.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:14 smithi141 ceph-mon[131184]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 12 KiB/s wr, 16 op/s 2023-11-16T16:51:16.326 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:16.326 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:16.479 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:16.479 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:16.502 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:16.502 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:16.502 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 1 -lt 100 ']' 2023-11-16T16:51:16.503 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 1 + 1 2023-11-16T16:51:16.533 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=2 2023-11-16T16:51:16.533 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:16.568 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:16.568 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:16.571 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:16.571 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:16.572 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:16.668 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:16 smithi155 ceph-mon[131607]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2023-11-16T16:51:16.668 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:16 smithi155 systemd[1]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.c.service: Service RestartSec=10s expired, scheduling restart. 2023-11-16T16:51:16.668 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:16 smithi155 systemd[1]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.c.service: Scheduled restart job, restart counter is at 1. 2023-11-16T16:51:16.668 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:16 smithi155 systemd[1]: Stopped Ceph mds.c for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:51:16.668 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:16 smithi155 systemd[1]: Starting Ceph mds.c for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T16:51:16.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:16 smithi002 ceph-mon[128712]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2023-11-16T16:51:16.928 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:16 smithi155 podman[161152]: 2023-11-16T16:51:17.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:16 smithi141 ceph-mon[131184]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2023-11-16T16:51:17.126 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:17.126 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:17.245 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:17.313 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:17.314 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:17.314 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:17.314 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -lt 100 ']' 2023-11-16T16:51:17.314 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 2 + 1 2023-11-16T16:51:17.314 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=3 2023-11-16T16:51:17.314 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:17.335 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:17.335 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:17.338 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:17.338 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:17.339 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:17.367 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:16 smithi155 ceph-mds[161192]: -- 172.21.15.155:0/3198710412 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561bf9519520 con 0x561bf94ee800 2023-11-16T16:51:17.367 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:16 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[161168]: starting mds.c at 2023-11-16T16:51:17.367 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:16 smithi155 bash[161152]: 8cf9fb8d0dda8c930ec17a4b2e7e63b79e1d601b1641099393ee2d8586a77a29 2023-11-16T16:51:17.368 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 16:51:16 smithi155 systemd[1]: Started Ceph mds.c for 875fcd24-849e-11ee-8dbe-212e2dc638e7. 2023-11-16T16:51:17.708 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:17.709 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:17.820 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:17.821 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:17.836 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:17.836 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:17.836 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 3 -lt 100 ']' 2023-11-16T16:51:17.837 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 3 + 1 2023-11-16T16:51:17.866 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=4 2023-11-16T16:51:17.866 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:17.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:17 smithi155 ceph-mon[131607]: daemon mds.c restarted 2023-11-16T16:51:17.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:17 smithi155 ceph-mon[131607]: osdmap e95: 12 total, 12 up, 12 in 2023-11-16T16:51:17.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:17 smithi002 ceph-mon[128712]: daemon mds.c restarted 2023-11-16T16:51:17.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:17 smithi002 ceph-mon[128712]: osdmap e95: 12 total, 12 up, 12 in 2023-11-16T16:51:17.906 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:17.906 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:17.909 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:17.909 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:17.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:18.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:17 smithi141 ceph-mon[131184]: daemon mds.c restarted 2023-11-16T16:51:18.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:17 smithi141 ceph-mon[131184]: osdmap e95: 12 total, 12 up, 12 in 2023-11-16T16:51:18.309 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:18.309 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:18.454 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:18.454 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:18.469 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:18.469 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:18.469 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 4 -lt 100 ']' 2023-11-16T16:51:18.469 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 4 + 1 2023-11-16T16:51:18.516 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=5 2023-11-16T16:51:18.516 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:18.555 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:18.555 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:18.558 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:18.558 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:18.559 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:18.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:18 smithi155 ceph-mon[131607]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2023-11-16T16:51:18.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:18 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.155:6836/333094585,v1:172.21.15.155:6837/333094585] up:boot 2023-11-16T16:51:18.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:18 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2023-11-16T16:51:18.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:18 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T16:51:18.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:18 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:51:18.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:18 smithi002 ceph-mon[128712]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2023-11-16T16:51:18.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:18 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.155:6836/333094585,v1:172.21.15.155:6837/333094585] up:boot 2023-11-16T16:51:18.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:18 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2023-11-16T16:51:18.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:18 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T16:51:18.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:18 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:51:19.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:18 smithi141 ceph-mon[131184]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2023-11-16T16:51:19.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:18 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.155:6836/333094585,v1:172.21.15.155:6837/333094585] up:boot 2023-11-16T16:51:19.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:18 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2023-11-16T16:51:19.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:18 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T16:51:19.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:18 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:51:19.225 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:19.225 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:19.379 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:19.379 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:19.396 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:19.396 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:19.396 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 5 -lt 100 ']' 2023-11-16T16:51:19.397 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 5 + 1 2023-11-16T16:51:19.454 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=6 2023-11-16T16:51:19.454 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:19.501 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:19.501 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:19.503 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:19.504 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:19.505 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:20.150 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:20.151 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:20.320 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:20.321 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:20.344 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:20.344 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:20.344 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 6 -lt 100 ']' 2023-11-16T16:51:20.344 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 6 + 1 2023-11-16T16:51:20.471 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=7 2023-11-16T16:51:20.471 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:20.535 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.533+0000 7f066dfff700 1 -- 172.21.15.2:0/148893584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0668105560 msgr2=0x7f066810d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.533+0000 7f066dfff700 1 --2- 172.21.15.2:0/148893584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0668105560 0x7f066810d170 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f066000b0d0 tx=0x7f066000b3e0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.534+0000 7f066dfff700 1 -- 172.21.15.2:0/148893584 shutdown_connections 2023-11-16T16:51:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.534+0000 7f066dfff700 1 --2- 172.21.15.2:0/148893584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0668105560 0x7f066810d170 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.534+0000 7f066dfff700 1 --2- 172.21.15.2:0/148893584 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0668100c50 0x7f0668101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.534+0000 7f066dfff700 1 --2- 172.21.15.2:0/148893584 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06681015f0 0x7f0668104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.534+0000 7f066dfff700 1 -- 172.21.15.2:0/148893584 >> 172.21.15.2:0/148893584 conn(0x7f0668077550 msgr2=0x7f0668077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.534+0000 7f066dfff700 1 -- 172.21.15.2:0/148893584 shutdown_connections 2023-11-16T16:51:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.534+0000 7f066dfff700 1 -- 172.21.15.2:0/148893584 wait complete. 2023-11-16T16:51:20.537 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.535+0000 7f066dfff700 1 Processor -- start 2023-11-16T16:51:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.536+0000 7f066dfff700 1 -- start start 2023-11-16T16:51:20.538 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.536+0000 7f066dfff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0668100c50 0x7f0668198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.536+0000 7f066dfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06681015f0 0x7f0668198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.536+0000 7f066dfff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0668105560 0x7f066819c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.536+0000 7f066dfff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066819c610 con 0x7f06681015f0 2023-11-16T16:51:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.536+0000 7f066dfff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066819c780 con 0x7f0668105560 2023-11-16T16:51:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.536+0000 7f066dfff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066819c940 con 0x7f0668100c50 2023-11-16T16:51:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.537+0000 7f06677fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0668100c50 0x7f0668198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.537+0000 7f06677fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0668100c50 0x7f0668198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:44054/0 (socket says 172.21.15.2:44054) 2023-11-16T16:51:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.537+0000 7f0666ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06681015f0 0x7f0668198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.537+0000 7f06677fe700 1 -- 172.21.15.2:0/2099082183 learned_addr learned my addr 172.21.15.2:0/2099082183 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.537+0000 7f0667fff700 1 --2- 172.21.15.2:0/2099082183 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0668105560 0x7f066819c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.540 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.537+0000 7f06677fe700 1 -- 172.21.15.2:0/2099082183 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0668105560 msgr2=0x7f066819c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.537+0000 7f06677fe700 1 --2- 172.21.15.2:0/2099082183 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0668105560 0x7f066819c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.537+0000 7f06677fe700 1 -- 172.21.15.2:0/2099082183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06681015f0 msgr2=0x7f0668198cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.537+0000 7f06677fe700 1 --2- 172.21.15.2:0/2099082183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06681015f0 0x7f0668198cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.537+0000 7f06677fe700 1 -- 172.21.15.2:0/2099082183 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0660009d00 con 0x7f0668100c50 2023-11-16T16:51:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.537+0000 7f06677fe700 1 --2- 172.21.15.2:0/2099082183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0668100c50 0x7f0668198770 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f065800d980 tx=0x7f065800dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:20.541 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.538+0000 7f0664ff9700 1 -- 172.21.15.2:0/2099082183 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0658004220 con 0x7f0668100c50 2023-11-16T16:51:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.538+0000 7f0664ff9700 1 -- 172.21.15.2:0/2099082183 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0658007640 con 0x7f0668100c50 2023-11-16T16:51:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.538+0000 7f066dfff700 1 -- 172.21.15.2:0/2099082183 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06681103d0 con 0x7f0668100c50 2023-11-16T16:51:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.538+0000 7f0664ff9700 1 -- 172.21.15.2:0/2099082183 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0658005690 con 0x7f0668100c50 2023-11-16T16:51:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.538+0000 7f066dfff700 1 -- 172.21.15.2:0/2099082183 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0668110240 con 0x7f0668100c50 2023-11-16T16:51:20.542 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.541+0000 7f0664ff9700 1 -- 172.21.15.2:0/2099082183 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 24) v1 ==== 135947+0+0 (secure 0 0 0) 0x7f0658004750 con 0x7f0668100c50 2023-11-16T16:51:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.542+0000 7f0664ff9700 1 --2- 172.21.15.2:0/2099082183 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f06500a1460 0x7f06500a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.542+0000 7f0664ff9700 1 -- 172.21.15.2:0/2099082183 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f06580cd0f0 con 0x7f0668100c50 2023-11-16T16:51:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.543+0000 7f0666ffd700 1 --2- 172.21.15.2:0/2099082183 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f06500a1460 0x7f06500a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.543+0000 7f0666ffd700 1 --2- 172.21.15.2:0/2099082183 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f06500a1460 0x7f06500a3910 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0668199d30 tx=0x7f065c009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.644+0000 7f0681a03b80 1 Processor -- start 2023-11-16T16:51:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.644+0000 7f0681a03b80 1 -- start start 2023-11-16T16:51:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.644+0000 7f0681a03b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db495e6b0 0x556db4b081e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.644+0000 7f0681a03b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db4b08720 0x556db4b02560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.644+0000 7f0681a03b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db4b08b90 0x556db4b02b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.644+0000 7f0681a03b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db4a08000 con 0x556db495e6b0 2023-11-16T16:51:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.644+0000 7f0681a03b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db4a5a840 con 0x556db4b08720 2023-11-16T16:51:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.644+0000 7f0681a03b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db4a53cd0 con 0x556db4b08b90 2023-11-16T16:51:20.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.644+0000 7f06677fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db495e6b0 0x556db4b081e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.644+0000 7f06677fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db495e6b0 0x556db4b081e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45408/0 (socket says 172.21.15.2:45408) 2023-11-16T16:51:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.644+0000 7f06677fe700 1 -- 172.21.15.2:0/3394476785 learned_addr learned my addr 172.21.15.2:0/3394476785 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.644+0000 7f0666ffd700 1 --2- 172.21.15.2:0/3394476785 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db4b08720 0x556db4b02560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.644+0000 7f0667fff700 1 --2- 172.21.15.2:0/3394476785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db4b08b90 0x556db4b02b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.645+0000 7f06677fe700 1 -- 172.21.15.2:0/3394476785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db4b08b90 msgr2=0x556db4b02b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.645+0000 7f06677fe700 1 --2- 172.21.15.2:0/3394476785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db4b08b90 0x556db4b02b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.645+0000 7f06677fe700 1 -- 172.21.15.2:0/3394476785 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db4b08720 msgr2=0x556db4b02560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.645+0000 7f06677fe700 1 --2- 172.21.15.2:0/3394476785 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db4b08720 0x556db4b02560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.645+0000 7f06677fe700 1 -- 172.21.15.2:0/3394476785 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556db48d55d0 con 0x556db495e6b0 2023-11-16T16:51:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.645+0000 7f06677fe700 1 --2- 172.21.15.2:0/3394476785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db495e6b0 0x556db4b081e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0658007e80 tx=0x7f0658022420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f9b03c5f5bdcb4e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.645+0000 7f064dffb700 1 -- 172.21.15.2:0/3394476785 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065809dbd0 con 0x556db495e6b0 2023-11-16T16:51:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.645+0000 7f064dffb700 1 -- 172.21.15.2:0/3394476785 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f06580996e0 con 0x556db495e6b0 2023-11-16T16:51:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.645+0000 7f064dffb700 1 -- 172.21.15.2:0/3394476785 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06580a1b60 con 0x556db495e6b0 2023-11-16T16:51:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.646+0000 7f0681a03b80 1 -- 172.21.15.2:0/3394476785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db495e6b0 msgr2=0x556db4b081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.646+0000 7f0681a03b80 1 --2- 172.21.15.2:0/3394476785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db495e6b0 0x556db4b081e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0658007e80 tx=0x7f0658022420 comp rx=0 tx=0).stop 2023-11-16T16:51:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.646+0000 7f0681a03b80 1 -- 172.21.15.2:0/3394476785 shutdown_connections 2023-11-16T16:51:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.646+0000 7f0681a03b80 1 --2- 172.21.15.2:0/3394476785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db4b08b90 0x556db4b02b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.646+0000 7f0681a03b80 1 --2- 172.21.15.2:0/3394476785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db495e6b0 0x556db4b081e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.646+0000 7f0681a03b80 1 --2- 172.21.15.2:0/3394476785 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db4b08720 0x556db4b02560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.646+0000 7f0681a03b80 1 -- 172.21.15.2:0/3394476785 >> 172.21.15.2:0/3394476785 conn(0x556db4af7f50 msgr2=0x556db4afd270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.646+0000 7f0681a03b80 1 -- 172.21.15.2:0/3394476785 shutdown_connections 2023-11-16T16:51:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.646+0000 7f0681a03b80 1 -- 172.21.15.2:0/3394476785 wait complete. 2023-11-16T16:51:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.647+0000 7f0681a03b80 1 -- start start 2023-11-16T16:51:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.647+0000 7f0681a03b80 1 Processor -- start 2023-11-16T16:51:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.647+0000 7f0681a03b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db495e6b0 0x556db4b11ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.648+0000 7f0681a03b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db4b08720 0x556db4b11ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.648+0000 7f0681a03b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db4b08b90 0x556db4b0bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.648+0000 7f0681a03b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db49f5880 con 0x556db4b08720 2023-11-16T16:51:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.648+0000 7f0681a03b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db49f5520 con 0x556db495e6b0 2023-11-16T16:51:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.648+0000 7f0681a03b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db48d78c0 con 0x556db4b08b90 2023-11-16T16:51:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.648+0000 7f0681a03b80 10 client.0 _ll_register_callbacks cb 0x556db4a13720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.648+0000 7f0666ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db4b08720 0x556db4b11ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.648+0000 7f0666ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db4b08720 0x556db4b11ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45414/0 (socket says 172.21.15.2:45414) 2023-11-16T16:51:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.648+0000 7f0666ffd700 1 -- 172.21.15.2:0/3846281180 learned_addr learned my addr 172.21.15.2:0/3846281180 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.648+0000 7f06677fe700 1 --2- 172.21.15.2:0/3846281180 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db495e6b0 0x556db4b11ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.648+0000 7f0667fff700 1 --2- 172.21.15.2:0/3846281180 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db4b08b90 0x556db4b0bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.648+0000 7f0666ffd700 1 -- 172.21.15.2:0/3846281180 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db4b08b90 msgr2=0x556db4b0bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.648+0000 7f0666ffd700 1 --2- 172.21.15.2:0/3846281180 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db4b08b90 0x556db4b0bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.648+0000 7f0666ffd700 1 -- 172.21.15.2:0/3846281180 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db495e6b0 msgr2=0x556db4b11ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.648+0000 7f0666ffd700 1 --2- 172.21.15.2:0/3846281180 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db495e6b0 0x556db4b11ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.648+0000 7f0666ffd700 1 -- 172.21.15.2:0/3846281180 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0668110240 con 0x556db4b08720 2023-11-16T16:51:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.649+0000 7f0666ffd700 1 --2- 172.21.15.2:0/3846281180 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db4b08720 0x556db4b11ff0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f06681019c0 tx=0x7f065c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4343d851a94f87a7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:20.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.649+0000 7f0646ffd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:51:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.649+0000 7f0646ffd700 1 -- 172.21.15.2:0/3846281180 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065c004260 con 0x556db4b08720 2023-11-16T16:51:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.649+0000 7f0646ffd700 1 -- 172.21.15.2:0/3846281180 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f065c0043c0 con 0x556db4b08720 2023-11-16T16:51:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.649+0000 7f0681a03b80 1 -- 172.21.15.2:0/3846281180 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556db4af5710 con 0x556db4b08720 2023-11-16T16:51:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.649+0000 7f0646ffd700 1 -- 172.21.15.2:0/3846281180 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f065c004580 con 0x556db4b08720 2023-11-16T16:51:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.649+0000 7f0681a03b80 10 client.15549 fetch_fsmap learned FSMap version 30 2023-11-16T16:51:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.649+0000 7f0681a03b80 1 -- 172.21.15.2:0/3846281180 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x556db4a1a130 con 0x556db4b08720 2023-11-16T16:51:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.649+0000 7f0646ffd700 1 -- 172.21.15.2:0/3846281180 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f065c031020 con 0x556db4b08720 2023-11-16T16:51:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.650+0000 7f0646ffd700 1 -- 172.21.15.2:0/3846281180 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f065c0288e0 con 0x556db4b08720 2023-11-16T16:51:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.650+0000 7f0681a03b80 10 client.15549 fetch_fsmap finished waiting for FSMap version 30 2023-11-16T16:51:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.650+0000 7f0681a03b80 10 client.15549 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.650+0000 7f0681a03b80 20 client.15549 populate_metadata read hostname 'smithi002' 2023-11-16T16:51:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.650+0000 7f0681a03b80 1 --2- 172.21.15.2:0/3846281180 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x556db4b1bfc0 0x556db4b1e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.650+0000 7f0681a03b80 4 client.15549 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:51:20.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.650+0000 7f0681a03b80 1 -- 172.21.15.2:0/3846281180 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556db4a16040 con 0x556db4b1bfc0 2023-11-16T16:51:20.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.650+0000 7f06677fe700 1 --2- 172.21.15.2:0/3846281180 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x556db4b1bfc0 0x556db4b1e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.651+0000 7f06677fe700 1 --2- 172.21.15.2:0/3846281180 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x556db4b1bfc0 0x556db4b1e3d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3f4edc35a7a8acb server_cookie=3a5fe85df3a20459 in_seq=0 out_seq=0 2023-11-16T16:51:20.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.651+0000 7f0646ffd700 10 client.15549 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:51:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.651+0000 7f0646ffd700 1 -- 172.21.15.2:0/3846281180 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x556db4a16040 con 0x556db4b1bfc0 2023-11-16T16:51:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.651+0000 7f0646ffd700 10 client.15549 handle_command_reply: tid=0 2023-11-16T16:51:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.652+0000 7f0681a03b80 1 client.15549 shutdown 2023-11-16T16:51:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.652+0000 7f0681a03b80 1 -- 172.21.15.2:0/3846281180 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db4b08720 msgr2=0x556db4b11ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.652+0000 7f0681a03b80 1 --2- 172.21.15.2:0/3846281180 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db4b08720 0x556db4b11ff0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f06681019c0 tx=0x7f065c0047c0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.652+0000 7f0681a03b80 1 -- 172.21.15.2:0/3846281180 shutdown_connections 2023-11-16T16:51:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.652+0000 7f0681a03b80 1 --2- 172.21.15.2:0/3846281180 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x556db4b1bfc0 0x556db4b1e3d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.652+0000 7f0681a03b80 1 --2- 172.21.15.2:0/3846281180 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db4b08b90 0x556db4b0bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.652+0000 7f0681a03b80 1 --2- 172.21.15.2:0/3846281180 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db495e6b0 0x556db4b11ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.652+0000 7f0646ffd700 0 client.15549 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:51:20.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.652+0000 7f0681a03b80 1 --2- 172.21.15.2:0/3846281180 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db4b08720 0x556db4b11ff0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.652+0000 7f0681a03b80 1 -- 172.21.15.2:0/3846281180 >> 172.21.15.2:0/3846281180 conn(0x556db4af7f50 msgr2=0x556db4afc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:20.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.652+0000 7f0681a03b80 1 -- 172.21.15.2:0/3846281180 shutdown_connections 2023-11-16T16:51:20.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.652+0000 7f0681a03b80 1 -- 172.21.15.2:0/3846281180 wait complete. 2023-11-16T16:51:20.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.652+0000 7f0681a03b80 20 client.15549 trim_cache size 0 max 16384 2023-11-16T16:51:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.669+0000 7f0681a03b80 1 Processor -- start 2023-11-16T16:51:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.670+0000 7f0681a03b80 1 -- start start 2023-11-16T16:51:20.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.670+0000 7f0681a03b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db495e6b0 0x556db4afcf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.670+0000 7f0681a03b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db4b08720 0x556db4b0b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.670+0000 7f0681a03b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db4b08b90 0x556db4b0bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.670+0000 7f0681a03b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db4af7b70 con 0x556db495e6b0 2023-11-16T16:51:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.670+0000 7f0681a03b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db4b069d0 con 0x556db4b08b90 2023-11-16T16:51:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.670+0000 7f0681a03b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db4b18890 con 0x556db4b08720 2023-11-16T16:51:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.670+0000 7f06677fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db495e6b0 0x556db4afcf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.670+0000 7f0666ffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db4b08720 0x556db4b0b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.670+0000 7f06677fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db495e6b0 0x556db4afcf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45422/0 (socket says 172.21.15.2:45422) 2023-11-16T16:51:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.670+0000 7f0666ffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db4b08720 0x556db4b0b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:44088/0 (socket says 172.21.15.2:44088) 2023-11-16T16:51:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.670+0000 7f06677fe700 1 -- 172.21.15.2:0/2098854612 learned_addr learned my addr 172.21.15.2:0/2098854612 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.670+0000 7f0667fff700 1 --2- 172.21.15.2:0/2098854612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db4b08b90 0x556db4b0bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.671+0000 7f06677fe700 1 -- 172.21.15.2:0/2098854612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db4b08720 msgr2=0x556db4b0b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.671+0000 7f06677fe700 1 --2- 172.21.15.2:0/2098854612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db4b08720 0x556db4b0b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.671+0000 7f06677fe700 1 -- 172.21.15.2:0/2098854612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db4b08b90 msgr2=0x556db4b0bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.671+0000 7f06677fe700 1 --2- 172.21.15.2:0/2098854612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db4b08b90 0x556db4b0bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.671+0000 7f06677fe700 1 -- 172.21.15.2:0/2098854612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556db4af96f0 con 0x556db495e6b0 2023-11-16T16:51:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.671+0000 7f06677fe700 1 --2- 172.21.15.2:0/2098854612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db495e6b0 0x556db4afcf60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0658097ef0 tx=0x7f0658099970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c00016e98bd9c0e server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.671+0000 7f064e7fc700 1 -- 172.21.15.2:0/2098854612 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065809dbd0 con 0x556db495e6b0 2023-11-16T16:51:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.671+0000 7f064e7fc700 1 -- 172.21.15.2:0/2098854612 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f06580c9070 con 0x556db495e6b0 2023-11-16T16:51:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.671+0000 7f064e7fc700 1 -- 172.21.15.2:0/2098854612 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06580992f0 con 0x556db495e6b0 2023-11-16T16:51:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.672+0000 7f0681a03b80 1 -- 172.21.15.2:0/2098854612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db495e6b0 msgr2=0x556db4afcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.672+0000 7f0681a03b80 1 --2- 172.21.15.2:0/2098854612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db495e6b0 0x556db4afcf60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0658097ef0 tx=0x7f0658099970 comp rx=0 tx=0).stop 2023-11-16T16:51:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.672+0000 7f0681a03b80 1 -- 172.21.15.2:0/2098854612 shutdown_connections 2023-11-16T16:51:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.672+0000 7f0681a03b80 1 --2- 172.21.15.2:0/2098854612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db4b08b90 0x556db4b0bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.672+0000 7f0681a03b80 1 --2- 172.21.15.2:0/2098854612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db495e6b0 0x556db4afcf60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.672+0000 7f0681a03b80 1 --2- 172.21.15.2:0/2098854612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db4b08720 0x556db4b0b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.672+0000 7f0681a03b80 1 -- 172.21.15.2:0/2098854612 >> 172.21.15.2:0/2098854612 conn(0x556db4af7f50 msgr2=0x556db4b0c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.672+0000 7f0681a03b80 1 -- 172.21.15.2:0/2098854612 shutdown_connections 2023-11-16T16:51:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.672+0000 7f0681a03b80 1 -- 172.21.15.2:0/2098854612 wait complete. 2023-11-16T16:51:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.672+0000 7f0681a03b80 1 -- start start 2023-11-16T16:51:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.673+0000 7f0681a03b80 1 Processor -- start 2023-11-16T16:51:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.673+0000 7f0681a03b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db495e6b0 0x556db4b0f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.673+0000 7f0681a03b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db4b08720 0x556db4b0fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.673+0000 7f0681a03b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db4b08b90 0x556db4b100a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.673+0000 7f0681a03b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db4b6ee80 con 0x556db4b08b90 2023-11-16T16:51:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.673+0000 7f0681a03b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db4b6efc0 con 0x556db4b08720 2023-11-16T16:51:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.673+0000 7f0681a03b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db4b6f100 con 0x556db495e6b0 2023-11-16T16:51:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.673+0000 7f0681a03b80 10 client.0 _ll_register_callbacks cb 0x7f065c01f4c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.674+0000 7f0667fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db4b08b90 0x556db4b100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.674+0000 7f0667fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db4b08b90 0x556db4b100a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45434/0 (socket says 172.21.15.2:45434) 2023-11-16T16:51:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.674+0000 7f0667fff700 1 -- 172.21.15.2:0/1083600291 learned_addr learned my addr 172.21.15.2:0/1083600291 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.674+0000 7f0666ffd700 1 --2- 172.21.15.2:0/1083600291 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db4b08720 0x556db4b0fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.674+0000 7f06677fe700 1 --2- 172.21.15.2:0/1083600291 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db495e6b0 0x556db4b0f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.674+0000 7f0667fff700 1 -- 172.21.15.2:0/1083600291 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db495e6b0 msgr2=0x556db4b0f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.674+0000 7f0667fff700 1 --2- 172.21.15.2:0/1083600291 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db495e6b0 0x556db4b0f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.674+0000 7f0667fff700 1 -- 172.21.15.2:0/1083600291 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db4b08720 msgr2=0x556db4b0fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.674+0000 7f0667fff700 1 --2- 172.21.15.2:0/1083600291 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db4b08720 0x556db4b0fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.674+0000 7f0667fff700 1 -- 172.21.15.2:0/1083600291 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556db48d55d0 con 0x556db4b08b90 2023-11-16T16:51:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.674+0000 7f0667fff700 1 --2- 172.21.15.2:0/1083600291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db4b08b90 0x556db4b100a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f066000bcd0 tx=0x7f066000bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac2cfc30ca28d1f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.674+0000 7f0646ffd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:51:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.675+0000 7f0646ffd700 1 -- 172.21.15.2:0/1083600291 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066000e040 con 0x556db4b08b90 2023-11-16T16:51:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.675+0000 7f0681a03b80 1 -- 172.21.15.2:0/1083600291 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f06580996e0 con 0x556db4b08b90 2023-11-16T16:51:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.675+0000 7f0646ffd700 1 -- 172.21.15.2:0/1083600291 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f066000be20 con 0x556db4b08b90 2023-11-16T16:51:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.675+0000 7f0646ffd700 1 -- 172.21.15.2:0/1083600291 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0660009ba0 con 0x556db4b08b90 2023-11-16T16:51:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.675+0000 7f0681a03b80 10 client.15555 fetch_fsmap learned FSMap version 30 2023-11-16T16:51:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.675+0000 7f0681a03b80 1 -- 172.21.15.2:0/1083600291 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x556db4881d80 con 0x556db4b08b90 2023-11-16T16:51:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.675+0000 7f0646ffd700 1 -- 172.21.15.2:0/1083600291 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f06580996e0 con 0x556db4b08b90 2023-11-16T16:51:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.675+0000 7f0646ffd700 1 -- 172.21.15.2:0/1083600291 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f066000b100 con 0x556db4b08b90 2023-11-16T16:51:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.675+0000 7f0681a03b80 10 client.15555 fetch_fsmap finished waiting for FSMap version 30 2023-11-16T16:51:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.675+0000 7f0681a03b80 10 client.15555 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.675+0000 7f0681a03b80 20 client.15555 populate_metadata read hostname 'smithi002' 2023-11-16T16:51:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.675+0000 7f0681a03b80 1 --2- 172.21.15.2:0/1083600291 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x556db4b707b0 0x556db4b72bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.675+0000 7f0681a03b80 4 client.15555 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:51:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.676+0000 7f0681a03b80 1 -- 172.21.15.2:0/1083600291 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x556db4a22600 con 0x556db4b707b0 2023-11-16T16:51:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.676+0000 7f06677fe700 1 --2- 172.21.15.2:0/1083600291 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x556db4b707b0 0x556db4b72bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.677+0000 7f06677fe700 1 --2- 172.21.15.2:0/1083600291 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x556db4b707b0 0x556db4b72bc0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4698074533abb7f4 server_cookie=93707ebc2f61391f in_seq=0 out_seq=0 2023-11-16T16:51:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.677+0000 7f0646ffd700 10 client.15555 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:51:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.681+0000 7f0646ffd700 1 -- 172.21.15.2:0/1083600291 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x556db4a22600 con 0x556db4b707b0 2023-11-16T16:51:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.681+0000 7f0646ffd700 10 client.15555 handle_command_reply: tid=0 2023-11-16T16:51:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.681+0000 7f0681a03b80 1 client.15555 shutdown 2023-11-16T16:51:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.681+0000 7f0681a03b80 1 -- 172.21.15.2:0/1083600291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db4b08b90 msgr2=0x556db4b100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.681+0000 7f0681a03b80 1 --2- 172.21.15.2:0/1083600291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db4b08b90 0x556db4b100a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f066000bcd0 tx=0x7f066000bd00 comp rx=0 tx=0).stop 2023-11-16T16:51:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.681+0000 7f0681a03b80 1 -- 172.21.15.2:0/1083600291 shutdown_connections 2023-11-16T16:51:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.681+0000 7f0681a03b80 1 --2- 172.21.15.2:0/1083600291 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x556db4b707b0 0x556db4b72bc0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.681+0000 7f0681a03b80 1 --2- 172.21.15.2:0/1083600291 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556db495e6b0 0x556db4b0f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.681+0000 7f0681a03b80 1 --2- 172.21.15.2:0/1083600291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556db4b08b90 0x556db4b100a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.681+0000 7f0646ffd700 0 client.15555 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:51:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.681+0000 7f0681a03b80 1 --2- 172.21.15.2:0/1083600291 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x556db4b08720 0x556db4b0fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.681+0000 7f0681a03b80 1 -- 172.21.15.2:0/1083600291 >> 172.21.15.2:0/1083600291 conn(0x556db4af7f50 msgr2=0x556db4b0b570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.682+0000 7f0681a03b80 1 -- 172.21.15.2:0/1083600291 shutdown_connections 2023-11-16T16:51:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.682+0000 7f0681a03b80 1 -- 172.21.15.2:0/1083600291 wait complete. 2023-11-16T16:51:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.682+0000 7f0681a03b80 20 client.15555 trim_cache size 0 max 16384 2023-11-16T16:51:20.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:51:20.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:51:20.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:51:20.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:51:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.682+0000 7f066dfff700 1 -- 172.21.15.2:0/2099082183 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f06500a1460 msgr2=0x7f06500a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.682+0000 7f066dfff700 1 --2- 172.21.15.2:0/2099082183 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f06500a1460 0x7f06500a3910 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0668199d30 tx=0x7f065c009260 comp rx=0 tx=0).stop 2023-11-16T16:51:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.682+0000 7f066dfff700 1 -- 172.21.15.2:0/2099082183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0668100c50 msgr2=0x7f0668198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.682+0000 7f066dfff700 1 --2- 172.21.15.2:0/2099082183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0668100c50 0x7f0668198770 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f065800d980 tx=0x7f065800dc90 comp rx=0 tx=0).stop 2023-11-16T16:51:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.682+0000 7f066dfff700 1 -- 172.21.15.2:0/2099082183 shutdown_connections 2023-11-16T16:51:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.682+0000 7f066dfff700 1 --2- 172.21.15.2:0/2099082183 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f06500a1460 0x7f06500a3910 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.682+0000 7f066dfff700 1 --2- 172.21.15.2:0/2099082183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0668100c50 0x7f0668198770 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.682+0000 7f066dfff700 1 --2- 172.21.15.2:0/2099082183 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0668105560 0x7f066819c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.682+0000 7f066dfff700 1 --2- 172.21.15.2:0/2099082183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06681015f0 0x7f0668198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.682+0000 7f066dfff700 1 -- 172.21.15.2:0/2099082183 >> 172.21.15.2:0/2099082183 conn(0x7f0668077550 msgr2=0x7f06680fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.683+0000 7f066dfff700 1 -- 172.21.15.2:0/2099082183 shutdown_connections 2023-11-16T16:51:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:20.683+0000 7f066dfff700 1 -- 172.21.15.2:0/2099082183 wait complete. 2023-11-16T16:51:20.843 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:20 smithi002 ceph-mon[128712]: pgmap v46: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2023-11-16T16:51:20.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:20 smithi155 ceph-mon[131607]: pgmap v46: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2023-11-16T16:51:21.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:20 smithi141 ceph-mon[131184]: pgmap v46: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2023-11-16T16:51:21.125 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:21.125 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:21.261 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:21.262 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:21.277 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:21.277 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:21.277 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 7 -lt 100 ']' 2023-11-16T16:51:21.277 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 7 + 1 2023-11-16T16:51:21.324 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=8 2023-11-16T16:51:21.324 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:21.360 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:21.360 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:21.363 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:21.363 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:21.363 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:21.758 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:21.758 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:21.845 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:21.845 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:21.858 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:21.858 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:21.859 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 8 -lt 100 ']' 2023-11-16T16:51:21.859 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 8 + 1 2023-11-16T16:51:21.866 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=9 2023-11-16T16:51:21.866 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:21.875 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:21.876 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:21.878 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:21.879 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:21.879 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:22.043 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:22.043 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:22.112 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:22.112 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:22.134 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:22.134 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:22.135 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 9 -lt 100 ']' 2023-11-16T16:51:22.135 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 9 + 1 2023-11-16T16:51:22.165 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=10 2023-11-16T16:51:22.165 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:22.190 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:22.190 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:22.192 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:22.193 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:22.193 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:22.550 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:22.550 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:22.653 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:22.653 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:22.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:22 smithi002 ceph-mon[128712]: pgmap v47: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2023-11-16T16:51:22.668 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:22.668 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:22.668 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 10 -lt 100 ']' 2023-11-16T16:51:22.669 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 10 + 1 2023-11-16T16:51:22.698 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=11 2023-11-16T16:51:22.698 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:22.742 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:22.742 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:22.745 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:22.745 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:22.746 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:22.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:22 smithi155 ceph-mon[131607]: pgmap v47: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2023-11-16T16:51:23.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:22 smithi141 ceph-mon[131184]: pgmap v47: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2023-11-16T16:51:23.099 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:23.100 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:23.286 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:23.286 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:23.302 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:23.302 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:23.302 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 11 -lt 100 ']' 2023-11-16T16:51:23.302 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 11 + 1 2023-11-16T16:51:23.332 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=12 2023-11-16T16:51:23.332 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:23.357 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:23.357 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:23.360 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:23.360 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:23.361 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:23.707 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:23.708 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:23.786 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:23.786 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:23.801 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:23.801 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:23.801 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 12 -lt 100 ']' 2023-11-16T16:51:23.801 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 12 + 1 2023-11-16T16:51:23.832 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=13 2023-11-16T16:51:23.832 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:23.865 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:23.866 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:23.868 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:23.868 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:23.869 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:24.216 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:24.216 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:24.328 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:24.328 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:24.343 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:24.343 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:24.343 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 13 -lt 100 ']' 2023-11-16T16:51:24.344 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 13 + 1 2023-11-16T16:51:24.373 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=14 2023-11-16T16:51:24.373 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:24.399 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:24.399 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:24.401 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:24.402 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:24.402 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:24.783 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:24.783 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:24.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:24 smithi155 ceph-mon[131607]: pgmap v48: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2023-11-16T16:51:24.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:24 smithi002 ceph-mon[128712]: pgmap v48: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2023-11-16T16:51:24.920 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:24.920 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:24.934 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:24.934 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:24.934 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 14 -lt 100 ']' 2023-11-16T16:51:24.935 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 14 + 1 2023-11-16T16:51:24.970 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=15 2023-11-16T16:51:24.970 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:25.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:24 smithi141 ceph-mon[131184]: pgmap v48: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2023-11-16T16:51:25.032 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:25.033 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:25.035 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:25.036 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:25.036 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:25.466 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:25.466 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:25.577 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:25.577 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:25.601 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:25.601 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:25.602 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 15 -lt 100 ']' 2023-11-16T16:51:25.602 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 15 + 1 2023-11-16T16:51:25.631 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=16 2023-11-16T16:51:25.631 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:25.667 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:25.668 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:25.670 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:25.671 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:25.671 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:26.058 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:26.058 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:26.169 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:26.169 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:26.184 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:26.184 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:26.184 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 16 -lt 100 ']' 2023-11-16T16:51:26.184 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 16 + 1 2023-11-16T16:51:26.215 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=17 2023-11-16T16:51:26.215 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:26.241 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:26.241 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:26.244 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:26.245 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:26.245 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:26.608 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:26.608 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:26.719 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:26.719 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:26.734 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:26.735 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:26.735 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 17 -lt 100 ']' 2023-11-16T16:51:26.735 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 17 + 1 2023-11-16T16:51:26.765 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=18 2023-11-16T16:51:26.765 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:26.817 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:26.817 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:26.820 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:26.820 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:26.821 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:26.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:26 smithi155 ceph-mon[131607]: pgmap v49: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2023-11-16T16:51:26.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:26 smithi155 ceph-mon[131607]: mgrmap e25: x(active, since 90s), standbys: z, y 2023-11-16T16:51:26.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:26 smithi002 ceph-mon[128712]: pgmap v49: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2023-11-16T16:51:26.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:26 smithi002 ceph-mon[128712]: mgrmap e25: x(active, since 90s), standbys: z, y 2023-11-16T16:51:27.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:26 smithi141 ceph-mon[131184]: pgmap v49: 129 pgs: 129 active+clean; 1.3 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2023-11-16T16:51:27.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:26 smithi141 ceph-mon[131184]: mgrmap e25: x(active, since 90s), standbys: z, y 2023-11-16T16:51:27.199 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:27.199 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:27.327 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:27.327 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:27.351 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:27.351 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:27.352 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 18 -lt 100 ']' 2023-11-16T16:51:27.352 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 18 + 1 2023-11-16T16:51:27.395 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=19 2023-11-16T16:51:27.395 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:27.492 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:27.492 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:27.494 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:27.495 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:27.495 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:27.840 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:27.841 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:27.952 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:27.952 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:27.968 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:27.968 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:27.968 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 19 -lt 100 ']' 2023-11-16T16:51:27.969 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 19 + 1 2023-11-16T16:51:27.997 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=20 2023-11-16T16:51:27.998 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:28.025 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:28.025 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:28.028 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:28.028 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:28.029 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:28.382 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:28.382 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:28.481 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:28.481 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:28.492 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:28.492 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:28.492 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 20 -lt 100 ']' 2023-11-16T16:51:28.492 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 20 + 1 2023-11-16T16:51:28.539 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=21 2023-11-16T16:51:28.539 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:28.567 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:28.567 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:28.570 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:28.570 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:28.571 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:28.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:28 smithi155 ceph-mon[131607]: pgmap v50: 129 pgs: 129 active+clean; 1.5 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 10 op/s 2023-11-16T16:51:28.875 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:28 smithi002 ceph-mon[128712]: pgmap v50: 129 pgs: 129 active+clean; 1.5 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 10 op/s 2023-11-16T16:51:28.932 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:28.932 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:29.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:28 smithi141 ceph-mon[131184]: pgmap v50: 129 pgs: 129 active+clean; 1.5 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 10 op/s 2023-11-16T16:51:29.052 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:29.052 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:29.069 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:29.069 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:29.069 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 21 -lt 100 ']' 2023-11-16T16:51:29.069 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 21 + 1 2023-11-16T16:51:29.097 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=22 2023-11-16T16:51:29.098 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:29.125 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:29.125 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:29.127 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:29.128 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:29.128 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:29.574 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:29.574 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:29.710 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:29.710 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:29.726 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:29.726 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:29.726 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 22 -lt 100 ']' 2023-11-16T16:51:29.727 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 22 + 1 2023-11-16T16:51:29.756 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=23 2023-11-16T16:51:29.756 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:29.784 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:29.784 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:29.786 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:29.787 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:29.788 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:30.165 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:30.166 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:30.318 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:30.318 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:30.334 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:30.335 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:30.335 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 23 -lt 100 ']' 2023-11-16T16:51:30.335 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 23 + 1 2023-11-16T16:51:30.364 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=24 2023-11-16T16:51:30.364 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:30.392 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:30.392 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:30.395 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:30.395 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:30.395 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:30.740 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:30.740 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:30.860 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:30.860 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:30.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:30 smithi155 ceph-mon[131607]: pgmap v51: 129 pgs: 129 active+clean; 1.5 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 9 op/s 2023-11-16T16:51:30.875 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:30.875 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:30.875 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 24 -lt 100 ']' 2023-11-16T16:51:30.876 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 24 + 1 2023-11-16T16:51:30.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:30 smithi002 ceph-mon[128712]: pgmap v51: 129 pgs: 129 active+clean; 1.5 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 9 op/s 2023-11-16T16:51:30.906 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=25 2023-11-16T16:51:30.906 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:30.934 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:30.934 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:30.936 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:30.937 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:30.937 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:31.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:30 smithi141 ceph-mon[131184]: pgmap v51: 129 pgs: 129 active+clean; 1.5 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 9 op/s 2023-11-16T16:51:31.324 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:31.324 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:31.443 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:31.443 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:31.459 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:31.460 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:31.460 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 25 -lt 100 ']' 2023-11-16T16:51:31.460 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 25 + 1 2023-11-16T16:51:31.489 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=26 2023-11-16T16:51:31.489 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:31.516 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:31.517 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:31.519 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:31.520 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:31.520 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:31.965 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:31.965 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:32.093 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:32.093 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:32.108 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:32.108 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:32.108 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 26 -lt 100 ']' 2023-11-16T16:51:32.109 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 26 + 1 2023-11-16T16:51:32.138 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=27 2023-11-16T16:51:32.139 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:32.166 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:32.166 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:32.169 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:32.170 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:32.170 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:32.590 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:32.590 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:32.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:32 smithi002 ceph-mon[128712]: pgmap v52: 129 pgs: 129 active+clean; 1.5 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 12 op/s 2023-11-16T16:51:32.685 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:32.685 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:32.700 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:32.701 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:32.701 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 27 -lt 100 ']' 2023-11-16T16:51:32.701 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 27 + 1 2023-11-16T16:51:32.730 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=28 2023-11-16T16:51:32.730 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:32.775 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:32.775 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:32.778 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:32.778 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:32.779 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:32.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:32 smithi155 ceph-mon[131607]: pgmap v52: 129 pgs: 129 active+clean; 1.5 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 12 op/s 2023-11-16T16:51:33.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:32 smithi141 ceph-mon[131184]: pgmap v52: 129 pgs: 129 active+clean; 1.5 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 12 op/s 2023-11-16T16:51:33.165 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:33.166 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:33.285 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:33.285 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:33.300 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:33.300 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:33.300 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 28 -lt 100 ']' 2023-11-16T16:51:33.301 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 28 + 1 2023-11-16T16:51:33.330 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=29 2023-11-16T16:51:33.330 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:33.375 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:33.375 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:33.377 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:33.378 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:33.378 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:33.823 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:33.823 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:33.985 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:33.985 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:34.008 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:34.008 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:34.008 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 29 -lt 100 ']' 2023-11-16T16:51:34.009 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 29 + 1 2023-11-16T16:51:34.044 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=30 2023-11-16T16:51:34.044 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:34.085 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:34.086 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:34.088 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:34.089 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:34.089 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:34.715 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:34.715 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:34.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:34 smithi155 ceph-mon[131607]: pgmap v53: 129 pgs: 129 active+clean; 1.5 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-11-16T16:51:34.893 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:34.893 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:34.900 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:34.900 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:34.900 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 30 -lt 100 ']' 2023-11-16T16:51:34.900 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 30 + 1 2023-11-16T16:51:34.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:34 smithi002 ceph-mon[128712]: pgmap v53: 129 pgs: 129 active+clean; 1.5 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-11-16T16:51:34.946 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=31 2023-11-16T16:51:34.947 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:35.000 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:35.000 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:35.003 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:35.003 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:35.004 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:35.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:34 smithi141 ceph-mon[131184]: pgmap v53: 129 pgs: 129 active+clean; 1.5 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-11-16T16:51:35.690 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:35.690 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:35.834 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:35.834 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:35.866 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:35.867 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:35.867 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 31 -lt 100 ']' 2023-11-16T16:51:35.867 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 31 + 1 2023-11-16T16:51:35.952 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=32 2023-11-16T16:51:35.952 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.107+0000 7f1521bff700 1 -- 172.21.15.2:0/2765529219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f151c105530 msgr2=0x7f151c10d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.107+0000 7f1521bff700 1 --2- 172.21.15.2:0/2765529219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f151c105530 0x7f151c10d100 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f151400b0d0 tx=0x7f151400b3e0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.107+0000 7f1521bff700 1 -- 172.21.15.2:0/2765529219 shutdown_connections 2023-11-16T16:51:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.107+0000 7f1521bff700 1 --2- 172.21.15.2:0/2765529219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f151c100540 0x7f151c100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.107+0000 7f1521bff700 1 --2- 172.21.15.2:0/2765529219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f151c105530 0x7f151c10d100 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.108 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.107+0000 7f1521bff700 1 --2- 172.21.15.2:0/2765529219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f151c100ee0 0x7f151c104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.109 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.107+0000 7f1521bff700 1 -- 172.21.15.2:0/2765529219 >> 172.21.15.2:0/2765529219 conn(0x7f151c078210 msgr2=0x7f151c078610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:36.109 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.107+0000 7f1521bff700 1 -- 172.21.15.2:0/2765529219 shutdown_connections 2023-11-16T16:51:36.109 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.107+0000 7f1521bff700 1 -- 172.21.15.2:0/2765529219 wait complete. 2023-11-16T16:51:36.110 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.108+0000 7f1521bff700 1 Processor -- start 2023-11-16T16:51:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.109+0000 7f1521bff700 1 -- start start 2023-11-16T16:51:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.109+0000 7f1521bff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f151c100540 0x7f151c072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.109+0000 7f1521bff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f151c100ee0 0x7f151c073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.109+0000 7f1521bff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f151c105530 0x7f151c1a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.109+0000 7f1521bff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c073bd0 con 0x7f151c105530 2023-11-16T16:51:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.109+0000 7f1521bff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c1a4c10 con 0x7f151c100540 2023-11-16T16:51:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.109+0000 7f1521bff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c1a4d50 con 0x7f151c100ee0 2023-11-16T16:51:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.109+0000 7f151bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f151c105530 0x7f151c1a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.109+0000 7f151bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f151c105530 0x7f151c1a46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33312/0 (socket says 172.21.15.2:33312) 2023-11-16T16:51:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.109+0000 7f151bfff700 1 -- 172.21.15.2:0/61507530 learned_addr learned my addr 172.21.15.2:0/61507530 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.109+0000 7f151affd700 1 --2- 172.21.15.2:0/61507530 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f151c100ee0 0x7f151c073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.110+0000 7f151b7fe700 1 --2- 172.21.15.2:0/61507530 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f151c100540 0x7f151c072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:36.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.110+0000 7f151bfff700 1 -- 172.21.15.2:0/61507530 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f151c100ee0 msgr2=0x7f151c073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:36.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.110+0000 7f151bfff700 1 --2- 172.21.15.2:0/61507530 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f151c100ee0 0x7f151c073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.110+0000 7f151bfff700 1 -- 172.21.15.2:0/61507530 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f151c100540 msgr2=0x7f151c072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:36.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.110+0000 7f151bfff700 1 --2- 172.21.15.2:0/61507530 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f151c100540 0x7f151c072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.110+0000 7f151bfff700 1 -- 172.21.15.2:0/61507530 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1514009d00 con 0x7f151c105530 2023-11-16T16:51:36.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.110+0000 7f151bfff700 1 --2- 172.21.15.2:0/61507530 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f151c105530 0x7f151c1a46d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1514007850 tx=0x7f1514007880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:36.113 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.110+0000 7f1518ff9700 1 -- 172.21.15.2:0/61507530 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151400e040 con 0x7f151c105530 2023-11-16T16:51:36.113 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.110+0000 7f1518ff9700 1 -- 172.21.15.2:0/61507530 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f151400bcb0 con 0x7f151c105530 2023-11-16T16:51:36.113 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.110+0000 7f1518ff9700 1 -- 172.21.15.2:0/61507530 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514005890 con 0x7f151c105530 2023-11-16T16:51:36.113 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.110+0000 7f1521bff700 1 -- 172.21.15.2:0/61507530 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f151c1103a0 con 0x7f151c105530 2023-11-16T16:51:36.113 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.111+0000 7f1521bff700 1 -- 172.21.15.2:0/61507530 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f151c110210 con 0x7f151c105530 2023-11-16T16:51:36.114 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.114+0000 7f1518ff9700 1 -- 172.21.15.2:0/61507530 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f1514004a50 con 0x7f151c105530 2023-11-16T16:51:36.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.115+0000 7f1518ff9700 1 --2- 172.21.15.2:0/61507530 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f15040a1340 0x7f15040a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:36.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.115+0000 7f1518ff9700 1 -- 172.21.15.2:0/61507530 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f15140ce130 con 0x7f151c105530 2023-11-16T16:51:36.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.116+0000 7f151b7fe700 1 --2- 172.21.15.2:0/61507530 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f15040a1340 0x7f15040a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.116+0000 7f151b7fe700 1 --2- 172.21.15.2:0/61507530 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f15040a1340 0x7f15040a37f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f150c009c60 tx=0x7f150c009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:36.140 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:36.141 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:36.143 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:36.144 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:36.144 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.208+0000 7f1535460b80 1 Processor -- start 2023-11-16T16:51:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.209+0000 7f1535460b80 1 -- start start 2023-11-16T16:51:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.209+0000 7f1535460b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b933626b0 0x557b9350c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.209+0000 7f1535460b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350c720 0x557b93506560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.209+0000 7f1535460b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b9350cb90 0x557b93506b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.209+0000 7f1535460b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b9340c000 con 0x557b9350c720 2023-11-16T16:51:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.209+0000 7f1535460b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b9345e840 con 0x557b9350cb90 2023-11-16T16:51:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.209+0000 7f1535460b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b93457cd0 con 0x557b933626b0 2023-11-16T16:51:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.209+0000 7f151affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350c720 0x557b93506560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.209+0000 7f151affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350c720 0x557b93506560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33316/0 (socket says 172.21.15.2:33316) 2023-11-16T16:51:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.209+0000 7f151affd700 1 -- 172.21.15.2:0/2254161832 learned_addr learned my addr 172.21.15.2:0/2254161832 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.210+0000 7f151b7fe700 1 --2- 172.21.15.2:0/2254161832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b933626b0 0x557b9350c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.210+0000 7f151bfff700 1 --2- 172.21.15.2:0/2254161832 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b9350cb90 0x557b93506b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.210+0000 7f151affd700 1 -- 172.21.15.2:0/2254161832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b933626b0 msgr2=0x557b9350c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.210+0000 7f151affd700 1 --2- 172.21.15.2:0/2254161832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b933626b0 0x557b9350c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.210+0000 7f151affd700 1 -- 172.21.15.2:0/2254161832 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b9350cb90 msgr2=0x557b93506b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.210+0000 7f151affd700 1 --2- 172.21.15.2:0/2254161832 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b9350cb90 0x557b93506b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.210+0000 7f151affd700 1 -- 172.21.15.2:0/2254161832 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b932d95d0 con 0x557b9350c720 2023-11-16T16:51:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.210+0000 7f151affd700 1 --2- 172.21.15.2:0/2254161832 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350c720 0x557b93506560 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1510009c90 tx=0x7f15100058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56977fb80e27729b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.210+0000 7f1501ffb700 1 -- 172.21.15.2:0/2254161832 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1510011c80 con 0x557b9350c720 2023-11-16T16:51:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.210+0000 7f1501ffb700 1 -- 172.21.15.2:0/2254161832 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1510004510 con 0x557b9350c720 2023-11-16T16:51:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.210+0000 7f1501ffb700 1 -- 172.21.15.2:0/2254161832 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1510005280 con 0x557b9350c720 2023-11-16T16:51:36.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.211+0000 7f1535460b80 1 -- 172.21.15.2:0/2254161832 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350c720 msgr2=0x557b93506560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:36.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.211+0000 7f1535460b80 1 --2- 172.21.15.2:0/2254161832 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350c720 0x557b93506560 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1510009c90 tx=0x7f15100058e0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.211+0000 7f1535460b80 1 -- 172.21.15.2:0/2254161832 shutdown_connections 2023-11-16T16:51:36.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.211+0000 7f1535460b80 1 --2- 172.21.15.2:0/2254161832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b933626b0 0x557b9350c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.211+0000 7f1535460b80 1 --2- 172.21.15.2:0/2254161832 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b9350cb90 0x557b93506b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.211+0000 7f1535460b80 1 --2- 172.21.15.2:0/2254161832 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350c720 0x557b93506560 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.211+0000 7f1535460b80 1 -- 172.21.15.2:0/2254161832 >> 172.21.15.2:0/2254161832 conn(0x557b934fbf50 msgr2=0x557b93501270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:36.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.211+0000 7f1535460b80 1 -- 172.21.15.2:0/2254161832 shutdown_connections 2023-11-16T16:51:36.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.211+0000 7f1535460b80 1 -- 172.21.15.2:0/2254161832 wait complete. 2023-11-16T16:51:36.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.212+0000 7f1535460b80 1 -- start start 2023-11-16T16:51:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.212+0000 7f1535460b80 1 Processor -- start 2023-11-16T16:51:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.212+0000 7f1535460b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b933626b0 0x557b93512a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.212+0000 7f1535460b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b9350c720 0x557b9350fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.213+0000 7f1535460b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350cb90 0x557b935102c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.213+0000 7f1535460b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b933f9880 con 0x557b9350cb90 2023-11-16T16:51:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.213+0000 7f1535460b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b933f9520 con 0x557b933626b0 2023-11-16T16:51:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.213+0000 7f1535460b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b932db8c0 con 0x557b9350c720 2023-11-16T16:51:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.213+0000 7f1535460b80 10 client.0 _ll_register_callbacks cb 0x557b93417720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.213+0000 7f151affd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b9350c720 0x557b9350fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.213+0000 7f151bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350cb90 0x557b935102c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.213+0000 7f151b7fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b933626b0 0x557b93512a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.213+0000 7f151affd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b9350c720 0x557b9350fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:58086/0 (socket says 172.21.15.2:58086) 2023-11-16T16:51:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.213+0000 7f151affd700 1 -- 172.21.15.2:0/2846887017 learned_addr learned my addr 172.21.15.2:0/2846887017 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.213+0000 7f151bfff700 1 -- 172.21.15.2:0/2846887017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b9350c720 msgr2=0x557b9350fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.213+0000 7f151bfff700 1 --2- 172.21.15.2:0/2846887017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b9350c720 0x557b9350fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.213+0000 7f151bfff700 1 -- 172.21.15.2:0/2846887017 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b933626b0 msgr2=0x557b93512a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.213+0000 7f151bfff700 1 --2- 172.21.15.2:0/2846887017 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b933626b0 0x557b93512a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.213+0000 7f151bfff700 1 -- 172.21.15.2:0/2846887017 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15100098c0 con 0x557b9350cb90 2023-11-16T16:51:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.214+0000 7f151bfff700 1 --2- 172.21.15.2:0/2846887017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350cb90 0x557b935102c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f15140cb4b0 tx=0x7f151409aaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562d323189c135cd server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.214+0000 7f14faffd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:51:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.214+0000 7f14faffd700 1 -- 172.21.15.2:0/2846887017 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151409b320 con 0x557b9350cb90 2023-11-16T16:51:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.214+0000 7f1535460b80 1 -- 172.21.15.2:0/2846887017 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557b934f9710 con 0x557b9350cb90 2023-11-16T16:51:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.214+0000 7f14faffd700 1 -- 172.21.15.2:0/2846887017 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f151409aec0 con 0x557b9350cb90 2023-11-16T16:51:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.214+0000 7f14faffd700 1 -- 172.21.15.2:0/2846887017 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f151409eef0 con 0x557b9350cb90 2023-11-16T16:51:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.214+0000 7f1535460b80 10 client.15567 fetch_fsmap learned FSMap version 30 2023-11-16T16:51:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.214+0000 7f1535460b80 1 -- 172.21.15.2:0/2846887017 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x557b9341e130 con 0x557b9350cb90 2023-11-16T16:51:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.214+0000 7f14faffd700 1 -- 172.21.15.2:0/2846887017 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f15140c9020 con 0x557b9350cb90 2023-11-16T16:51:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.214+0000 7f14faffd700 1 -- 172.21.15.2:0/2846887017 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f151409a440 con 0x557b9350cb90 2023-11-16T16:51:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.214+0000 7f1535460b80 10 client.15567 fetch_fsmap finished waiting for FSMap version 30 2023-11-16T16:51:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.215+0000 7f1535460b80 10 client.15567 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.215+0000 7f1535460b80 20 client.15567 populate_metadata read hostname 'smithi002' 2023-11-16T16:51:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.215+0000 7f1535460b80 1 --2- 172.21.15.2:0/2846887017 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x557b9351fd20 0x557b93522130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.215+0000 7f1535460b80 4 client.15567 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:51:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.215+0000 7f1535460b80 1 -- 172.21.15.2:0/2846887017 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557b9341a040 con 0x557b9351fd20 2023-11-16T16:51:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.215+0000 7f151b7fe700 1 --2- 172.21.15.2:0/2846887017 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x557b9351fd20 0x557b93522130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.216+0000 7f151b7fe700 1 --2- 172.21.15.2:0/2846887017 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x557b9351fd20 0x557b93522130 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75ddac23812bade8 server_cookie=a12df9211dca230 in_seq=0 out_seq=0 2023-11-16T16:51:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.216+0000 7f14faffd700 10 client.15567 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:51:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.216+0000 7f14faffd700 1 -- 172.21.15.2:0/2846887017 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x557b9341a040 con 0x557b9351fd20 2023-11-16T16:51:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.216+0000 7f14faffd700 10 client.15567 handle_command_reply: tid=0 2023-11-16T16:51:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.216+0000 7f1535460b80 1 client.15567 shutdown 2023-11-16T16:51:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.216+0000 7f1535460b80 1 -- 172.21.15.2:0/2846887017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350cb90 msgr2=0x557b935102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.216+0000 7f1535460b80 1 --2- 172.21.15.2:0/2846887017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350cb90 0x557b935102c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f15140cb4b0 tx=0x7f151409aaf0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.217+0000 7f1535460b80 1 -- 172.21.15.2:0/2846887017 shutdown_connections 2023-11-16T16:51:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.217+0000 7f1535460b80 1 --2- 172.21.15.2:0/2846887017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350cb90 0x557b935102c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.217+0000 7f1535460b80 1 --2- 172.21.15.2:0/2846887017 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b933626b0 0x557b93512a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.217+0000 7f1535460b80 1 --2- 172.21.15.2:0/2846887017 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x557b9351fd20 0x557b93522130 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.217+0000 7f1535460b80 1 --2- 172.21.15.2:0/2846887017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b9350c720 0x557b9350fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.217+0000 7f1535460b80 1 -- 172.21.15.2:0/2846887017 >> 172.21.15.2:0/2846887017 conn(0x557b934fbf50 msgr2=0x557b935007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:36.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.217+0000 7f14faffd700 0 client.15567 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:51:36.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.217+0000 7f1535460b80 1 -- 172.21.15.2:0/2846887017 shutdown_connections 2023-11-16T16:51:36.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.217+0000 7f1535460b80 1 -- 172.21.15.2:0/2846887017 wait complete. 2023-11-16T16:51:36.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.217+0000 7f1535460b80 20 client.15567 trim_cache size 0 max 16384 2023-11-16T16:51:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.233+0000 7f1535460b80 1 Processor -- start 2023-11-16T16:51:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.234+0000 7f1535460b80 1 -- start start 2023-11-16T16:51:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.234+0000 7f1535460b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b933626b0 0x557b93517090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.234+0000 7f1535460b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350c720 0x557b93500f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.234+0000 7f1535460b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b9350cb90 0x557b935125b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.234+0000 7f1535460b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b93502370 con 0x557b9350c720 2023-11-16T16:51:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.234+0000 7f1535460b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b9350e1f0 con 0x557b933626b0 2023-11-16T16:51:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.234+0000 7f1535460b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b9350a940 con 0x557b9350cb90 2023-11-16T16:51:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.234+0000 7f151affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350c720 0x557b93500f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.234+0000 7f151affd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350c720 0x557b93500f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33336/0 (socket says 172.21.15.2:33336) 2023-11-16T16:51:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.234+0000 7f151affd700 1 -- 172.21.15.2:0/1526261189 learned_addr learned my addr 172.21.15.2:0/1526261189 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.234+0000 7f151bfff700 1 --2- 172.21.15.2:0/1526261189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b9350cb90 0x557b935125b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.234+0000 7f151b7fe700 1 --2- 172.21.15.2:0/1526261189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b933626b0 0x557b93517090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.235+0000 7f151affd700 1 -- 172.21.15.2:0/1526261189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b9350cb90 msgr2=0x557b935125b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.235+0000 7f151affd700 1 --2- 172.21.15.2:0/1526261189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b9350cb90 0x557b935125b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.235+0000 7f151affd700 1 -- 172.21.15.2:0/1526261189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b933626b0 msgr2=0x557b93517090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.235+0000 7f151affd700 1 --2- 172.21.15.2:0/1526261189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b933626b0 0x557b93517090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.235+0000 7f151affd700 1 -- 172.21.15.2:0/1526261189 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b934fd6f0 con 0x557b9350c720 2023-11-16T16:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.235+0000 7f151affd700 1 --2- 172.21.15.2:0/1526261189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350c720 0x557b93500f60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f15100097a0 tx=0x7f1510009440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12fb8e9936b019ff server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.235+0000 7f15027fc700 1 -- 172.21.15.2:0/1526261189 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1510004290 con 0x557b9350c720 2023-11-16T16:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.235+0000 7f15027fc700 1 -- 172.21.15.2:0/1526261189 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1510004710 con 0x557b9350c720 2023-11-16T16:51:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.235+0000 7f15027fc700 1 -- 172.21.15.2:0/1526261189 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15100039d0 con 0x557b9350c720 2023-11-16T16:51:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.236+0000 7f1535460b80 1 -- 172.21.15.2:0/1526261189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350c720 msgr2=0x557b93500f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.236+0000 7f1535460b80 1 --2- 172.21.15.2:0/1526261189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350c720 0x557b93500f60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f15100097a0 tx=0x7f1510009440 comp rx=0 tx=0).stop 2023-11-16T16:51:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.236+0000 7f1535460b80 1 -- 172.21.15.2:0/1526261189 shutdown_connections 2023-11-16T16:51:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.236+0000 7f1535460b80 1 --2- 172.21.15.2:0/1526261189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b9350cb90 0x557b935125b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.236+0000 7f1535460b80 1 --2- 172.21.15.2:0/1526261189 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b933626b0 0x557b93517090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.236+0000 7f1535460b80 1 --2- 172.21.15.2:0/1526261189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350c720 0x557b93500f60 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.236+0000 7f1535460b80 1 -- 172.21.15.2:0/1526261189 >> 172.21.15.2:0/1526261189 conn(0x557b934fbf50 msgr2=0x557b93509e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.236+0000 7f1535460b80 1 -- 172.21.15.2:0/1526261189 shutdown_connections 2023-11-16T16:51:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.236+0000 7f1535460b80 1 -- 172.21.15.2:0/1526261189 wait complete. 2023-11-16T16:51:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.236+0000 7f1535460b80 1 -- start start 2023-11-16T16:51:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.237+0000 7f1535460b80 1 Processor -- start 2023-11-16T16:51:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.237+0000 7f1535460b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b933626b0 0x557b93507c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.237+0000 7f1535460b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b9350c720 0x557b935081a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.237+0000 7f1535460b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350cb90 0x557b935086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.237+0000 7f1535460b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b93508ce0 con 0x557b9350cb90 2023-11-16T16:51:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.237+0000 7f1535460b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b93508e50 con 0x557b9350c720 2023-11-16T16:51:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.237+0000 7f1535460b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b93509010 con 0x557b933626b0 2023-11-16T16:51:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.237+0000 7f1535460b80 10 client.0 _ll_register_callbacks cb 0x7f1514097240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.237+0000 7f151bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350cb90 0x557b935086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.237+0000 7f151b7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b933626b0 0x557b93507c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.237+0000 7f151bfff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350cb90 0x557b935086e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33344/0 (socket says 172.21.15.2:33344) 2023-11-16T16:51:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.237+0000 7f151bfff700 1 -- 172.21.15.2:0/2006796782 learned_addr learned my addr 172.21.15.2:0/2006796782 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.237+0000 7f151affd700 1 --2- 172.21.15.2:0/2006796782 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b9350c720 0x557b935081a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.238+0000 7f151bfff700 1 -- 172.21.15.2:0/2006796782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b933626b0 msgr2=0x557b93507c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.238+0000 7f151bfff700 1 --2- 172.21.15.2:0/2006796782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b933626b0 0x557b93507c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.238+0000 7f151bfff700 1 -- 172.21.15.2:0/2006796782 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b9350c720 msgr2=0x557b935081a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.238+0000 7f151bfff700 1 --2- 172.21.15.2:0/2006796782 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b9350c720 0x557b935081a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.238+0000 7f151bfff700 1 -- 172.21.15.2:0/2006796782 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f151001d040 con 0x557b9350cb90 2023-11-16T16:51:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.238+0000 7f151bfff700 1 --2- 172.21.15.2:0/2006796782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350cb90 0x557b935086e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f151409ae50 tx=0x7f15140b9350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8782f53f2f5f49f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.238+0000 7f14faffd700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:51:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.238+0000 7f14faffd700 1 -- 172.21.15.2:0/2006796782 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514005000 con 0x557b9350cb90 2023-11-16T16:51:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.238+0000 7f1535460b80 1 -- 172.21.15.2:0/2006796782 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1510004510 con 0x557b9350cb90 2023-11-16T16:51:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.238+0000 7f14faffd700 1 -- 172.21.15.2:0/2006796782 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f15140ca080 con 0x557b9350cb90 2023-11-16T16:51:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.239+0000 7f14faffd700 1 -- 172.21.15.2:0/2006796782 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15140a2f10 con 0x557b9350cb90 2023-11-16T16:51:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.239+0000 7f1535460b80 10 client.15573 fetch_fsmap learned FSMap version 30 2023-11-16T16:51:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.239+0000 7f1535460b80 1 -- 172.21.15.2:0/2006796782 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x557b93285d80 con 0x557b9350cb90 2023-11-16T16:51:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.239+0000 7f14faffd700 1 -- 172.21.15.2:0/2006796782 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1510004510 con 0x557b9350cb90 2023-11-16T16:51:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.239+0000 7f14faffd700 1 -- 172.21.15.2:0/2006796782 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1514096cb0 con 0x557b9350cb90 2023-11-16T16:51:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.239+0000 7f1535460b80 10 client.15573 fetch_fsmap finished waiting for FSMap version 30 2023-11-16T16:51:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.239+0000 7f1535460b80 10 client.15573 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.239+0000 7f1535460b80 20 client.15573 populate_metadata read hostname 'smithi002' 2023-11-16T16:51:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.239+0000 7f1535460b80 1 --2- 172.21.15.2:0/2006796782 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x557b935742c0 0x557b935766a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.239+0000 7f1535460b80 4 client.15573 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:51:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.239+0000 7f1535460b80 1 -- 172.21.15.2:0/2006796782 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557b93426600 con 0x557b935742c0 2023-11-16T16:51:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.239+0000 7f151b7fe700 1 --2- 172.21.15.2:0/2006796782 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x557b935742c0 0x557b935766a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.240+0000 7f151b7fe700 1 --2- 172.21.15.2:0/2006796782 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x557b935742c0 0x557b935766a0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47c1e5f376b96317 server_cookie=418c0d701d5d7c35 in_seq=0 out_seq=0 2023-11-16T16:51:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.240+0000 7f14faffd700 10 client.15573 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:51:36.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.248+0000 7f14faffd700 1 -- 172.21.15.2:0/2006796782 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557b93426600 con 0x557b935742c0 2023-11-16T16:51:36.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.248+0000 7f14faffd700 10 client.15573 handle_command_reply: tid=0 2023-11-16T16:51:36.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.248+0000 7f1535460b80 1 client.15573 shutdown 2023-11-16T16:51:36.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.248+0000 7f1535460b80 1 -- 172.21.15.2:0/2006796782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350cb90 msgr2=0x557b935086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:36.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.248+0000 7f1535460b80 1 --2- 172.21.15.2:0/2006796782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350cb90 0x557b935086e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f151409ae50 tx=0x7f15140b9350 comp rx=0 tx=0).stop 2023-11-16T16:51:36.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.248+0000 7f1535460b80 1 -- 172.21.15.2:0/2006796782 shutdown_connections 2023-11-16T16:51:36.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.248+0000 7f1535460b80 1 --2- 172.21.15.2:0/2006796782 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x557b935742c0 0x557b935766a0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.248+0000 7f1535460b80 1 --2- 172.21.15.2:0/2006796782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b933626b0 0x557b93507c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.248+0000 7f1535460b80 1 --2- 172.21.15.2:0/2006796782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b9350cb90 0x557b935086e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.248+0000 7f1535460b80 1 --2- 172.21.15.2:0/2006796782 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557b9350c720 0x557b935081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.248+0000 7f14faffd700 0 client.15573 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:51:36.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.248+0000 7f1535460b80 1 -- 172.21.15.2:0/2006796782 >> 172.21.15.2:0/2006796782 conn(0x557b934fbf50 msgr2=0x557b9350cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:36.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.248+0000 7f1535460b80 1 -- 172.21.15.2:0/2006796782 shutdown_connections 2023-11-16T16:51:36.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.248+0000 7f1535460b80 1 -- 172.21.15.2:0/2006796782 wait complete. 2023-11-16T16:51:36.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.248+0000 7f1535460b80 20 client.15573 trim_cache size 0 max 16384 2023-11-16T16:51:36.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:51:36.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:51:36.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:51:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:51:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.249+0000 7f1521bff700 1 -- 172.21.15.2:0/61507530 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f15040a1340 msgr2=0x7f15040a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.249+0000 7f1521bff700 1 --2- 172.21.15.2:0/61507530 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f15040a1340 0x7f15040a37f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f150c009c60 tx=0x7f150c009260 comp rx=0 tx=0).stop 2023-11-16T16:51:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.249+0000 7f1521bff700 1 -- 172.21.15.2:0/61507530 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f151c105530 msgr2=0x7f151c1a46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.249+0000 7f1521bff700 1 --2- 172.21.15.2:0/61507530 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f151c105530 0x7f151c1a46d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1514007850 tx=0x7f1514007880 comp rx=0 tx=0).stop 2023-11-16T16:51:36.251 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.249+0000 7f1521bff700 1 -- 172.21.15.2:0/61507530 shutdown_connections 2023-11-16T16:51:36.252 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.249+0000 7f1521bff700 1 --2- 172.21.15.2:0/61507530 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f15040a1340 0x7f15040a37f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.252 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.249+0000 7f1521bff700 1 --2- 172.21.15.2:0/61507530 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f151c105530 0x7f151c1a46d0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.252 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.249+0000 7f1521bff700 1 --2- 172.21.15.2:0/61507530 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f151c100540 0x7f151c072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.252 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.249+0000 7f1521bff700 1 --2- 172.21.15.2:0/61507530 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f151c100ee0 0x7f151c073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:36.252 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.249+0000 7f1521bff700 1 -- 172.21.15.2:0/61507530 >> 172.21.15.2:0/61507530 conn(0x7f151c078210 msgr2=0x7f151c0fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:36.252 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.249+0000 7f1521bff700 1 -- 172.21.15.2:0/61507530 shutdown_connections 2023-11-16T16:51:36.252 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:36.249+0000 7f1521bff700 1 -- 172.21.15.2:0/61507530 wait complete. 2023-11-16T16:51:36.690 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:36.690 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:36.826 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:36.826 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:36.841 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:36.841 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:36.841 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 32 -lt 100 ']' 2023-11-16T16:51:36.841 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 32 + 1 2023-11-16T16:51:36.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:36 smithi155 ceph-mon[131607]: pgmap v54: 129 pgs: 129 active+clean; 1.5 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-11-16T16:51:36.871 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=33 2023-11-16T16:51:36.871 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:36.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:36 smithi002 ceph-mon[128712]: pgmap v54: 129 pgs: 129 active+clean; 1.5 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-11-16T16:51:36.907 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:36.908 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:36.910 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:36.911 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:36.911 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:37.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:36 smithi141 ceph-mon[131184]: pgmap v54: 129 pgs: 129 active+clean; 1.5 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-11-16T16:51:37.306 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:37.306 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:37.401 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:37.401 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:37.416 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:37.416 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:37.416 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 33 -lt 100 ']' 2023-11-16T16:51:37.416 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 33 + 1 2023-11-16T16:51:37.446 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=34 2023-11-16T16:51:37.446 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:37.487 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:37.487 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:37.489 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:37.490 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:37.490 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:37.923 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:37.923 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:38.034 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:38.034 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:38.049 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:38.049 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:38.049 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 34 -lt 100 ']' 2023-11-16T16:51:38.050 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 34 + 1 2023-11-16T16:51:38.079 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=35 2023-11-16T16:51:38.079 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:38.104 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:38.105 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:38.107 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:38.108 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:38.108 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:38.531 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:38.531 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:38.617 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:38.617 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:38.632 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:38.632 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:38.633 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 35 -lt 100 ']' 2023-11-16T16:51:38.633 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 35 + 1 2023-11-16T16:51:38.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:38 smithi002 ceph-mon[128712]: pgmap v55: 129 pgs: 129 active+clean; 1.6 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 12 op/s 2023-11-16T16:51:38.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:38 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:51:38.663 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=36 2023-11-16T16:51:38.663 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:38.689 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:38.690 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:38.692 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:38.693 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:38.693 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:38.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:38 smithi155 ceph-mon[131607]: pgmap v55: 129 pgs: 129 active+clean; 1.6 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 12 op/s 2023-11-16T16:51:38.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:38 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:51:39.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:38 smithi141 ceph-mon[131184]: pgmap v55: 129 pgs: 129 active+clean; 1.6 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 12 op/s 2023-11-16T16:51:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:38 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:51:39.089 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:39.089 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:39.184 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:39.184 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:39.199 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:39.200 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:39.200 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 36 -lt 100 ']' 2023-11-16T16:51:39.200 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 36 + 1 2023-11-16T16:51:39.229 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=37 2023-11-16T16:51:39.230 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:39.256 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:39.257 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:39.259 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:39.260 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:39.260 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:39.656 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:39.656 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:39.750 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:39.751 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:39.765 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:39.766 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:39.766 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 37 -lt 100 ']' 2023-11-16T16:51:39.766 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 37 + 1 2023-11-16T16:51:39.796 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=38 2023-11-16T16:51:39.796 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:39.832 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:39.833 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:39.835 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:39.836 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:39.836 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:40.231 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:40.231 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:40.392 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:40.392 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:40.426 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:40.426 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:40.426 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 38 -lt 100 ']' 2023-11-16T16:51:40.427 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 38 + 1 2023-11-16T16:51:40.459 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=39 2023-11-16T16:51:40.459 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:40.515 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:40.515 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:40.518 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:40.519 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:40.519 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:40.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:40 smithi155 ceph-mon[131607]: pgmap v56: 129 pgs: 129 active+clean; 1.6 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-11-16T16:51:40.883 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:40.883 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:40.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:40 smithi002 ceph-mon[128712]: pgmap v56: 129 pgs: 129 active+clean; 1.6 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-11-16T16:51:40.975 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:40.976 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:40.991 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:40.991 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:40.991 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 39 -lt 100 ']' 2023-11-16T16:51:40.991 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 39 + 1 2023-11-16T16:51:41.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:40 smithi141 ceph-mon[131184]: pgmap v56: 129 pgs: 129 active+clean; 1.6 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-11-16T16:51:41.021 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=40 2023-11-16T16:51:41.021 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:41.066 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:41.066 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:41.068 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:41.069 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:41.069 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:42.663 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:43.746 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:42 smithi141 ceph-mon[131184]: pgmap v57: 129 pgs: 129 active+clean; 1.7 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 12 op/s 2023-11-16T16:51:43.746 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:42 smithi155 ceph-mon[131607]: pgmap v57: 129 pgs: 129 active+clean; 1.7 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 12 op/s 2023-11-16T16:51:43.746 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:43.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:42 smithi002 ceph-mon[128712]: pgmap v57: 129 pgs: 129 active+clean; 1.7 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 12 op/s 2023-11-16T16:51:43.748 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:43.748 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:43.748 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:43.748 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:43.748 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 40 -lt 100 ']' 2023-11-16T16:51:43.748 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 40 + 1 2023-11-16T16:51:43.748 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=41 2023-11-16T16:51:43.748 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:43.748 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:43.749 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:43.749 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:43.749 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:43.749 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:43.749 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:43.749 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:43.899 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:43.900 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:43.915 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:43.915 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:43.915 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 41 -lt 100 ']' 2023-11-16T16:51:43.915 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 41 + 1 2023-11-16T16:51:43.945 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=42 2023-11-16T16:51:43.945 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:44.274 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:44.274 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:44.277 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:44.278 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:44.278 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:44.746 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:44.747 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:44.883 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:44.883 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:44.898 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:44.899 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:44.899 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 42 -lt 100 ']' 2023-11-16T16:51:44.899 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 42 + 1 2023-11-16T16:51:44.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:44 smithi002 ceph-mon[128712]: pgmap v58: 129 pgs: 129 active+clean; 1.7 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-11-16T16:51:44.928 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=43 2023-11-16T16:51:44.928 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:44.955 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:44.956 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:44.959 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:44.959 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:44.960 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:45.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:44 smithi141 ceph-mon[131184]: pgmap v58: 129 pgs: 129 active+clean; 1.7 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-11-16T16:51:45.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:44 smithi155 ceph-mon[131607]: pgmap v58: 129 pgs: 129 active+clean; 1.7 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-11-16T16:51:45.213 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:45.213 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:45.299 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:45.299 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:45.314 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:45.315 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:45.315 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 43 -lt 100 ']' 2023-11-16T16:51:45.315 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 43 + 1 2023-11-16T16:51:45.345 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=44 2023-11-16T16:51:45.345 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:45.368 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:45.368 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:45.371 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:45.372 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:45.372 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:45.846 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:45.847 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:45.945 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:45.945 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:45.956 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:45.956 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:45.956 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 44 -lt 100 ']' 2023-11-16T16:51:45.957 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 44 + 1 2023-11-16T16:51:45.986 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=45 2023-11-16T16:51:45.986 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:46.027 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:46.028 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:46.030 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:46.031 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:46.031 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:46.371 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:46.372 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:46.491 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:46.491 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:46.507 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:46.508 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:46.508 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 45 -lt 100 ']' 2023-11-16T16:51:46.508 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 45 + 1 2023-11-16T16:51:46.537 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=46 2023-11-16T16:51:46.537 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:46.562 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:46.562 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:46.565 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:46.565 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:46.566 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:46.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:46 smithi002 ceph-mon[128712]: pgmap v59: 129 pgs: 129 active+clean; 1.7 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-11-16T16:51:46.921 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:46.922 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:47.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:46 smithi141 ceph-mon[131184]: pgmap v59: 129 pgs: 129 active+clean; 1.7 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-11-16T16:51:47.041 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:47.041 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:47.056 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:47.056 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:47.056 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 46 -lt 100 ']' 2023-11-16T16:51:47.057 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 46 + 1 2023-11-16T16:51:47.086 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=47 2023-11-16T16:51:47.087 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:47.112 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:47.113 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:47.115 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:47.116 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:47.116 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:47.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:46 smithi155 ceph-mon[131607]: pgmap v59: 129 pgs: 129 active+clean; 1.7 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-11-16T16:51:47.571 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:47.571 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:47.708 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:47.708 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:47.723 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:47.723 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:47.723 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 47 -lt 100 ']' 2023-11-16T16:51:47.724 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 47 + 1 2023-11-16T16:51:47.753 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=48 2023-11-16T16:51:47.753 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:47.781 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:47.781 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:47.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:47.784 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:47.785 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:48.221 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:48.221 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:48.349 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:48.349 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:48.369 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:48.369 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:48.369 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 48 -lt 100 ']' 2023-11-16T16:51:48.369 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 48 + 1 2023-11-16T16:51:48.395 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=49 2023-11-16T16:51:48.395 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:48.423 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:48.423 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:48.425 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:48.425 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:48.426 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:48.838 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:48.838 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:48.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:48 smithi002 ceph-mon[128712]: pgmap v60: 129 pgs: 129 active+clean; 1.8 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 12 op/s 2023-11-16T16:51:49.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:48 smithi141 ceph-mon[131184]: pgmap v60: 129 pgs: 129 active+clean; 1.8 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 12 op/s 2023-11-16T16:51:49.040 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:49.040 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:49.057 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:49.057 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:49.057 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 49 -lt 100 ']' 2023-11-16T16:51:49.057 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 49 + 1 2023-11-16T16:51:49.086 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=50 2023-11-16T16:51:49.086 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:49.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:48 smithi155 ceph-mon[131607]: pgmap v60: 129 pgs: 129 active+clean; 1.8 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 12 op/s 2023-11-16T16:51:49.122 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:49.123 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:49.125 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:49.126 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:49.126 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:49.629 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:49.629 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:49.849 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:49.849 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:49.864 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:49.864 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:49.864 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 50 -lt 100 ']' 2023-11-16T16:51:49.865 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 50 + 1 2023-11-16T16:51:49.900 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=51 2023-11-16T16:51:49.900 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:49.947 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:49.948 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:49.950 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:49.951 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:49.951 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:50.662 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:50.662 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:50.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:50 smithi002 ceph-mon[128712]: pgmap v61: 129 pgs: 129 active+clean; 1.8 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-11-16T16:51:51.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:50 smithi141 ceph-mon[131184]: pgmap v61: 129 pgs: 129 active+clean; 1.8 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-11-16T16:51:51.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:50 smithi155 ceph-mon[131607]: pgmap v61: 129 pgs: 129 active+clean; 1.8 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-11-16T16:51:51.424 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:51.424 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:51.456 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:51.457 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:51.457 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 51 -lt 100 ']' 2023-11-16T16:51:51.457 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 51 + 1 2023-11-16T16:51:51.494 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=52 2023-11-16T16:51:51.494 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:51.521 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:51.521 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:51.524 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:51.525 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:51.525 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.913+0000 7fd6a3dff700 1 -- 172.21.15.2:0/1853050478 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd69c108470 msgr2=0x7fd69c108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.913+0000 7fd6a3dff700 1 --2- 172.21.15.2:0/1853050478 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd69c108470 0x7fd69c108840 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd6980098f0 tx=0x7fd698009c00 comp rx=0 tx=0).stop 2023-11-16T16:51:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.913+0000 7fd6a3dff700 1 -- 172.21.15.2:0/1853050478 shutdown_connections 2023-11-16T16:51:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.913+0000 7fd6a3dff700 1 --2- 172.21.15.2:0/1853050478 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd69c102e10 0x7fd69c10f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.913+0000 7fd6a3dff700 1 --2- 172.21.15.2:0/1853050478 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd69c108470 0x7fd69c108840 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.913+0000 7fd6a3dff700 1 --2- 172.21.15.2:0/1853050478 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd69c102460 0x7fd69c1028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.913+0000 7fd6a3dff700 1 -- 172.21.15.2:0/1853050478 >> 172.21.15.2:0/1853050478 conn(0x7fd69c0fdf60 msgr2=0x7fd69c100370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.913+0000 7fd6a3dff700 1 -- 172.21.15.2:0/1853050478 shutdown_connections 2023-11-16T16:51:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.913+0000 7fd6a3dff700 1 -- 172.21.15.2:0/1853050478 wait complete. 2023-11-16T16:51:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.914+0000 7fd6a3dff700 1 Processor -- start 2023-11-16T16:51:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.914+0000 7fd6a3dff700 1 -- start start 2023-11-16T16:51:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.915+0000 7fd6a3dff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd69c102460 0x7fd69c198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.915+0000 7fd6a3dff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd69c102e10 0x7fd69c198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.915+0000 7fd6a3dff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd69c108470 0x7fd69c19cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.915+0000 7fd6a3dff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c19d600 con 0x7fd69c108470 2023-11-16T16:51:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.915+0000 7fd6a3dff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c19d770 con 0x7fd69c102e10 2023-11-16T16:51:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.915+0000 7fd6a3dff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c19d930 con 0x7fd69c102460 2023-11-16T16:51:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.915+0000 7fd6a239c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd69c108470 0x7fd69c19cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.915+0000 7fd6a1b9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd69c102460 0x7fd69c198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.918 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.915+0000 7fd6a1b9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd69c102460 0x7fd69c198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:39766/0 (socket says 172.21.15.2:39766) 2023-11-16T16:51:51.918 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.915+0000 7fd6a239c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd69c108470 0x7fd69c19cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46022/0 (socket says 172.21.15.2:46022) 2023-11-16T16:51:51.918 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.915+0000 7fd6a1b9b700 1 -- 172.21.15.2:0/1920657458 learned_addr learned my addr 172.21.15.2:0/1920657458 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:51.918 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.915+0000 7fd6a139a700 1 --2- 172.21.15.2:0/1920657458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd69c102e10 0x7fd69c198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.918 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.915+0000 7fd6a1b9b700 1 -- 172.21.15.2:0/1920657458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd69c102e10 msgr2=0x7fd69c198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:51.918 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.915+0000 7fd6a1b9b700 1 --2- 172.21.15.2:0/1920657458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd69c102e10 0x7fd69c198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.918 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.915+0000 7fd6a1b9b700 1 -- 172.21.15.2:0/1920657458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd69c108470 msgr2=0x7fd69c19cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:51.918 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.915+0000 7fd6a1b9b700 1 --2- 172.21.15.2:0/1920657458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd69c108470 0x7fd69c19cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.918 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.915+0000 7fd6a1b9b700 1 -- 172.21.15.2:0/1920657458 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd694009d00 con 0x7fd69c102460 2023-11-16T16:51:51.918 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.915+0000 7fd6a239c700 1 --2- 172.21.15.2:0/1920657458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd69c108470 0x7fd69c19cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:51.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.916+0000 7fd6a1b9b700 1 --2- 172.21.15.2:0/1920657458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd69c102460 0x7fd69c198620 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd698009810 tx=0x7fd69800fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:51.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.916+0000 7fd692ffd700 1 -- 172.21.15.2:0/1920657458 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69801c070 con 0x7fd69c102460 2023-11-16T16:51:51.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.916+0000 7fd6a3dff700 1 -- 172.21.15.2:0/1920657458 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd698009520 con 0x7fd69c102460 2023-11-16T16:51:51.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.916+0000 7fd692ffd700 1 -- 172.21.15.2:0/1920657458 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd69800bb20 con 0x7fd69c102460 2023-11-16T16:51:51.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.916+0000 7fd692ffd700 1 -- 172.21.15.2:0/1920657458 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd698004930 con 0x7fd69c102460 2023-11-16T16:51:51.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.916+0000 7fd6a3dff700 1 -- 172.21.15.2:0/1920657458 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69c1124b0 con 0x7fd69c102460 2023-11-16T16:51:51.921 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.919+0000 7fd692ffd700 1 -- 172.21.15.2:0/1920657458 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fd698020070 con 0x7fd69c102460 2023-11-16T16:51:51.922 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.921+0000 7fd692ffd700 1 --2- 172.21.15.2:0/1920657458 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fd6880a12c0 0x7fd6880a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.921+0000 7fd692ffd700 1 -- 172.21.15.2:0/1920657458 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fd6980cdda0 con 0x7fd69c102460 2023-11-16T16:51:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.921+0000 7fd6a139a700 1 --2- 172.21.15.2:0/1920657458 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fd6880a12c0 0x7fd6880a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.922+0000 7fd6a139a700 1 --2- 172.21.15.2:0/1920657458 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fd6880a12c0 0x7fd6880a3770 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd69c199c30 tx=0x7fd68c009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.980+0000 7fd6b76fbb80 1 Processor -- start 2023-11-16T16:51:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.980+0000 7fd6b76fbb80 1 -- start start 2023-11-16T16:51:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.981+0000 7fd6b76fbb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 0x55df117da1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.981+0000 7fd6b76fbb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df117da720 0x55df117d4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.981+0000 7fd6b76fbb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117dab90 0x55df117d4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.981+0000 7fd6b76fbb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df116da000 con 0x55df116306b0 2023-11-16T16:51:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.981+0000 7fd6b76fbb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df1172c840 con 0x55df117dab90 2023-11-16T16:51:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.981+0000 7fd6b76fbb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df11725cd0 con 0x55df117da720 2023-11-16T16:51:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.981+0000 7fd6a1b9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 0x55df117da1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.981+0000 7fd6a1b9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 0x55df117da1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46032/0 (socket says 172.21.15.2:46032) 2023-11-16T16:51:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.981+0000 7fd6a1b9b700 1 -- 172.21.15.2:0/117904914 learned_addr learned my addr 172.21.15.2:0/117904914 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.981+0000 7fd6a139a700 1 --2- 172.21.15.2:0/117904914 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df117da720 0x55df117d4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.981+0000 7fd6a239c700 1 --2- 172.21.15.2:0/117904914 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117dab90 0x55df117d4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.982+0000 7fd6a1b9b700 1 -- 172.21.15.2:0/117904914 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df117da720 msgr2=0x55df117d4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.982+0000 7fd6a1b9b700 1 --2- 172.21.15.2:0/117904914 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df117da720 0x55df117d4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.982+0000 7fd6a1b9b700 1 -- 172.21.15.2:0/117904914 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117dab90 msgr2=0x55df117d4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.982+0000 7fd6a1b9b700 1 --2- 172.21.15.2:0/117904914 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117dab90 0x55df117d4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.982+0000 7fd6a1b9b700 1 -- 172.21.15.2:0/117904914 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df115a75d0 con 0x55df116306b0 2023-11-16T16:51:51.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.982+0000 7fd6a1b9b700 1 --2- 172.21.15.2:0/117904914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 0x55df117da1e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd698097030 tx=0x7fd6980054e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7107bcf7a1f4161 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.982+0000 7fd687fff700 1 -- 172.21.15.2:0/117904914 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69809eb60 con 0x55df116306b0 2023-11-16T16:51:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.982+0000 7fd687fff700 1 -- 172.21.15.2:0/117904914 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6980a2ef0 con 0x55df116306b0 2023-11-16T16:51:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.982+0000 7fd687fff700 1 -- 172.21.15.2:0/117904914 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69809a020 con 0x55df116306b0 2023-11-16T16:51:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.983+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/117904914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 msgr2=0x55df117da1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.983+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/117904914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 0x55df117da1e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd698097030 tx=0x7fd6980054e0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.983+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/117904914 shutdown_connections 2023-11-16T16:51:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.983+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/117904914 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117dab90 0x55df117d4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.983+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/117904914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 0x55df117da1e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.983+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/117904914 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df117da720 0x55df117d4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.983+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/117904914 >> 172.21.15.2:0/117904914 conn(0x55df117c9f50 msgr2=0x55df117cf270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.983+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/117904914 shutdown_connections 2023-11-16T16:51:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.983+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/117904914 wait complete. 2023-11-16T16:51:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.984+0000 7fd6b76fbb80 1 -- start start 2023-11-16T16:51:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.984+0000 7fd6b76fbb80 1 Processor -- start 2023-11-16T16:51:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.985+0000 7fd6b76fbb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df116306b0 0x55df117e3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.985+0000 7fd6b76fbb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117da720 0x55df117e3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.985+0000 7fd6b76fbb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df117dab90 0x55df117ddb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.985+0000 7fd6b76fbb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df116c7880 con 0x55df117dab90 2023-11-16T16:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.985+0000 7fd6b76fbb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df116c7520 con 0x55df117da720 2023-11-16T16:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.985+0000 7fd6b76fbb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df115a98c0 con 0x55df116306b0 2023-11-16T16:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.985+0000 7fd6b76fbb80 10 client.0 _ll_register_callbacks cb 0x55df116e5720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.985+0000 7fd6a1b9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df116306b0 0x55df117e3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.985+0000 7fd6a239c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df117dab90 0x55df117ddb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.985+0000 7fd6a1b9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df116306b0 0x55df117e3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:39794/0 (socket says 172.21.15.2:39794) 2023-11-16T16:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.985+0000 7fd6a1b9b700 1 -- 172.21.15.2:0/415986724 learned_addr learned my addr 172.21.15.2:0/415986724 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.985+0000 7fd6a139a700 1 --2- 172.21.15.2:0/415986724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117da720 0x55df117e3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.986+0000 7fd6a1b9b700 1 -- 172.21.15.2:0/415986724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117da720 msgr2=0x55df117e3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.986+0000 7fd6a1b9b700 1 --2- 172.21.15.2:0/415986724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117da720 0x55df117e3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.986+0000 7fd6a1b9b700 1 -- 172.21.15.2:0/415986724 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df117dab90 msgr2=0x55df117ddb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.986+0000 7fd6a1b9b700 1 --2- 172.21.15.2:0/415986724 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df117dab90 0x55df117ddb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.986+0000 7fd6a1b9b700 1 -- 172.21.15.2:0/415986724 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69c1124b0 con 0x55df116306b0 2023-11-16T16:51:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.986+0000 7fd6a239c700 1 --2- 172.21.15.2:0/415986724 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df117dab90 0x55df117ddb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.986+0000 7fd6a1b9b700 1 --2- 172.21.15.2:0/415986724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df116306b0 0x55df117e3810 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd6980a29c0 tx=0x7fd698005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76bbd66af1067df7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.986+0000 7fd6857fa700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:51:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.986+0000 7fd6857fa700 1 -- 172.21.15.2:0/415986724 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69809eb60 con 0x55df116306b0 2023-11-16T16:51:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.986+0000 7fd6857fa700 1 -- 172.21.15.2:0/415986724 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fd69809ad90 con 0x55df116306b0 2023-11-16T16:51:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.986+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/415986724 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55df117c7710 con 0x55df116306b0 2023-11-16T16:51:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.987+0000 7fd6857fa700 1 -- 172.21.15.2:0/415986724 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd69809af50 con 0x55df116306b0 2023-11-16T16:51:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.987+0000 7fd6b76fbb80 10 client.25591 fetch_fsmap learned FSMap version 31 2023-11-16T16:51:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.987+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/415986724 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55df116ec130 con 0x55df116306b0 2023-11-16T16:51:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.987+0000 7fd6857fa700 1 -- 172.21.15.2:0/415986724 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55df117c7710 con 0x55df116306b0 2023-11-16T16:51:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.987+0000 7fd6857fa700 1 -- 172.21.15.2:0/415986724 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 31) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fd69800b5e0 con 0x55df116306b0 2023-11-16T16:51:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.987+0000 7fd6b76fbb80 10 client.25591 fetch_fsmap finished waiting for FSMap version 31 2023-11-16T16:51:51.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.987+0000 7fd6b76fbb80 10 client.25591 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:51.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.987+0000 7fd6b76fbb80 20 client.25591 populate_metadata read hostname 'smithi002' 2023-11-16T16:51:51.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.987+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/415986724 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55df117ed7e0 0x55df117efbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:51.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.987+0000 7fd6b76fbb80 4 client.25591 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:51:51.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.987+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/415986724 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55df116e8040 con 0x55df117ed7e0 2023-11-16T16:51:51.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.988+0000 7fd6a139a700 1 --2- 172.21.15.2:0/415986724 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55df117ed7e0 0x55df117efbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.988+0000 7fd6a139a700 1 --2- 172.21.15.2:0/415986724 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55df117ed7e0 0x55df117efbf0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f830f1a2c9d012d6 server_cookie=76c51672af771851 in_seq=0 out_seq=0 2023-11-16T16:51:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.988+0000 7fd6857fa700 10 client.25591 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:51:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.989+0000 7fd6857fa700 1 -- 172.21.15.2:0/415986724 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55df116e8040 con 0x55df117ed7e0 2023-11-16T16:51:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.989+0000 7fd6857fa700 10 client.25591 handle_command_reply: tid=0 2023-11-16T16:51:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.989+0000 7fd6b76fbb80 1 client.25591 shutdown 2023-11-16T16:51:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.989+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/415986724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df116306b0 msgr2=0x55df117e3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.989+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/415986724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df116306b0 0x55df117e3810 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd6980a29c0 tx=0x7fd698005040 comp rx=0 tx=0).stop 2023-11-16T16:51:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.989+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/415986724 shutdown_connections 2023-11-16T16:51:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.989+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/415986724 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55df117ed7e0 0x55df117efbf0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.989+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/415986724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df116306b0 0x55df117e3810 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.989+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/415986724 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df117dab90 0x55df117ddb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.989+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/415986724 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117da720 0x55df117e3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.989+0000 7fd6857fa700 0 client.25591 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:51:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.989+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/415986724 >> 172.21.15.2:0/415986724 conn(0x55df117c9f50 msgr2=0x55df117ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.989+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/415986724 shutdown_connections 2023-11-16T16:51:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.990+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/415986724 wait complete. 2023-11-16T16:51:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:51.990+0000 7fd6b76fbb80 20 client.25591 trim_cache size 0 max 16384 2023-11-16T16:51:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.006+0000 7fd6b76fbb80 1 Processor -- start 2023-11-16T16:51:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.006+0000 7fd6b76fbb80 1 -- start start 2023-11-16T16:51:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.006+0000 7fd6b76fbb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 0x55df117cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.006+0000 7fd6b76fbb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df117da720 0x55df117d4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.006+0000 7fd6b76fbb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117dab90 0x55df117d4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.006+0000 7fd6b76fbb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df117df150 con 0x55df116306b0 2023-11-16T16:51:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.006+0000 7fd6b76fbb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df117d8850 con 0x55df117dab90 2023-11-16T16:51:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.006+0000 7fd6b76fbb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df117ea560 con 0x55df117da720 2023-11-16T16:51:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.007+0000 7fd6a1b9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 0x55df117cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.007+0000 7fd6a239c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117dab90 0x55df117d4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.007+0000 7fd6a1b9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 0x55df117cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46050/0 (socket says 172.21.15.2:46050) 2023-11-16T16:51:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.007+0000 7fd6a1b9b700 1 -- 172.21.15.2:0/3948899780 learned_addr learned my addr 172.21.15.2:0/3948899780 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.007+0000 7fd6a139a700 1 --2- 172.21.15.2:0/3948899780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df117da720 0x55df117d4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.007+0000 7fd6a239c700 1 --2- 172.21.15.2:0/3948899780 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117dab90 0x55df117d4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:42780/0 (socket says 172.21.15.2:42780) 2023-11-16T16:51:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.007+0000 7fd6a139a700 1 -- 172.21.15.2:0/3948899780 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117dab90 msgr2=0x55df117d4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.007+0000 7fd6a139a700 1 --2- 172.21.15.2:0/3948899780 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117dab90 0x55df117d4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.007+0000 7fd6a139a700 1 -- 172.21.15.2:0/3948899780 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 msgr2=0x55df117cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.007+0000 7fd6a139a700 1 --2- 172.21.15.2:0/3948899780 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 0x55df117cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.007+0000 7fd6a139a700 1 -- 172.21.15.2:0/3948899780 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df117cb6f0 con 0x55df117da720 2023-11-16T16:51:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.007+0000 7fd6a1b9b700 1 --2- 172.21.15.2:0/3948899780 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 0x55df117cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.007+0000 7fd6a139a700 1 --2- 172.21.15.2:0/3948899780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df117da720 0x55df117d4560 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x55df117f0460 tx=0x7fd68c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=625148008358bebd server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.008+0000 7fd690bff700 1 -- 172.21.15.2:0/3948899780 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c022040 con 0x55df117da720 2023-11-16T16:51:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.008+0000 7fd690bff700 1 -- 172.21.15.2:0/3948899780 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd68c023040 con 0x55df117da720 2023-11-16T16:51:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.008+0000 7fd690bff700 1 -- 172.21.15.2:0/3948899780 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c003d60 con 0x55df117da720 2023-11-16T16:51:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.008+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/3948899780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df117da720 msgr2=0x55df117d4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.008+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/3948899780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df117da720 0x55df117d4560 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x55df117f0460 tx=0x7fd68c014000 comp rx=0 tx=0).stop 2023-11-16T16:51:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.008+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/3948899780 shutdown_connections 2023-11-16T16:51:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.008+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/3948899780 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117dab90 0x55df117d4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.008+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/3948899780 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 0x55df117cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.008+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/3948899780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df117da720 0x55df117d4560 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.008+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/3948899780 >> 172.21.15.2:0/3948899780 conn(0x55df117c9f50 msgr2=0x55df117de470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.009+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/3948899780 shutdown_connections 2023-11-16T16:51:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.009+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/3948899780 wait complete. 2023-11-16T16:51:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.009+0000 7fd6b76fbb80 1 -- start start 2023-11-16T16:51:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.009+0000 7fd6b76fbb80 1 Processor -- start 2023-11-16T16:51:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.010+0000 7fd6b76fbb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 0x55df117e12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.010+0000 7fd6b76fbb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df117da720 0x55df117e1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.010+0000 7fd6b76fbb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117dab90 0x55df117e1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.010+0000 7fd6b76fbb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df118406a0 con 0x55df116306b0 2023-11-16T16:51:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.010+0000 7fd6b76fbb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df11840810 con 0x55df117dab90 2023-11-16T16:51:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.010+0000 7fd6b76fbb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df118409d0 con 0x55df117da720 2023-11-16T16:51:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.010+0000 7fd6b76fbb80 10 client.0 _ll_register_callbacks cb 0x7fd69400e5f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:51:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.010+0000 7fd6a1b9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 0x55df117e12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.010+0000 7fd6a1b9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 0x55df117e12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46052/0 (socket says 172.21.15.2:46052) 2023-11-16T16:51:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.010+0000 7fd6a1b9b700 1 -- 172.21.15.2:0/3254580111 learned_addr learned my addr 172.21.15.2:0/3254580111 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:51:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.010+0000 7fd6a139a700 1 --2- 172.21.15.2:0/3254580111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df117da720 0x55df117e1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.010+0000 7fd6a239c700 1 --2- 172.21.15.2:0/3254580111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117dab90 0x55df117e1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.011+0000 7fd6a1b9b700 1 -- 172.21.15.2:0/3254580111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df117da720 msgr2=0x55df117e1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.011+0000 7fd6a1b9b700 1 --2- 172.21.15.2:0/3254580111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df117da720 0x55df117e1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.011+0000 7fd6a1b9b700 1 -- 172.21.15.2:0/3254580111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117dab90 msgr2=0x55df117e1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:51:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.011+0000 7fd6a1b9b700 1 --2- 172.21.15.2:0/3254580111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117dab90 0x55df117e1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.011+0000 7fd6a1b9b700 1 -- 172.21.15.2:0/3254580111 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55df116ec130 con 0x55df116306b0 2023-11-16T16:51:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.011+0000 7fd6a139a700 1 --2- 172.21.15.2:0/3254580111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df117da720 0x55df117e1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:51:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.011+0000 7fd6a1b9b700 1 --2- 172.21.15.2:0/3254580111 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 0x55df117e12e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd69800fab0 tx=0x7fd69800bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=574a32ec33831e00 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:51:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.011+0000 7fd6857fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:51:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.011+0000 7fd6857fa700 1 -- 172.21.15.2:0/3254580111 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69809a490 con 0x55df116306b0 2023-11-16T16:51:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.011+0000 7fd6857fa700 1 -- 172.21.15.2:0/3254580111 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fd69809eb60 con 0x55df116306b0 2023-11-16T16:51:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.011+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/3254580111 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6980a2ef0 con 0x55df116306b0 2023-11-16T16:51:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.011+0000 7fd6857fa700 1 -- 172.21.15.2:0/3254580111 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd69809ad90 con 0x55df116306b0 2023-11-16T16:51:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.011+0000 7fd6b76fbb80 10 client.15591 fetch_fsmap learned FSMap version 31 2023-11-16T16:51:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.011+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/3254580111 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7fd68c017070 con 0x55df116306b0 2023-11-16T16:51:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.012+0000 7fd6857fa700 1 -- 172.21.15.2:0/3254580111 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6980a2ef0 con 0x55df116306b0 2023-11-16T16:51:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.012+0000 7fd6857fa700 1 -- 172.21.15.2:0/3254580111 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 31) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fd698099c90 con 0x55df116306b0 2023-11-16T16:51:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.012+0000 7fd6b76fbb80 10 client.15591 fetch_fsmap finished waiting for FSMap version 31 2023-11-16T16:51:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.012+0000 7fd6b76fbb80 10 client.15591 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:51:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.012+0000 7fd6b76fbb80 20 client.15591 populate_metadata read hostname 'smithi002' 2023-11-16T16:51:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.012+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/3254580111 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55df11841cb0 0x55df118440c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:51:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.012+0000 7fd6b76fbb80 4 client.15591 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:51:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.012+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/3254580111 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55df116f4600 con 0x55df11841cb0 2023-11-16T16:51:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.012+0000 7fd6a139a700 1 --2- 172.21.15.2:0/3254580111 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55df11841cb0 0x55df118440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:51:52.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.013+0000 7fd6a139a700 1 --2- 172.21.15.2:0/3254580111 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55df11841cb0 0x55df118440c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=606bdbf110bb91d0 server_cookie=8fe3ba962051107f in_seq=0 out_seq=0 2023-11-16T16:51:52.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.013+0000 7fd6857fa700 10 client.15591 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:51:52.023 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.021+0000 7fd6857fa700 1 -- 172.21.15.2:0/3254580111 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55df116f4600 con 0x55df11841cb0 2023-11-16T16:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.022+0000 7fd6857fa700 10 client.15591 handle_command_reply: tid=0 2023-11-16T16:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.022+0000 7fd6b76fbb80 1 client.15591 shutdown 2023-11-16T16:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.022+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/3254580111 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 msgr2=0x55df117e12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.022+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/3254580111 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 0x55df117e12e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd69800fab0 tx=0x7fd69800bc90 comp rx=0 tx=0).stop 2023-11-16T16:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.022+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/3254580111 shutdown_connections 2023-11-16T16:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.022+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/3254580111 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55df117dab90 0x55df117e1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.022+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/3254580111 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55df116306b0 0x55df117e12e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.022+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/3254580111 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55df11841cb0 0x55df118440c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.022+0000 7fd6b76fbb80 1 --2- 172.21.15.2:0/3254580111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df117da720 0x55df117e1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.022+0000 7fd6857fa700 0 client.15591 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.022+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/3254580111 >> 172.21.15.2:0/3254580111 conn(0x55df117c9f50 msgr2=0x55df117dc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.022+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/3254580111 shutdown_connections 2023-11-16T16:51:52.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.022+0000 7fd6b76fbb80 1 -- 172.21.15.2:0/3254580111 wait complete. 2023-11-16T16:51:52.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.022+0000 7fd6b76fbb80 20 client.15591 trim_cache size 0 max 16384 2023-11-16T16:51:52.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:51:52.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:51:52.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:51:52.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:51:52.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.023+0000 7fd6a3dff700 1 -- 172.21.15.2:0/1920657458 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fd6880a12c0 msgr2=0x7fd6880a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:52.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.023+0000 7fd6a3dff700 1 --2- 172.21.15.2:0/1920657458 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fd6880a12c0 0x7fd6880a3770 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd69c199c30 tx=0x7fd68c009260 comp rx=0 tx=0).stop 2023-11-16T16:51:52.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.023+0000 7fd6a3dff700 1 -- 172.21.15.2:0/1920657458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd69c102460 msgr2=0x7fd69c198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:51:52.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.023+0000 7fd6a3dff700 1 --2- 172.21.15.2:0/1920657458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd69c102460 0x7fd69c198620 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd698009810 tx=0x7fd69800fa00 comp rx=0 tx=0).stop 2023-11-16T16:51:52.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.023+0000 7fd6a3dff700 1 -- 172.21.15.2:0/1920657458 shutdown_connections 2023-11-16T16:51:52.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.023+0000 7fd6a3dff700 1 --2- 172.21.15.2:0/1920657458 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fd6880a12c0 0x7fd6880a3770 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.023+0000 7fd6a3dff700 1 --2- 172.21.15.2:0/1920657458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd69c102460 0x7fd69c198620 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.023+0000 7fd6a3dff700 1 --2- 172.21.15.2:0/1920657458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd69c108470 0x7fd69c19cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.023+0000 7fd6a3dff700 1 --2- 172.21.15.2:0/1920657458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fd69c102e10 0x7fd69c198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:51:52.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.023+0000 7fd6a3dff700 1 -- 172.21.15.2:0/1920657458 >> 172.21.15.2:0/1920657458 conn(0x7fd69c0fdf60 msgr2=0x7fd69c105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:51:52.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.023+0000 7fd6a3dff700 1 -- 172.21.15.2:0/1920657458 shutdown_connections 2023-11-16T16:51:52.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:51:52.023+0000 7fd6a3dff700 1 -- 172.21.15.2:0/1920657458 wait complete. 2023-11-16T16:51:52.271 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:52.271 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:52.407 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:52.407 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:52.422 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:52.422 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:52.422 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 52 -lt 100 ']' 2023-11-16T16:51:52.422 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 52 + 1 2023-11-16T16:51:52.452 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=53 2023-11-16T16:51:52.452 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:52.489 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:52.489 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:52.492 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:52.492 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:52.493 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:52.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:52 smithi002 ceph-mon[128712]: pgmap v62: 129 pgs: 129 active+clean; 1.8 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 11 op/s 2023-11-16T16:51:52.911 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:52.912 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:53.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:52 smithi141 ceph-mon[131184]: pgmap v62: 129 pgs: 129 active+clean; 1.8 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 11 op/s 2023-11-16T16:51:53.040 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:53.040 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:53.055 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:53.056 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:53.056 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 53 -lt 100 ']' 2023-11-16T16:51:53.056 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 53 + 1 2023-11-16T16:51:53.085 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=54 2023-11-16T16:51:53.085 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:53.114 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:53.114 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:53.117 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:53.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:52 smithi155 ceph-mon[131607]: pgmap v62: 129 pgs: 129 active+clean; 1.8 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 11 op/s 2023-11-16T16:51:53.117 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:53.118 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:53.495 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:53.495 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:53.623 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:53.623 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:53.639 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:53.639 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:53.639 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 54 -lt 100 ']' 2023-11-16T16:51:53.639 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 54 + 1 2023-11-16T16:51:53.668 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=55 2023-11-16T16:51:53.669 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:53.716 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:53.716 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:53.719 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:53.719 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:53.720 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:54.136 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:54.137 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:54.240 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:54.240 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:54.255 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:54.255 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:54.255 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 55 -lt 100 ']' 2023-11-16T16:51:54.256 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 55 + 1 2023-11-16T16:51:54.285 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=56 2023-11-16T16:51:54.285 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:54.321 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:54.322 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:54.324 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:54.325 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:54.325 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:54.687 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:54.687 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:54.841 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:54.841 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:54.864 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:54.864 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:54.864 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 56 -lt 100 ']' 2023-11-16T16:51:54.865 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 56 + 1 2023-11-16T16:51:54.893 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=57 2023-11-16T16:51:54.894 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:54.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:54 smithi002 ceph-mon[128712]: pgmap v63: 129 pgs: 129 active+clean; 1.8 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-11-16T16:51:54.921 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:54.921 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:54.924 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:54.925 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:54.925 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:55.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:54 smithi141 ceph-mon[131184]: pgmap v63: 129 pgs: 129 active+clean; 1.8 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-11-16T16:51:55.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:54 smithi155 ceph-mon[131607]: pgmap v63: 129 pgs: 129 active+clean; 1.8 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-11-16T16:51:55.303 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:55.303 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:55.439 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:55.439 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:55.455 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:55.455 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:55.455 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 57 -lt 100 ']' 2023-11-16T16:51:55.455 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 57 + 1 2023-11-16T16:51:55.491 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=58 2023-11-16T16:51:55.491 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:55.563 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:55.563 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:55.566 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:55.566 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:55.567 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:55.969 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:55.970 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:56.073 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:56.073 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:56.088 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:56.088 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:56.089 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 58 -lt 100 ']' 2023-11-16T16:51:56.089 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 58 + 1 2023-11-16T16:51:56.118 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=59 2023-11-16T16:51:56.118 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:56.162 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:56.163 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:56.165 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:56.166 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:56.166 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:56.595 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:56.595 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:56.722 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:56.723 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:56.738 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:56.738 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:56.738 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 59 -lt 100 ']' 2023-11-16T16:51:56.738 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 59 + 1 2023-11-16T16:51:56.785 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=60 2023-11-16T16:51:56.785 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:56.820 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:56.820 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:56.823 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:56.823 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:56.824 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:56.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:56 smithi002 ceph-mon[128712]: pgmap v64: 129 pgs: 129 active+clean; 1.8 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-11-16T16:51:57.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:56 smithi141 ceph-mon[131184]: pgmap v64: 129 pgs: 129 active+clean; 1.8 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-11-16T16:51:57.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:56 smithi155 ceph-mon[131607]: pgmap v64: 129 pgs: 129 active+clean; 1.8 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-11-16T16:51:57.227 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:57.228 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:57.331 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:57.331 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:57.347 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:57.347 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:57.347 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 60 -lt 100 ']' 2023-11-16T16:51:57.348 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 60 + 1 2023-11-16T16:51:57.376 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=61 2023-11-16T16:51:57.376 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:57.404 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:57.404 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:57.407 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:57.407 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:57.408 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:57.903 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:57.903 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:58.014 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:58.014 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:58.029 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:58.029 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:58.029 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 61 -lt 100 ']' 2023-11-16T16:51:58.030 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 61 + 1 2023-11-16T16:51:58.068 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=62 2023-11-16T16:51:58.068 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:58.094 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:58.094 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:58.097 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:58.097 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:58.098 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:58.452 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:58.453 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:58.572 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:58.572 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:58.588 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:58.588 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:58.588 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 62 -lt 100 ']' 2023-11-16T16:51:58.588 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 62 + 1 2023-11-16T16:51:58.618 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=63 2023-11-16T16:51:58.618 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:58.651 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:58.651 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:58.654 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:58.654 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:58.655 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:58.903 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:51:58 smithi002 ceph-mon[128712]: pgmap v65: 129 pgs: 129 active+clean; 1.9 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 11 op/s 2023-11-16T16:51:59.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:51:58 smithi141 ceph-mon[131184]: pgmap v65: 129 pgs: 129 active+clean; 1.9 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 11 op/s 2023-11-16T16:51:59.044 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:59.044 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:59.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:51:58 smithi155 ceph-mon[131607]: pgmap v65: 129 pgs: 129 active+clean; 1.9 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 11 op/s 2023-11-16T16:51:59.155 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:59.156 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:59.170 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:59.171 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:59.171 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 63 -lt 100 ']' 2023-11-16T16:51:59.171 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 63 + 1 2023-11-16T16:51:59.218 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=64 2023-11-16T16:51:59.218 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:59.256 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:59.256 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:59.258 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:59.259 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:59.259 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:51:59.636 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:59.636 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:59.781 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:51:59.781 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:51:59.796 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:59.796 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:51:59.796 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 64 -lt 100 ']' 2023-11-16T16:51:59.797 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 64 + 1 2023-11-16T16:51:59.832 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=65 2023-11-16T16:51:59.832 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:51:59.890 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:51:59.890 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:51:59.892 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:51:59.893 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:51:59.894 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:00.252 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:00.253 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:00.372 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:00.372 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:00.387 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:00.387 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:00.387 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 65 -lt 100 ']' 2023-11-16T16:52:00.388 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 65 + 1 2023-11-16T16:52:00.417 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=66 2023-11-16T16:52:00.417 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:00.444 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:00.445 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:00.448 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:00.448 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:00.449 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:00.802 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:00.802 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:00.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:00 smithi002 ceph-mon[128712]: pgmap v66: 129 pgs: 129 active+clean; 1.9 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-11-16T16:52:00.931 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:00.931 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:00.946 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:00.946 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:00.946 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 66 -lt 100 ']' 2023-11-16T16:52:00.946 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 66 + 1 2023-11-16T16:52:00.976 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=67 2023-11-16T16:52:00.976 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:01.004 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:01.004 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:01.006 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:01.007 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:01.007 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:01.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:00 smithi141 ceph-mon[131184]: pgmap v66: 129 pgs: 129 active+clean; 1.9 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-11-16T16:52:01.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:00 smithi155 ceph-mon[131607]: pgmap v66: 129 pgs: 129 active+clean; 1.9 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-11-16T16:52:01.452 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:01.452 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:01.589 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:01.589 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:01.604 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:01.604 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:01.604 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 67 -lt 100 ']' 2023-11-16T16:52:01.604 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 67 + 1 2023-11-16T16:52:01.651 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=68 2023-11-16T16:52:01.651 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:01.679 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:01.679 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:01.682 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:01.682 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:01.683 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:02.035 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:02.036 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:02.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:02.147 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:02.163 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:02.163 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:02.163 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 68 -lt 100 ']' 2023-11-16T16:52:02.164 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 68 + 1 2023-11-16T16:52:02.192 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=69 2023-11-16T16:52:02.193 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:02.220 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:02.220 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:02.222 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:02.223 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:02.223 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:02.635 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:02.636 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:02.772 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:02.772 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:02.787 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:02.787 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:02.787 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 69 -lt 100 ']' 2023-11-16T16:52:02.788 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 69 + 1 2023-11-16T16:52:02.834 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=70 2023-11-16T16:52:02.834 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:02.861 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:02.862 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:02.864 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:02.865 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:02.865 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:02.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:02 smithi002 ceph-mon[128712]: pgmap v67: 129 pgs: 129 active+clean; 2.1 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 46 KiB/s wr, 23 op/s 2023-11-16T16:52:03.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:02 smithi141 ceph-mon[131184]: pgmap v67: 129 pgs: 129 active+clean; 2.1 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 46 KiB/s wr, 23 op/s 2023-11-16T16:52:03.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:02 smithi155 ceph-mon[131607]: pgmap v67: 129 pgs: 129 active+clean; 2.1 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 46 KiB/s wr, 23 op/s 2023-11-16T16:52:03.218 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:03.219 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:03.388 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:03.388 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:03.404 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:03.404 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:03.404 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 70 -lt 100 ']' 2023-11-16T16:52:03.404 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 70 + 1 2023-11-16T16:52:03.434 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=71 2023-11-16T16:52:03.434 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:03.470 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:03.470 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:03.473 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:03.473 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:03.473 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:03.818 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:03.819 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:03.930 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:03.930 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:03.979 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:03.980 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:03.980 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 71 -lt 100 ']' 2023-11-16T16:52:03.980 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 71 + 1 2023-11-16T16:52:04.031 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=72 2023-11-16T16:52:04.031 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:04.081 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:04.082 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:04.084 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:04.085 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:04.085 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:04.486 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:04.486 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:04.622 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:04.622 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:04.638 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:04.638 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:04.638 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 72 -lt 100 ']' 2023-11-16T16:52:04.638 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 72 + 1 2023-11-16T16:52:04.667 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=73 2023-11-16T16:52:04.667 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:04.703 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:04.704 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:04.706 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:04.707 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:04.707 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:04.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:04 smithi002 ceph-mon[128712]: pgmap v68: 129 pgs: 129 active+clean; 2.1 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 42 KiB/s wr, 21 op/s 2023-11-16T16:52:05.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:04 smithi141 ceph-mon[131184]: pgmap v68: 129 pgs: 129 active+clean; 2.1 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 42 KiB/s wr, 21 op/s 2023-11-16T16:52:05.068 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:05.069 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:05.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:04 smithi155 ceph-mon[131607]: pgmap v68: 129 pgs: 129 active+clean; 2.1 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 42 KiB/s wr, 21 op/s 2023-11-16T16:52:05.188 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:05.188 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:05.203 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:05.203 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:05.204 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 73 -lt 100 ']' 2023-11-16T16:52:05.204 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 73 + 1 2023-11-16T16:52:05.233 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=74 2023-11-16T16:52:05.234 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:05.262 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:05.262 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:05.265 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:05.265 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:05.266 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:05.818 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:05.818 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:06.005 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:06.005 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:06.020 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:06.020 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:06.020 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 74 -lt 100 ']' 2023-11-16T16:52:06.021 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 74 + 1 2023-11-16T16:52:06.056 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=75 2023-11-16T16:52:06.056 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:06.103 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:06.104 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:06.106 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:06.107 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:06.107 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:06.835 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:06.835 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:06.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:06 smithi002 ceph-mon[128712]: pgmap v69: 129 pgs: 129 active+clean; 2.1 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 42 KiB/s wr, 21 op/s 2023-11-16T16:52:06.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:06 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:52:06.989 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:06.989 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:07.004 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:07.004 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:07.005 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 75 -lt 100 ']' 2023-11-16T16:52:07.005 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 75 + 1 2023-11-16T16:52:07.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:06 smithi141 ceph-mon[131184]: pgmap v69: 129 pgs: 129 active+clean; 2.1 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 42 KiB/s wr, 21 op/s 2023-11-16T16:52:07.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:06 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:52:07.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:06 smithi155 ceph-mon[131607]: pgmap v69: 129 pgs: 129 active+clean; 2.1 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 42 KiB/s wr, 21 op/s 2023-11-16T16:52:07.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:06 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:52:07.131 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=76 2023-11-16T16:52:07.131 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:07.165 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:07.166 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:07.168 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:07.168 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:07.169 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.960+0000 7ffa9e9ff700 1 -- 172.21.15.2:0/3992791056 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa98104400 msgr2=0x7ffa9810bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.960+0000 7ffa9e9ff700 1 --2- 172.21.15.2:0/3992791056 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa98104400 0x7ffa9810bfd0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000b0d0 tx=0x7ffa9000b3e0 comp rx=0 tx=0).stop 2023-11-16T16:52:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.960+0000 7ffa9e9ff700 1 -- 172.21.15.2:0/3992791056 shutdown_connections 2023-11-16T16:52:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.960+0000 7ffa9e9ff700 1 --2- 172.21.15.2:0/3992791056 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa98104400 0x7ffa9810bfd0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.960+0000 7ffa9e9ff700 1 --2- 172.21.15.2:0/3992791056 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffa980ff3c0 0x7ffa980ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.960+0000 7ffa9e9ff700 1 --2- 172.21.15.2:0/3992791056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa980ffd60 0x7ffa98103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:07.962 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.960+0000 7ffa9e9ff700 1 -- 172.21.15.2:0/3992791056 >> 172.21.15.2:0/3992791056 conn(0x7ffa980763a0 msgr2=0x7ffa980767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.960+0000 7ffa9e9ff700 1 -- 172.21.15.2:0/3992791056 shutdown_connections 2023-11-16T16:52:07.963 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.960+0000 7ffa9e9ff700 1 -- 172.21.15.2:0/3992791056 wait complete. 2023-11-16T16:52:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.995+0000 7ffa9e9ff700 1 Processor -- start 2023-11-16T16:52:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.995+0000 7ffa9e9ff700 1 -- start start 2023-11-16T16:52:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.996+0000 7ffa9e9ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa980ff3c0 0x7ffa9819e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.996+0000 7ffa9e9ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffa980ffd60 0x7ffa9819ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.996+0000 7ffa9e9ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa98104400 0x7ffa98198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.996+0000 7ffa9e9ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa98198be0 con 0x7ffa980ff3c0 2023-11-16T16:52:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.996+0000 7ffa9e9ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa98198d50 con 0x7ffa980ffd60 2023-11-16T16:52:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.996+0000 7ffa9e9ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa98198f10 con 0x7ffa98104400 2023-11-16T16:52:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.996+0000 7ffa97fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa980ff3c0 0x7ffa9819e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.996+0000 7ffa97fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa980ff3c0 0x7ffa9819e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54142/0 (socket says 172.21.15.2:54142) 2023-11-16T16:52:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.996+0000 7ffa97fff700 1 -- 172.21.15.2:0/1588637948 learned_addr learned my addr 172.21.15.2:0/1588637948 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.996+0000 7ffa8ffff700 1 --2- 172.21.15.2:0/1588637948 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffa980ffd60 0x7ffa9819ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.996+0000 7ffa9cf9c700 1 --2- 172.21.15.2:0/1588637948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa98104400 0x7ffa98198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.997+0000 7ffa97fff700 1 -- 172.21.15.2:0/1588637948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa98104400 msgr2=0x7ffa98198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.997+0000 7ffa97fff700 1 --2- 172.21.15.2:0/1588637948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa98104400 0x7ffa98198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.997+0000 7ffa97fff700 1 -- 172.21.15.2:0/1588637948 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffa980ffd60 msgr2=0x7ffa9819ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:08.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.997+0000 7ffa97fff700 1 --2- 172.21.15.2:0/1588637948 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffa980ffd60 0x7ffa9819ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.997+0000 7ffa97fff700 1 -- 172.21.15.2:0/1588637948 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa90009d00 con 0x7ffa980ff3c0 2023-11-16T16:52:08.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.997+0000 7ffa97fff700 1 --2- 172.21.15.2:0/1588637948 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa980ff3c0 0x7ffa9819e500 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ffa8800e9a0 tx=0x7ffa8800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:08.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.997+0000 7ffa95ffb700 1 -- 172.21.15.2:0/1588637948 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa88009790 con 0x7ffa980ff3c0 2023-11-16T16:52:08.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.997+0000 7ffa9e9ff700 1 -- 172.21.15.2:0/1588637948 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa9810f220 con 0x7ffa980ff3c0 2023-11-16T16:52:08.003 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.997+0000 7ffa95ffb700 1 -- 172.21.15.2:0/1588637948 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa88010430 con 0x7ffa980ff3c0 2023-11-16T16:52:08.003 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.997+0000 7ffa95ffb700 1 -- 172.21.15.2:0/1588637948 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa88018620 con 0x7ffa980ff3c0 2023-11-16T16:52:08.003 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:07.997+0000 7ffa9e9ff700 1 -- 172.21.15.2:0/1588637948 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa9810f090 con 0x7ffa980ff3c0 2023-11-16T16:52:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.000+0000 7ffa95ffb700 1 -- 172.21.15.2:0/1588637948 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7ffa880105a0 con 0x7ffa980ff3c0 2023-11-16T16:52:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.002+0000 7ffa95ffb700 1 --2- 172.21.15.2:0/1588637948 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ffa840a12c0 0x7ffa840a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.002+0000 7ffa95ffb700 1 -- 172.21.15.2:0/1588637948 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ffa88014070 con 0x7ffa980ff3c0 2023-11-16T16:52:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.002+0000 7ffa8ffff700 1 --2- 172.21.15.2:0/1588637948 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ffa840a12c0 0x7ffa840a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:08.004 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.003+0000 7ffa8ffff700 1 --2- 172.21.15.2:0/1588637948 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ffa840a12c0 0x7ffa840a3770 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ffa98105740 tx=0x7ffa80009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:08.034 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:08.034 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:08.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:07 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:52:08.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:07 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:52:08.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:07 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:52:08.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:07 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:52:08.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:07 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:52:08.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:07 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:52:08.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:07 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:52:08.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:07 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:52:08.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.202+0000 7ffab23ccb80 1 Processor -- start 2023-11-16T16:52:08.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.203+0000 7ffab23ccb80 1 -- start start 2023-11-16T16:52:08.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.203+0000 7ffab23ccb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dadf6b0 0x56020dc891e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.203+0000 7ffab23ccb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 0x56020dc83560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.203+0000 7ffab23ccb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dc89b90 0x56020dc83b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.203+0000 7ffab23ccb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56020db89000 con 0x56020dc89720 2023-11-16T16:52:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.203+0000 7ffab23ccb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x56020dbdb840 con 0x56020dadf6b0 2023-11-16T16:52:08.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.203+0000 7ffab23ccb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x56020dbd4cd0 con 0x56020dc89b90 2023-11-16T16:52:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.203+0000 7ffa8ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 0x56020dc83560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:08.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.203+0000 7ffa9cf9c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dc89b90 0x56020dc83b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.203+0000 7ffa8ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 0x56020dc83560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54152/0 (socket says 172.21.15.2:54152) 2023-11-16T16:52:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.203+0000 7ffa97fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dadf6b0 0x56020dc891e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.203+0000 7ffa8ffff700 1 -- 172.21.15.2:0/3038244664 learned_addr learned my addr 172.21.15.2:0/3038244664 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.204+0000 7ffa8ffff700 1 -- 172.21.15.2:0/3038244664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dc89b90 msgr2=0x56020dc83b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.204+0000 7ffa8ffff700 1 --2- 172.21.15.2:0/3038244664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dc89b90 0x56020dc83b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.204+0000 7ffa8ffff700 1 -- 172.21.15.2:0/3038244664 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dadf6b0 msgr2=0x56020dc891e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.204+0000 7ffa8ffff700 1 --2- 172.21.15.2:0/3038244664 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dadf6b0 0x56020dc891e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.204+0000 7ffa8ffff700 1 -- 172.21.15.2:0/3038244664 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56020da565d0 con 0x56020dc89720 2023-11-16T16:52:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.204+0000 7ffa8ffff700 1 --2- 172.21.15.2:0/3038244664 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 0x56020dc83560 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ffa80011cd0 tx=0x7ffa80014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3329b79d230628d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.204+0000 7ffa8e7fc700 1 -- 172.21.15.2:0/3038244664 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa80004030 con 0x56020dc89720 2023-11-16T16:52:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.204+0000 7ffa8e7fc700 1 -- 172.21.15.2:0/3038244664 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa8000e840 con 0x56020dc89720 2023-11-16T16:52:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.204+0000 7ffa8e7fc700 1 -- 172.21.15.2:0/3038244664 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa80003ab0 con 0x56020dc89720 2023-11-16T16:52:08.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.205+0000 7ffab23ccb80 1 -- 172.21.15.2:0/3038244664 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 msgr2=0x56020dc83560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:08.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.205+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/3038244664 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 0x56020dc83560 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ffa80011cd0 tx=0x7ffa80014000 comp rx=0 tx=0).stop 2023-11-16T16:52:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.205+0000 7ffab23ccb80 1 -- 172.21.15.2:0/3038244664 shutdown_connections 2023-11-16T16:52:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.205+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/3038244664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dc89b90 0x56020dc83b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.205+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/3038244664 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dadf6b0 0x56020dc891e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.205+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/3038244664 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 0x56020dc83560 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.205+0000 7ffab23ccb80 1 -- 172.21.15.2:0/3038244664 >> 172.21.15.2:0/3038244664 conn(0x56020dc78f50 msgr2=0x56020dc7e270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.205+0000 7ffab23ccb80 1 -- 172.21.15.2:0/3038244664 shutdown_connections 2023-11-16T16:52:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.205+0000 7ffab23ccb80 1 -- 172.21.15.2:0/3038244664 wait complete. 2023-11-16T16:52:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.206+0000 7ffab23ccb80 1 -- start start 2023-11-16T16:52:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.206+0000 7ffab23ccb80 1 Processor -- start 2023-11-16T16:52:08.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.207+0000 7ffab23ccb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dadf6b0 0x56020dc8fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:08.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.207+0000 7ffab23ccb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 0x56020dc8cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:08.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.207+0000 7ffab23ccb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dc89b90 0x56020dc8d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:08.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.207+0000 7ffab23ccb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56020db76880 con 0x56020dc89720 2023-11-16T16:52:08.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.207+0000 7ffab23ccb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x56020db76520 con 0x56020dc89b90 2023-11-16T16:52:08.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.207+0000 7ffab23ccb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x56020da588c0 con 0x56020dadf6b0 2023-11-16T16:52:08.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.207+0000 7ffab23ccb80 10 client.0 _ll_register_callbacks cb 0x56020db94720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:08.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.207+0000 7ffa8ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 0x56020dc8cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:08.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.207+0000 7ffa97fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dadf6b0 0x56020dc8fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:08.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.207+0000 7ffa8ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 0x56020dc8cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54166/0 (socket says 172.21.15.2:54166) 2023-11-16T16:52:08.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.207+0000 7ffa97fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dadf6b0 0x56020dc8fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:35716/0 (socket says 172.21.15.2:35716) 2023-11-16T16:52:08.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.207+0000 7ffa8ffff700 1 -- 172.21.15.2:0/3587556031 learned_addr learned my addr 172.21.15.2:0/3587556031 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:08.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.207+0000 7ffa9cf9c700 1 --2- 172.21.15.2:0/3587556031 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dc89b90 0x56020dc8d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:08.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.207+0000 7ffa97fff700 1 -- 172.21.15.2:0/3587556031 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dc89b90 msgr2=0x56020dc8d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:08.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.207+0000 7ffa97fff700 1 --2- 172.21.15.2:0/3587556031 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dc89b90 0x56020dc8d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.208+0000 7ffa97fff700 1 -- 172.21.15.2:0/3587556031 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 msgr2=0x56020dc8cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.208+0000 7ffa97fff700 1 --2- 172.21.15.2:0/3587556031 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 0x56020dc8cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.208+0000 7ffa97fff700 1 -- 172.21.15.2:0/3587556031 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa80017070 con 0x56020dadf6b0 2023-11-16T16:52:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.208+0000 7ffa97fff700 1 --2- 172.21.15.2:0/3587556031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dadf6b0 0x56020dc8fa00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ffa880ca5c0 tx=0x7ffa88099fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f58de63317d471c1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.208+0000 7ffa777fe700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:52:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.208+0000 7ffa777fe700 1 -- 172.21.15.2:0/3587556031 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa88099630 con 0x56020dadf6b0 2023-11-16T16:52:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.208+0000 7ffa777fe700 1 -- 172.21.15.2:0/3587556031 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ffa880a2390 con 0x56020dadf6b0 2023-11-16T16:52:08.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.208+0000 7ffab23ccb80 1 -- 172.21.15.2:0/3587556031 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56020dc76710 con 0x56020dadf6b0 2023-11-16T16:52:08.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.209+0000 7ffa777fe700 1 -- 172.21.15.2:0/3587556031 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa880a1f00 con 0x56020dadf6b0 2023-11-16T16:52:08.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.209+0000 7ffab23ccb80 10 client.25609 fetch_fsmap learned FSMap version 31 2023-11-16T16:52:08.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.209+0000 7ffab23ccb80 1 -- 172.21.15.2:0/3587556031 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56020db9b130 con 0x56020dadf6b0 2023-11-16T16:52:08.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.209+0000 7ffa777fe700 1 -- 172.21.15.2:0/3587556031 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa880b8030 con 0x56020dadf6b0 2023-11-16T16:52:08.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.209+0000 7ffa777fe700 1 -- 172.21.15.2:0/3587556031 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 31) v1 ==== 7474+0+0 (secure 0 0 0) 0x7ffa88094c50 con 0x56020dadf6b0 2023-11-16T16:52:08.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.209+0000 7ffab23ccb80 10 client.25609 fetch_fsmap finished waiting for FSMap version 31 2023-11-16T16:52:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.209+0000 7ffab23ccb80 10 client.25609 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.209+0000 7ffab23ccb80 20 client.25609 populate_metadata read hostname 'smithi002' 2023-11-16T16:52:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.209+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/3587556031 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x56020dc9cfe0 0x56020dc9f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.209+0000 7ffab23ccb80 4 client.25609 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.209+0000 7ffab23ccb80 1 -- 172.21.15.2:0/3587556031 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56020db97040 con 0x56020dc9cfe0 2023-11-16T16:52:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.210+0000 7ffa8ffff700 1 --2- 172.21.15.2:0/3587556031 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x56020dc9cfe0 0x56020dc9f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.210+0000 7ffa8ffff700 1 --2- 172.21.15.2:0/3587556031 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x56020dc9cfe0 0x56020dc9f3f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5fb4645cf9b3c28 server_cookie=be6b02b64d54fcc4 in_seq=0 out_seq=0 2023-11-16T16:52:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.210+0000 7ffa777fe700 10 client.25609 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:52:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.211+0000 7ffa777fe700 1 -- 172.21.15.2:0/3587556031 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56020db97040 con 0x56020dc9cfe0 2023-11-16T16:52:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.211+0000 7ffa777fe700 10 client.25609 handle_command_reply: tid=0 2023-11-16T16:52:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.211+0000 7ffab23ccb80 1 client.25609 shutdown 2023-11-16T16:52:08.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.211+0000 7ffab23ccb80 1 -- 172.21.15.2:0/3587556031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dadf6b0 msgr2=0x56020dc8fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.211+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/3587556031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dadf6b0 0x56020dc8fa00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ffa880ca5c0 tx=0x7ffa88099fb0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.211+0000 7ffab23ccb80 1 -- 172.21.15.2:0/3587556031 shutdown_connections 2023-11-16T16:52:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.211+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/3587556031 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x56020dc9cfe0 0x56020dc9f3f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.211+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/3587556031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dadf6b0 0x56020dc8fa00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.211+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/3587556031 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dc89b90 0x56020dc8d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.211+0000 7ffa777fe700 0 client.25609 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:52:08.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.211+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/3587556031 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 0x56020dc8cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.211+0000 7ffab23ccb80 1 -- 172.21.15.2:0/3587556031 >> 172.21.15.2:0/3587556031 conn(0x56020dc78f50 msgr2=0x56020dc7d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.211+0000 7ffab23ccb80 1 -- 172.21.15.2:0/3587556031 shutdown_connections 2023-11-16T16:52:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.212+0000 7ffab23ccb80 1 -- 172.21.15.2:0/3587556031 wait complete. 2023-11-16T16:52:08.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.212+0000 7ffab23ccb80 20 client.25609 trim_cache size 0 max 16384 2023-11-16T16:52:08.221 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:08.222 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:08.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.227+0000 7ffab23ccb80 1 Processor -- start 2023-11-16T16:52:08.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.227+0000 7ffab23ccb80 1 -- start start 2023-11-16T16:52:08.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.227+0000 7ffab23ccb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dadf6b0 0x56020dc94000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:08.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.228+0000 7ffab23ccb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dc89720 0x56020dc7df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:08.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.228+0000 7ffab23ccb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89b90 0x56020dc8f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:08.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.228+0000 7ffab23ccb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56020dc7afd0 con 0x56020dc89b90 2023-11-16T16:52:08.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.228+0000 7ffab23ccb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x56020dc8e260 con 0x56020dadf6b0 2023-11-16T16:52:08.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.228+0000 7ffab23ccb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x56020dc78b70 con 0x56020dc89720 2023-11-16T16:52:08.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.228+0000 7ffa9cf9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89b90 0x56020dc8f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:08.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.228+0000 7ffa9cf9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89b90 0x56020dc8f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54172/0 (socket says 172.21.15.2:54172) 2023-11-16T16:52:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.228+0000 7ffa9cf9c700 1 -- 172.21.15.2:0/922717139 learned_addr learned my addr 172.21.15.2:0/922717139 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.228+0000 7ffa8ffff700 1 --2- 172.21.15.2:0/922717139 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dc89720 0x56020dc7df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.228+0000 7ffa97fff700 1 --2- 172.21.15.2:0/922717139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dadf6b0 0x56020dc94000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.228+0000 7ffa9cf9c700 1 -- 172.21.15.2:0/922717139 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dc89720 msgr2=0x56020dc7df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.228+0000 7ffa9cf9c700 1 --2- 172.21.15.2:0/922717139 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dc89720 0x56020dc7df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.228+0000 7ffa9cf9c700 1 -- 172.21.15.2:0/922717139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dadf6b0 msgr2=0x56020dc94000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.228+0000 7ffa9cf9c700 1 --2- 172.21.15.2:0/922717139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dadf6b0 0x56020dc94000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.228+0000 7ffa9cf9c700 1 -- 172.21.15.2:0/922717139 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56020db9ae80 con 0x56020dc89b90 2023-11-16T16:52:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.228+0000 7ffa8ffff700 1 --2- 172.21.15.2:0/922717139 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dc89720 0x56020dc7df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:52:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.228+0000 7ffa9cf9c700 1 --2- 172.21.15.2:0/922717139 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89b90 0x56020dc8f5a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ffa90009cb0 tx=0x7ffa900092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8d115964a820f3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.229+0000 7ffa8effd700 1 -- 172.21.15.2:0/922717139 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa90021070 con 0x56020dc89b90 2023-11-16T16:52:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.229+0000 7ffa8effd700 1 -- 172.21.15.2:0/922717139 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa900094b0 con 0x56020dc89b90 2023-11-16T16:52:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.229+0000 7ffa8effd700 1 -- 172.21.15.2:0/922717139 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9000e040 con 0x56020dc89b90 2023-11-16T16:52:08.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.229+0000 7ffab23ccb80 1 -- 172.21.15.2:0/922717139 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89b90 msgr2=0x56020dc8f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:08.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.229+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/922717139 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89b90 0x56020dc8f5a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ffa90009cb0 tx=0x7ffa900092e0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.229+0000 7ffab23ccb80 1 -- 172.21.15.2:0/922717139 shutdown_connections 2023-11-16T16:52:08.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.229+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/922717139 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89b90 0x56020dc8f5a0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.229+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/922717139 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dadf6b0 0x56020dc94000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.229+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/922717139 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dc89720 0x56020dc7df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.229+0000 7ffab23ccb80 1 -- 172.21.15.2:0/922717139 >> 172.21.15.2:0/922717139 conn(0x56020dc78f50 msgr2=0x56020dc837a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:08.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.230+0000 7ffab23ccb80 1 -- 172.21.15.2:0/922717139 shutdown_connections 2023-11-16T16:52:08.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.230+0000 7ffab23ccb80 1 -- 172.21.15.2:0/922717139 wait complete. 2023-11-16T16:52:08.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.230+0000 7ffab23ccb80 1 -- start start 2023-11-16T16:52:08.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.230+0000 7ffab23ccb80 1 Processor -- start 2023-11-16T16:52:08.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.231+0000 7ffab23ccb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dadf6b0 0x56020dc85ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:08.239 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.231+0000 7ffab23ccb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 0x56020dc863e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:08.239 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.231+0000 7ffab23ccb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dc89b90 0x56020dc84c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:08.239 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.231+0000 7ffab23ccb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56020dc851a0 con 0x56020dc89720 2023-11-16T16:52:08.239 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.231+0000 7ffab23ccb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x56020dc85310 con 0x56020dc89b90 2023-11-16T16:52:08.239 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.231+0000 7ffab23ccb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x56020dc854d0 con 0x56020dadf6b0 2023-11-16T16:52:08.239 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.231+0000 7ffab23ccb80 10 client.0 _ll_register_callbacks cb 0x7ffa8800e8f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.231+0000 7ffa8ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 0x56020dc863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.231+0000 7ffa8ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 0x56020dc863e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54188/0 (socket says 172.21.15.2:54188) 2023-11-16T16:52:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.231+0000 7ffa8ffff700 1 -- 172.21.15.2:0/346373811 learned_addr learned my addr 172.21.15.2:0/346373811 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.231+0000 7ffa97fff700 1 --2- 172.21.15.2:0/346373811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dadf6b0 0x56020dc85ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.231+0000 7ffa9cf9c700 1 --2- 172.21.15.2:0/346373811 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dc89b90 0x56020dc84c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:08.240 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.231+0000 7ffa8ffff700 1 -- 172.21.15.2:0/346373811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dadf6b0 msgr2=0x56020dc85ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:08.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.231+0000 7ffa8ffff700 1 --2- 172.21.15.2:0/346373811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dadf6b0 0x56020dc85ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.231+0000 7ffa8ffff700 1 -- 172.21.15.2:0/346373811 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dc89b90 msgr2=0x56020dc84c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:08.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.231+0000 7ffa8ffff700 1 --2- 172.21.15.2:0/346373811 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dc89b90 0x56020dc84c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.231+0000 7ffa8ffff700 1 -- 172.21.15.2:0/346373811 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa9810c810 con 0x56020dc89720 2023-11-16T16:52:08.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.232+0000 7ffa8ffff700 1 --2- 172.21.15.2:0/346373811 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 0x56020dc863e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ffa80005950 tx=0x7ffa800092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d42ecbe347f50d4 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:08.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.232+0000 7ffa777fe700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:52:08.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.232+0000 7ffa777fe700 1 -- 172.21.15.2:0/346373811 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8002b040 con 0x56020dc89720 2023-11-16T16:52:08.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.232+0000 7ffa777fe700 1 -- 172.21.15.2:0/346373811 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ffa80032c30 con 0x56020dc89720 2023-11-16T16:52:08.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.232+0000 7ffab23ccb80 1 -- 172.21.15.2:0/346373811 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffa8000e840 con 0x56020dc89720 2023-11-16T16:52:08.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.232+0000 7ffa777fe700 1 -- 172.21.15.2:0/346373811 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa80032df0 con 0x56020dc89720 2023-11-16T16:52:08.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.232+0000 7ffab23ccb80 10 client.15609 fetch_fsmap learned FSMap version 31 2023-11-16T16:52:08.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.232+0000 7ffab23ccb80 1 -- 172.21.15.2:0/346373811 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56020d9ebc00 con 0x56020dc89720 2023-11-16T16:52:08.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.232+0000 7ffa777fe700 1 -- 172.21.15.2:0/346373811 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa8000e840 con 0x56020dc89720 2023-11-16T16:52:08.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.233+0000 7ffa777fe700 1 -- 172.21.15.2:0/346373811 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 31) v1 ==== 7474+0+0 (secure 0 0 0) 0x7ffa800046e0 con 0x56020dc89720 2023-11-16T16:52:08.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.233+0000 7ffab23ccb80 10 client.15609 fetch_fsmap finished waiting for FSMap version 31 2023-11-16T16:52:08.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.233+0000 7ffab23ccb80 10 client.15609 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:08.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.233+0000 7ffab23ccb80 20 client.15609 populate_metadata read hostname 'smithi002' 2023-11-16T16:52:08.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.233+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/346373811 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x56020dcf1c50 0x56020dcf4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:08.243 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.233+0000 7ffab23ccb80 4 client.15609 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:52:08.244 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.233+0000 7ffab23ccb80 1 -- 172.21.15.2:0/346373811 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56020dba3600 con 0x56020dcf1c50 2023-11-16T16:52:08.244 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.233+0000 7ffa97fff700 1 --2- 172.21.15.2:0/346373811 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x56020dcf1c50 0x56020dcf4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:08.244 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.233+0000 7ffa97fff700 1 --2- 172.21.15.2:0/346373811 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x56020dcf1c50 0x56020dcf4060 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=321fa2462d2f70f2 server_cookie=1bd9e4388cc8d73 in_seq=0 out_seq=0 2023-11-16T16:52:08.244 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.233+0000 7ffa777fe700 10 client.15609 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:52:08.245 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.241+0000 7ffa777fe700 1 -- 172.21.15.2:0/346373811 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56020dba3600 con 0x56020dcf1c50 2023-11-16T16:52:08.245 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.241+0000 7ffa777fe700 10 client.15609 handle_command_reply: tid=0 2023-11-16T16:52:08.245 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.241+0000 7ffab23ccb80 1 client.15609 shutdown 2023-11-16T16:52:08.245 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.241+0000 7ffab23ccb80 1 -- 172.21.15.2:0/346373811 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 msgr2=0x56020dc863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:08.245 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.241+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/346373811 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 0x56020dc863e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ffa80005950 tx=0x7ffa800092d0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.246 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.241+0000 7ffab23ccb80 1 -- 172.21.15.2:0/346373811 shutdown_connections 2023-11-16T16:52:08.246 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.241+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/346373811 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x56020dcf1c50 0x56020dcf4060 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.246 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.241+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/346373811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56020dadf6b0 0x56020dc85ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.246 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.241+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/346373811 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56020dc89b90 0x56020dc84c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.246 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.241+0000 7ffab23ccb80 1 --2- 172.21.15.2:0/346373811 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56020dc89720 0x56020dc863e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.246 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.241+0000 7ffa777fe700 0 client.15609 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:52:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.241+0000 7ffab23ccb80 1 -- 172.21.15.2:0/346373811 >> 172.21.15.2:0/346373811 conn(0x56020dc78f50 msgr2=0x56020dc86e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.241+0000 7ffab23ccb80 1 -- 172.21.15.2:0/346373811 shutdown_connections 2023-11-16T16:52:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.241+0000 7ffab23ccb80 1 -- 172.21.15.2:0/346373811 wait complete. 2023-11-16T16:52:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.242+0000 7ffab23ccb80 20 client.15609 trim_cache size 0 max 16384 2023-11-16T16:52:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:52:08.247 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:52:08.248 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:52:08.248 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:52:08.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.242+0000 7ffa9e9ff700 1 -- 172.21.15.2:0/1588637948 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ffa840a12c0 msgr2=0x7ffa840a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:08.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.242+0000 7ffa9e9ff700 1 --2- 172.21.15.2:0/1588637948 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ffa840a12c0 0x7ffa840a3770 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ffa98105740 tx=0x7ffa80009260 comp rx=0 tx=0).stop 2023-11-16T16:52:08.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.242+0000 7ffa9e9ff700 1 -- 172.21.15.2:0/1588637948 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa980ff3c0 msgr2=0x7ffa9819e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:08.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.242+0000 7ffa9e9ff700 1 --2- 172.21.15.2:0/1588637948 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa980ff3c0 0x7ffa9819e500 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ffa8800e9a0 tx=0x7ffa8800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.243+0000 7ffa9e9ff700 1 -- 172.21.15.2:0/1588637948 shutdown_connections 2023-11-16T16:52:08.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.243+0000 7ffa9e9ff700 1 --2- 172.21.15.2:0/1588637948 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ffa840a12c0 0x7ffa840a3770 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.243+0000 7ffa9e9ff700 1 --2- 172.21.15.2:0/1588637948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa98104400 0x7ffa98198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.243+0000 7ffa9e9ff700 1 --2- 172.21.15.2:0/1588637948 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa980ff3c0 0x7ffa9819e500 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.243+0000 7ffa9e9ff700 1 --2- 172.21.15.2:0/1588637948 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ffa980ffd60 0x7ffa9819ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:08.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.243+0000 7ffa9e9ff700 1 -- 172.21.15.2:0/1588637948 >> 172.21.15.2:0/1588637948 conn(0x7ffa980763a0 msgr2=0x7ffa98068060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:08.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.243+0000 7ffa9e9ff700 1 -- 172.21.15.2:0/1588637948 shutdown_connections 2023-11-16T16:52:08.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:08.243+0000 7ffa9e9ff700 1 -- 172.21.15.2:0/1588637948 wait complete. 2023-11-16T16:52:08.251 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:08.251 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:08.251 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 76 -lt 100 ']' 2023-11-16T16:52:08.251 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 76 + 1 2023-11-16T16:52:08.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:07 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:52:08.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:07 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:52:08.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:07 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:52:08.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:07 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:52:08.330 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=77 2023-11-16T16:52:08.330 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:08.410 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:08.411 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:08.413 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:08.414 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:08.414 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:08.942 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:08.943 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:09.095 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:09.096 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:09.128 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:09.128 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:09.128 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 77 -lt 100 ']' 2023-11-16T16:52:09.129 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 77 + 1 2023-11-16T16:52:09.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:08 smithi002 ceph-mon[128712]: pgmap v70: 129 pgs: 129 active+clean; 2.1 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 46 KiB/s wr, 25 op/s 2023-11-16T16:52:09.180 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=78 2023-11-16T16:52:09.180 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:09.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:08 smithi141 ceph-mon[131184]: pgmap v70: 129 pgs: 129 active+clean; 2.1 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 46 KiB/s wr, 25 op/s 2023-11-16T16:52:09.286 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:09.286 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:09.289 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:09.290 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:09.290 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:09.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:08 smithi155 ceph-mon[131607]: pgmap v70: 129 pgs: 129 active+clean; 2.1 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 46 KiB/s wr, 25 op/s 2023-11-16T16:52:10.067 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.233 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.236 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:10 smithi155 ceph-mon[131607]: pgmap v71: 129 pgs: 129 active+clean; 2.1 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 38 KiB/s wr, 19 op/s 2023-11-16T16:52:16.236 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:12 smithi155 ceph-mon[131607]: pgmap v72: 129 pgs: 129 active+clean; 2.2 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 48 KiB/s wr, 24 op/s 2023-11-16T16:52:16.236 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:12 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:52:16.236 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:12 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:52:16.236 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:12 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:52:16.237 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:12 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:52:16.237 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:12 smithi155 ceph-mon[131607]: Service mds.all has no related daemon already placed. 2023-11-16T16:52:16.237 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:12 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:52:16.237 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:12 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:52:16.237 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:14 smithi155 ceph-mon[131607]: pgmap v73: 129 pgs: 129 active+clean; 2.2 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-11-16T16:52:16.238 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:10 smithi141 ceph-mon[131184]: pgmap v71: 129 pgs: 129 active+clean; 2.1 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 38 KiB/s wr, 19 op/s 2023-11-16T16:52:16.238 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:12 smithi141 ceph-mon[131184]: pgmap v72: 129 pgs: 129 active+clean; 2.2 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 48 KiB/s wr, 24 op/s 2023-11-16T16:52:16.238 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:12 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:52:16.238 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:12 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:52:16.238 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:12 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:52:16.238 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:12 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:52:16.239 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:12 smithi141 ceph-mon[131184]: Service mds.all has no related daemon already placed. 2023-11-16T16:52:16.239 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:12 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:52:16.239 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:12 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:52:16.239 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:14 smithi141 ceph-mon[131184]: pgmap v73: 129 pgs: 129 active+clean; 2.2 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-11-16T16:52:16.241 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:16.241 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:16.241 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.241 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.242 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 78 -lt 100 ']' 2023-11-16T16:52:16.242 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 78 + 1 2023-11-16T16:52:16.242 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=79 2023-11-16T16:52:16.242 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:16.242 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.242 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.243 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:16.243 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:16.243 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:16.243 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.243 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.243 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:10 smithi002 ceph-mon[128712]: pgmap v71: 129 pgs: 129 active+clean; 2.1 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 38 KiB/s wr, 19 op/s 2023-11-16T16:52:16.246 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:16.246 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:16.246 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.246 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.246 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 79 -lt 100 ']' 2023-11-16T16:52:16.247 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 79 + 1 2023-11-16T16:52:16.247 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=80 2023-11-16T16:52:16.247 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:16.247 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.247 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.247 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:16.247 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:16.247 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:16.247 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.247 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.247 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:16.247 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:16.247 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.248 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.248 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 80 -lt 100 ']' 2023-11-16T16:52:16.248 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 80 + 1 2023-11-16T16:52:16.248 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=81 2023-11-16T16:52:16.248 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:16.248 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.248 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.249 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:16.249 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:16.249 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:16.249 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.249 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.249 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:16.249 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:16.249 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.250 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.250 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 81 -lt 100 ']' 2023-11-16T16:52:16.250 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 81 + 1 2023-11-16T16:52:16.250 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=82 2023-11-16T16:52:16.250 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:16.250 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.250 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.250 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:16.250 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:16.250 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:16.250 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.251 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.277 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:12 smithi002 ceph-mon[128712]: pgmap v72: 129 pgs: 129 active+clean; 2.2 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 48 KiB/s wr, 24 op/s 2023-11-16T16:52:16.277 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:12 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:52:16.277 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:12 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:52:16.277 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:12 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:52:16.278 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:12 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:52:16.278 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:12 smithi002 ceph-mon[128712]: Service mds.all has no related daemon already placed. 2023-11-16T16:52:16.278 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:12 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:52:16.278 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:12 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:52:16.285 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:16.285 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:16.285 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.285 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.286 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 82 -lt 100 ']' 2023-11-16T16:52:16.286 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 82 + 1 2023-11-16T16:52:16.286 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=83 2023-11-16T16:52:16.286 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:16.286 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.286 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.287 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:16.287 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:16.287 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:16.287 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.287 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.287 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:16.287 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:16.288 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.288 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.288 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 83 -lt 100 ']' 2023-11-16T16:52:16.288 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 83 + 1 2023-11-16T16:52:16.288 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=84 2023-11-16T16:52:16.288 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:16.288 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.289 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.289 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:16.289 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:16.289 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:16.289 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.289 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.289 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:16.290 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:16.290 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.290 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.290 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 84 -lt 100 ']' 2023-11-16T16:52:16.290 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 84 + 1 2023-11-16T16:52:16.290 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=85 2023-11-16T16:52:16.290 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:16.291 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.291 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.291 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:16.291 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:16.291 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:16.291 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.291 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.292 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:16.292 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:16.292 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.292 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.292 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 85 -lt 100 ']' 2023-11-16T16:52:16.292 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 85 + 1 2023-11-16T16:52:16.292 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=86 2023-11-16T16:52:16.293 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:16.293 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:14 smithi002 ceph-mon[128712]: pgmap v73: 129 pgs: 129 active+clean; 2.2 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-11-16T16:52:16.297 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.297 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.297 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:16.297 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:16.297 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:16.297 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.298 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.298 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:16.298 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:16.298 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.298 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.298 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 86 -lt 100 ']' 2023-11-16T16:52:16.298 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 86 + 1 2023-11-16T16:52:16.298 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=87 2023-11-16T16:52:16.298 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:16.298 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.298 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.298 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:16.299 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:16.299 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:16.299 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.299 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.345 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:16.345 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:16.360 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.360 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:16.360 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 87 -lt 100 ']' 2023-11-16T16:52:16.361 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 87 + 1 2023-11-16T16:52:16.390 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=88 2023-11-16T16:52:16.390 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:16.415 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.415 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:16.418 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:16.418 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:16.419 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:16.849 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:16.850 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:17.003 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:17.003 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:17.018 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:17.018 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:17.018 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 88 -lt 100 ']' 2023-11-16T16:52:17.019 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 88 + 1 2023-11-16T16:52:17.048 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=89 2023-11-16T16:52:17.048 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:17.084 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:17.085 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:17.087 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:17.088 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:17.088 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:17.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:16 smithi002 ceph-mon[128712]: pgmap v74: 129 pgs: 129 active+clean; 2.2 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-11-16T16:52:17.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:16 smithi141 ceph-mon[131184]: pgmap v74: 129 pgs: 129 active+clean; 2.2 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-11-16T16:52:17.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:16 smithi155 ceph-mon[131607]: pgmap v74: 129 pgs: 129 active+clean; 2.2 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-11-16T16:52:17.433 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:17.433 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:17.561 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:17.561 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:17.577 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:17.577 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:17.578 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 89 -lt 100 ']' 2023-11-16T16:52:17.578 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 89 + 1 2023-11-16T16:52:17.606 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=90 2023-11-16T16:52:17.606 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:17.651 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:17.651 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:17.654 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:17.655 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:17.655 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:18.383 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:18.383 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:18.528 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:18.528 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:18.544 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:18.544 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:18.544 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 90 -lt 100 ']' 2023-11-16T16:52:18.544 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 90 + 1 2023-11-16T16:52:18.573 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=91 2023-11-16T16:52:18.573 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:18.628 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:18.628 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:18.631 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:18.631 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:18.632 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:18.948 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:18.949 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:19.086 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:19.086 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:19.101 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:19.102 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:19.102 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 91 -lt 100 ']' 2023-11-16T16:52:19.102 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 91 + 1 2023-11-16T16:52:19.131 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=92 2023-11-16T16:52:19.132 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:19.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:18 smithi002 ceph-mon[128712]: pgmap v75: 129 pgs: 129 active+clean; 2.3 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 23 KiB/s wr, 12 op/s 2023-11-16T16:52:19.165 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:19.166 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:19.168 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:19.169 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:19.169 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:18 smithi141 ceph-mon[131184]: pgmap v75: 129 pgs: 129 active+clean; 2.3 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 23 KiB/s wr, 12 op/s 2023-11-16T16:52:19.350 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:19.350 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:19.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:18 smithi155 ceph-mon[131607]: pgmap v75: 129 pgs: 129 active+clean; 2.3 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 23 KiB/s wr, 12 op/s 2023-11-16T16:52:19.427 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:19.427 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:19.443 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:19.443 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:19.443 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 92 -lt 100 ']' 2023-11-16T16:52:19.444 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 92 + 1 2023-11-16T16:52:19.473 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=93 2023-11-16T16:52:19.473 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:19.516 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:19.516 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:19.518 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:19.519 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:19.519 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:19.849 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:19.849 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:19.977 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:19.977 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:20.004 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:20.005 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:20.005 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 93 -lt 100 ']' 2023-11-16T16:52:20.005 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 93 + 1 2023-11-16T16:52:20.031 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=94 2023-11-16T16:52:20.031 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:20.055 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:20.055 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:20.057 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:20.058 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:20.058 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:20.416 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:20.416 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:20.519 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:20.519 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:20.534 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:20.534 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:20.534 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 94 -lt 100 ']' 2023-11-16T16:52:20.534 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 94 + 1 2023-11-16T16:52:20.564 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=95 2023-11-16T16:52:20.565 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:20.591 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:20.592 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:20.594 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:20.595 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:20.595 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:20.949 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:20.949 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:21.052 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:21.052 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:21.067 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:21.068 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:21.068 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 95 -lt 100 ']' 2023-11-16T16:52:21.068 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 95 + 1 2023-11-16T16:52:21.098 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=96 2023-11-16T16:52:21.098 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:21.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:20 smithi002 ceph-mon[128712]: pgmap v76: 129 pgs: 129 active+clean; 2.3 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-11-16T16:52:21.192 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:21.193 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:21.195 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:21.196 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:21.196 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:21.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:20 smithi141 ceph-mon[131184]: pgmap v76: 129 pgs: 129 active+clean; 2.3 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-11-16T16:52:21.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:20 smithi155 ceph-mon[131607]: pgmap v76: 129 pgs: 129 active+clean; 2.3 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-11-16T16:52:21.590 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:21.591 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:21.727 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:21.727 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:21.742 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:21.742 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:21.742 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 96 -lt 100 ']' 2023-11-16T16:52:21.743 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 96 + 1 2023-11-16T16:52:21.769 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=97 2023-11-16T16:52:21.769 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:21.842 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:21.842 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:21.845 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:21.846 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:21.846 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:22.490 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:22.491 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:22.752 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:22.752 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:22.776 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:22.776 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:22.776 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 97 -lt 100 ']' 2023-11-16T16:52:22.776 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 97 + 1 2023-11-16T16:52:22.822 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=98 2023-11-16T16:52:22.822 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:22.848 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:22.848 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:22.851 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:22.851 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:22.852 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:23.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:22 smithi002 ceph-mon[128712]: pgmap v77: 129 pgs: 129 active+clean; 2.4 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 11 op/s 2023-11-16T16:52:23.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:22 smithi141 ceph-mon[131184]: pgmap v77: 129 pgs: 129 active+clean; 2.4 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 11 op/s 2023-11-16T16:52:23.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:22 smithi155 ceph-mon[131607]: pgmap v77: 129 pgs: 129 active+clean; 2.4 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 11 op/s 2023-11-16T16:52:23.732 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:23.732 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:23.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.749+0000 7fb4503ff700 1 -- 172.21.15.2:0/1625379853 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb448105560 msgr2=0x7fb44810d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:23.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.749+0000 7fb4503ff700 1 --2- 172.21.15.2:0/1625379853 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb448105560 0x7fb44810d0f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb44000b0d0 tx=0x7fb44000b3e0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.749+0000 7fb4503ff700 1 -- 172.21.15.2:0/1625379853 shutdown_connections 2023-11-16T16:52:23.752 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.749+0000 7fb4503ff700 1 --2- 172.21.15.2:0/1625379853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb448100c50 0x7fb448101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.752 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.749+0000 7fb4503ff700 1 --2- 172.21.15.2:0/1625379853 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb448105560 0x7fb44810d0f0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.752 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.749+0000 7fb4503ff700 1 --2- 172.21.15.2:0/1625379853 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb4481015f0 0x7fb448104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.752 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.749+0000 7fb4503ff700 1 -- 172.21.15.2:0/1625379853 >> 172.21.15.2:0/1625379853 conn(0x7fb448077560 msgr2=0x7fb448077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:23.752 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.750+0000 7fb4503ff700 1 -- 172.21.15.2:0/1625379853 shutdown_connections 2023-11-16T16:52:23.752 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.750+0000 7fb4503ff700 1 -- 172.21.15.2:0/1625379853 wait complete. 2023-11-16T16:52:23.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.751+0000 7fb4503ff700 1 Processor -- start 2023-11-16T16:52:23.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.751+0000 7fb4503ff700 1 -- start start 2023-11-16T16:52:23.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.751+0000 7fb4503ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb448100c50 0x7fb448072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.751+0000 7fb4503ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4481015f0 0x7fb448073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.751+0000 7fb4503ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb448105560 0x7fb4481a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.751+0000 7fb4503ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb448073b60 con 0x7fb4481015f0 2023-11-16T16:52:23.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.751+0000 7fb4503ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4481a4be0 con 0x7fb448100c50 2023-11-16T16:52:23.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.751+0000 7fb4503ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4481a4d20 con 0x7fb448105560 2023-11-16T16:52:23.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.751+0000 7fb44e19b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb448100c50 0x7fb448072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.752+0000 7fb44d99a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4481015f0 0x7fb448073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.752+0000 7fb44e19b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb448100c50 0x7fb448072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:37864/0 (socket says 172.21.15.2:37864) 2023-11-16T16:52:23.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.752+0000 7fb44e19b700 1 -- 172.21.15.2:0/2584973746 learned_addr learned my addr 172.21.15.2:0/2584973746 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:23.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.752+0000 7fb44e99c700 1 --2- 172.21.15.2:0/2584973746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb448105560 0x7fb4481a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.752+0000 7fb44d99a700 1 -- 172.21.15.2:0/2584973746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb448105560 msgr2=0x7fb4481a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:23.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.752+0000 7fb44d99a700 1 --2- 172.21.15.2:0/2584973746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb448105560 0x7fb4481a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.752+0000 7fb44d99a700 1 -- 172.21.15.2:0/2584973746 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb448100c50 msgr2=0x7fb448072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:23.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.752+0000 7fb44d99a700 1 --2- 172.21.15.2:0/2584973746 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb448100c50 0x7fb448072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.752+0000 7fb44d99a700 1 -- 172.21.15.2:0/2584973746 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb438009520 con 0x7fb4481015f0 2023-11-16T16:52:23.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.752+0000 7fb44e19b700 1 --2- 172.21.15.2:0/2584973746 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb448100c50 0x7fb448072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:23.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.752+0000 7fb44d99a700 1 --2- 172.21.15.2:0/2584973746 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4481015f0 0x7fb448073330 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb4481019c0 tx=0x7fb43800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:23.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.752+0000 7fb43f7fe700 1 -- 172.21.15.2:0/2584973746 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb438004220 con 0x7fb4481015f0 2023-11-16T16:52:23.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.752+0000 7fb43f7fe700 1 -- 172.21.15.2:0/2584973746 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb438004380 con 0x7fb4481015f0 2023-11-16T16:52:23.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.753+0000 7fb4503ff700 1 -- 172.21.15.2:0/2584973746 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb440009d00 con 0x7fb4481015f0 2023-11-16T16:52:23.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.753+0000 7fb43f7fe700 1 -- 172.21.15.2:0/2584973746 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb438003c20 con 0x7fb4481015f0 2023-11-16T16:52:23.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.753+0000 7fb4503ff700 1 -- 172.21.15.2:0/2584973746 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb448110330 con 0x7fb4481015f0 2023-11-16T16:52:23.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.756+0000 7fb43f7fe700 1 -- 172.21.15.2:0/2584973746 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fb43800c7e0 con 0x7fb4481015f0 2023-11-16T16:52:23.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.757+0000 7fb43f7fe700 1 --2- 172.21.15.2:0/2584973746 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fb4340a1340 0x7fb4340a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.757+0000 7fb43f7fe700 1 -- 172.21.15.2:0/2584973746 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fb438014070 con 0x7fb4481015f0 2023-11-16T16:52:23.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.758+0000 7fb44e19b700 1 --2- 172.21.15.2:0/2584973746 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fb4340a1340 0x7fb4340a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.758+0000 7fb44e19b700 1 --2- 172.21.15.2:0/2584973746 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fb4340a1340 0x7fb4340a37f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb444011550 tx=0x7fb444009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.884+0000 7fb463cabb80 1 Processor -- start 2023-11-16T16:52:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.884+0000 7fb463cabb80 1 -- start start 2023-11-16T16:52:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.884+0000 7fb463cabb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe2a86b0 0x5631fe4521e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.884+0000 7fb463cabb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe452720 0x5631fe44c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.884+0000 7fb463cabb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452b90 0x5631fe44cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.884+0000 7fb463cabb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fe352000 con 0x5631fe452720 2023-11-16T16:52:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.884+0000 7fb463cabb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fe3a4840 con 0x5631fe2a86b0 2023-11-16T16:52:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.884+0000 7fb463cabb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fe39dcd0 con 0x5631fe452b90 2023-11-16T16:52:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.885+0000 7fb44d99a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe452720 0x5631fe44c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.885+0000 7fb44d99a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe452720 0x5631fe44c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34236/0 (socket says 172.21.15.2:34236) 2023-11-16T16:52:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.885+0000 7fb44d99a700 1 -- 172.21.15.2:0/2258686786 learned_addr learned my addr 172.21.15.2:0/2258686786 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.885+0000 7fb44e19b700 1 --2- 172.21.15.2:0/2258686786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe2a86b0 0x5631fe4521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.885+0000 7fb44e99c700 1 --2- 172.21.15.2:0/2258686786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452b90 0x5631fe44cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.885+0000 7fb44d99a700 1 -- 172.21.15.2:0/2258686786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452b90 msgr2=0x5631fe44cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.885+0000 7fb44d99a700 1 --2- 172.21.15.2:0/2258686786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452b90 0x5631fe44cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.885+0000 7fb44d99a700 1 -- 172.21.15.2:0/2258686786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe2a86b0 msgr2=0x5631fe4521e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.885+0000 7fb44d99a700 1 --2- 172.21.15.2:0/2258686786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe2a86b0 0x5631fe4521e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.885+0000 7fb44d99a700 1 -- 172.21.15.2:0/2258686786 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5631fe21f5d0 con 0x5631fe452720 2023-11-16T16:52:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.885+0000 7fb44d99a700 1 --2- 172.21.15.2:0/2258686786 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe452720 0x5631fe44c560 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb4380a1e30 tx=0x7fb438099e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8de55c8de2abc824 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.886+0000 7fb43cbfe700 1 -- 172.21.15.2:0/2258686786 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb438099f10 con 0x5631fe452720 2023-11-16T16:52:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.886+0000 7fb43cbfe700 1 -- 172.21.15.2:0/2258686786 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb43809df40 con 0x5631fe452720 2023-11-16T16:52:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.886+0000 7fb43cbfe700 1 -- 172.21.15.2:0/2258686786 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43809a170 con 0x5631fe452720 2023-11-16T16:52:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.887+0000 7fb463cabb80 1 -- 172.21.15.2:0/2258686786 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe452720 msgr2=0x5631fe44c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.887+0000 7fb463cabb80 1 --2- 172.21.15.2:0/2258686786 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe452720 0x5631fe44c560 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb4380a1e30 tx=0x7fb438099e10 comp rx=0 tx=0).stop 2023-11-16T16:52:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.887+0000 7fb463cabb80 1 -- 172.21.15.2:0/2258686786 shutdown_connections 2023-11-16T16:52:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.887+0000 7fb463cabb80 1 --2- 172.21.15.2:0/2258686786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452b90 0x5631fe44cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.887+0000 7fb463cabb80 1 --2- 172.21.15.2:0/2258686786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe2a86b0 0x5631fe4521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.887+0000 7fb463cabb80 1 --2- 172.21.15.2:0/2258686786 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe452720 0x5631fe44c560 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.887+0000 7fb463cabb80 1 -- 172.21.15.2:0/2258686786 >> 172.21.15.2:0/2258686786 conn(0x5631fe441f50 msgr2=0x5631fe447270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.887+0000 7fb463cabb80 1 -- 172.21.15.2:0/2258686786 shutdown_connections 2023-11-16T16:52:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.887+0000 7fb463cabb80 1 -- 172.21.15.2:0/2258686786 wait complete. 2023-11-16T16:52:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.887+0000 7fb463cabb80 1 -- start start 2023-11-16T16:52:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.888+0000 7fb463cabb80 1 Processor -- start 2023-11-16T16:52:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.888+0000 7fb463cabb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe2a86b0 0x5631fe45bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.888+0000 7fb463cabb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452720 0x5631fe45c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.888+0000 7fb463cabb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe452b90 0x5631fe455e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.888+0000 7fb463cabb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fe33f880 con 0x5631fe2a86b0 2023-11-16T16:52:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.888+0000 7fb463cabb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fe33f520 con 0x5631fe452b90 2023-11-16T16:52:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.888+0000 7fb463cabb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fe2218c0 con 0x5631fe452720 2023-11-16T16:52:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.888+0000 7fb463cabb80 10 client.0 _ll_register_callbacks cb 0x5631fe35d720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.888+0000 7fb44e19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe2a86b0 0x5631fe45bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.889+0000 7fb44e19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe2a86b0 0x5631fe45bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34238/0 (socket says 172.21.15.2:34238) 2023-11-16T16:52:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.889+0000 7fb44d99a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452720 0x5631fe45c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.889+0000 7fb44e19b700 1 -- 172.21.15.2:0/2495340397 learned_addr learned my addr 172.21.15.2:0/2495340397 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.889+0000 7fb44e99c700 1 --2- 172.21.15.2:0/2495340397 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe452b90 0x5631fe455e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.889+0000 7fb44d99a700 1 -- 172.21.15.2:0/2495340397 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe452b90 msgr2=0x5631fe455e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.889+0000 7fb44d99a700 1 --2- 172.21.15.2:0/2495340397 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe452b90 0x5631fe455e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.889+0000 7fb44d99a700 1 -- 172.21.15.2:0/2495340397 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe2a86b0 msgr2=0x5631fe45bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.889+0000 7fb44d99a700 1 --2- 172.21.15.2:0/2495340397 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe2a86b0 0x5631fe45bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.889+0000 7fb44d99a700 1 -- 172.21.15.2:0/2495340397 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb448110330 con 0x5631fe452720 2023-11-16T16:52:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.889+0000 7fb44d99a700 1 --2- 172.21.15.2:0/2495340397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452720 0x5631fe45c0c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb4380a1e30 tx=0x7fb4380ca530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e07efe2de0ca7a10 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.889+0000 7fb431ffb700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:52:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.889+0000 7fb431ffb700 1 -- 172.21.15.2:0/2495340397 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb438099f10 con 0x5631fe452720 2023-11-16T16:52:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.889+0000 7fb431ffb700 1 -- 172.21.15.2:0/2495340397 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fb438099610 con 0x5631fe452720 2023-11-16T16:52:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.889+0000 7fb463cabb80 1 -- 172.21.15.2:0/2495340397 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5631fe43f710 con 0x5631fe452720 2023-11-16T16:52:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.890+0000 7fb431ffb700 1 -- 172.21.15.2:0/2495340397 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4380ca250 con 0x5631fe452720 2023-11-16T16:52:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.890+0000 7fb463cabb80 10 client.25627 fetch_fsmap learned FSMap version 31 2023-11-16T16:52:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.890+0000 7fb463cabb80 1 -- 172.21.15.2:0/2495340397 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5631fe364130 con 0x5631fe452720 2023-11-16T16:52:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.890+0000 7fb431ffb700 1 -- 172.21.15.2:0/2495340397 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5631fe43f710 con 0x5631fe452720 2023-11-16T16:52:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.890+0000 7fb431ffb700 1 -- 172.21.15.2:0/2495340397 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 31) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fb43807c070 con 0x5631fe452720 2023-11-16T16:52:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.890+0000 7fb463cabb80 10 client.25627 fetch_fsmap finished waiting for FSMap version 31 2023-11-16T16:52:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.890+0000 7fb463cabb80 10 client.25627 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.890+0000 7fb463cabb80 20 client.25627 populate_metadata read hostname 'smithi002' 2023-11-16T16:52:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.890+0000 7fb463cabb80 1 --2- 172.21.15.2:0/2495340397 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5631fe465b00 0x5631fe467ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.890+0000 7fb463cabb80 4 client.25627 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.890+0000 7fb463cabb80 1 -- 172.21.15.2:0/2495340397 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5631fe360040 con 0x5631fe465b00 2023-11-16T16:52:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.891+0000 7fb44e19b700 1 --2- 172.21.15.2:0/2495340397 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5631fe465b00 0x5631fe467ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.891+0000 7fb431ffb700 10 client.25627 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:52:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.891+0000 7fb44e19b700 1 --2- 172.21.15.2:0/2495340397 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5631fe465b00 0x5631fe467ee0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=193e9ee3d768f574 server_cookie=c3fd8dc2e6d8db3f in_seq=0 out_seq=0 2023-11-16T16:52:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.892+0000 7fb431ffb700 1 -- 172.21.15.2:0/2495340397 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5631fe360040 con 0x5631fe465b00 2023-11-16T16:52:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.892+0000 7fb431ffb700 10 client.25627 handle_command_reply: tid=0 2023-11-16T16:52:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.892+0000 7fb463cabb80 1 client.25627 shutdown 2023-11-16T16:52:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.892+0000 7fb463cabb80 1 -- 172.21.15.2:0/2495340397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452720 msgr2=0x5631fe45c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.892+0000 7fb463cabb80 1 --2- 172.21.15.2:0/2495340397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452720 0x5631fe45c0c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb4380a1e30 tx=0x7fb4380ca530 comp rx=0 tx=0).stop 2023-11-16T16:52:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.893+0000 7fb463cabb80 1 -- 172.21.15.2:0/2495340397 shutdown_connections 2023-11-16T16:52:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.893+0000 7fb463cabb80 1 --2- 172.21.15.2:0/2495340397 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe452b90 0x5631fe455e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.893+0000 7fb463cabb80 1 --2- 172.21.15.2:0/2495340397 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe2a86b0 0x5631fe45bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.893+0000 7fb463cabb80 1 --2- 172.21.15.2:0/2495340397 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5631fe465b00 0x5631fe467ee0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.893+0000 7fb463cabb80 1 --2- 172.21.15.2:0/2495340397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452720 0x5631fe45c0c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.893+0000 7fb463cabb80 1 -- 172.21.15.2:0/2495340397 >> 172.21.15.2:0/2495340397 conn(0x5631fe441f50 msgr2=0x5631fe4467e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.893+0000 7fb431ffb700 0 client.25627 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:52:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.893+0000 7fb463cabb80 1 -- 172.21.15.2:0/2495340397 shutdown_connections 2023-11-16T16:52:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.893+0000 7fb463cabb80 1 -- 172.21.15.2:0/2495340397 wait complete. 2023-11-16T16:52:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.893+0000 7fb463cabb80 20 client.25627 trim_cache size 0 max 16384 2023-11-16T16:52:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.910+0000 7fb463cabb80 1 Processor -- start 2023-11-16T16:52:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.910+0000 7fb463cabb80 1 -- start start 2023-11-16T16:52:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.910+0000 7fb463cabb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe2a86b0 0x5631fe448a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.910+0000 7fb463cabb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe452720 0x5631fe456740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.910+0000 7fb463cabb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452b90 0x5631fe446f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.910+0000 7fb463cabb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fe450b70 con 0x5631fe2a86b0 2023-11-16T16:52:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.910+0000 7fb463cabb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fe462960 con 0x5631fe452720 2023-11-16T16:52:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.910+0000 7fb463cabb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fe4524a0 con 0x5631fe452b90 2023-11-16T16:52:23.916 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.910+0000 7fb44e19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe2a86b0 0x5631fe448a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.916 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.910+0000 7fb44e99c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452b90 0x5631fe446f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.916 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.911+0000 7fb44d99a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe452720 0x5631fe456740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.916 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.911+0000 7fb44e99c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452b90 0x5631fe446f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:60874/0 (socket says 172.21.15.2:60874) 2023-11-16T16:52:23.916 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.911+0000 7fb44e99c700 1 -- 172.21.15.2:0/2994990557 learned_addr learned my addr 172.21.15.2:0/2994990557 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:23.916 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.911+0000 7fb44e99c700 1 -- 172.21.15.2:0/2994990557 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe452720 msgr2=0x5631fe456740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.911+0000 7fb44e99c700 1 --2- 172.21.15.2:0/2994990557 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe452720 0x5631fe456740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.911+0000 7fb44e99c700 1 -- 172.21.15.2:0/2994990557 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe2a86b0 msgr2=0x5631fe448a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.911+0000 7fb44e99c700 1 --2- 172.21.15.2:0/2994990557 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe2a86b0 0x5631fe448a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.911+0000 7fb44e99c700 1 -- 172.21.15.2:0/2994990557 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5631fe4436f0 con 0x5631fe452b90 2023-11-16T16:52:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.911+0000 7fb44e99c700 1 --2- 172.21.15.2:0/2994990557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452b90 0x5631fe446f60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb44000b0d0 tx=0x7fb440009390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f20f93d37129c011 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.912+0000 7fb43d3ff700 1 -- 172.21.15.2:0/2994990557 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb440021070 con 0x5631fe452b90 2023-11-16T16:52:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.912+0000 7fb43d3ff700 1 -- 172.21.15.2:0/2994990557 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb440009480 con 0x5631fe452b90 2023-11-16T16:52:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.912+0000 7fb43d3ff700 1 -- 172.21.15.2:0/2994990557 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44000e040 con 0x5631fe452b90 2023-11-16T16:52:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.912+0000 7fb463cabb80 1 -- 172.21.15.2:0/2994990557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452b90 msgr2=0x5631fe446f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.912+0000 7fb463cabb80 1 --2- 172.21.15.2:0/2994990557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452b90 0x5631fe446f60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb44000b0d0 tx=0x7fb440009390 comp rx=0 tx=0).stop 2023-11-16T16:52:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.912+0000 7fb463cabb80 1 -- 172.21.15.2:0/2994990557 shutdown_connections 2023-11-16T16:52:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.912+0000 7fb463cabb80 1 --2- 172.21.15.2:0/2994990557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452b90 0x5631fe446f60 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.912+0000 7fb463cabb80 1 --2- 172.21.15.2:0/2994990557 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe2a86b0 0x5631fe448a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.912+0000 7fb463cabb80 1 --2- 172.21.15.2:0/2994990557 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe452720 0x5631fe456740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.912+0000 7fb463cabb80 1 -- 172.21.15.2:0/2994990557 >> 172.21.15.2:0/2994990557 conn(0x5631fe441f50 msgr2=0x5631fe453820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:23.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.912+0000 7fb463cabb80 1 -- 172.21.15.2:0/2994990557 shutdown_connections 2023-11-16T16:52:23.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.913+0000 7fb463cabb80 1 -- 172.21.15.2:0/2994990557 wait complete. 2023-11-16T16:52:23.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.913+0000 7fb463cabb80 1 -- start start 2023-11-16T16:52:23.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.913+0000 7fb463cabb80 1 Processor -- start 2023-11-16T16:52:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.914+0000 7fb463cabb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe2a86b0 0x5631fe4ada40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.914+0000 7fb463cabb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452720 0x5631fe4adf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.914+0000 7fb463cabb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe452b90 0x5631fe4b88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.914+0000 7fb463cabb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fe4ae710 con 0x5631fe452b90 2023-11-16T16:52:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.914+0000 7fb463cabb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fe4b8e10 con 0x5631fe2a86b0 2023-11-16T16:52:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.914+0000 7fb463cabb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fe4b8fd0 con 0x5631fe452720 2023-11-16T16:52:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.914+0000 7fb463cabb80 10 client.0 _ll_register_callbacks cb 0x7fb444009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.914+0000 7fb44e99c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe452b90 0x5631fe4b88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.914+0000 7fb44e99c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe452b90 0x5631fe4b88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34250/0 (socket says 172.21.15.2:34250) 2023-11-16T16:52:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.914+0000 7fb44e99c700 1 -- 172.21.15.2:0/605281980 learned_addr learned my addr 172.21.15.2:0/605281980 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.914+0000 7fb44d99a700 1 --2- 172.21.15.2:0/605281980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452720 0x5631fe4adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.914+0000 7fb44e19b700 1 --2- 172.21.15.2:0/605281980 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe2a86b0 0x5631fe4ada40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.914+0000 7fb44e99c700 1 -- 172.21.15.2:0/605281980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452720 msgr2=0x5631fe4adf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.914+0000 7fb44e99c700 1 --2- 172.21.15.2:0/605281980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452720 0x5631fe4adf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.914+0000 7fb44e99c700 1 -- 172.21.15.2:0/605281980 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe2a86b0 msgr2=0x5631fe4ada40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.914+0000 7fb44e99c700 1 --2- 172.21.15.2:0/605281980 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe2a86b0 0x5631fe4ada40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.914+0000 7fb44e99c700 1 -- 172.21.15.2:0/605281980 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb44810d900 con 0x5631fe452b90 2023-11-16T16:52:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.915+0000 7fb44e99c700 1 --2- 172.21.15.2:0/605281980 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe452b90 0x5631fe4b88d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fb440006250 tx=0x7fb44000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=828aa572aec0c2b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.915+0000 7fb431ffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:52:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.915+0000 7fb431ffb700 1 -- 172.21.15.2:0/605281980 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44000e070 con 0x5631fe452b90 2023-11-16T16:52:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.915+0000 7fb431ffb700 1 -- 172.21.15.2:0/605281980 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fb440003c10 con 0x5631fe452b90 2023-11-16T16:52:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.915+0000 7fb463cabb80 1 -- 172.21.15.2:0/605281980 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb43809df40 con 0x5631fe452b90 2023-11-16T16:52:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.915+0000 7fb431ffb700 1 -- 172.21.15.2:0/605281980 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb440021040 con 0x5631fe452b90 2023-11-16T16:52:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.915+0000 7fb463cabb80 10 client.15627 fetch_fsmap learned FSMap version 31 2023-11-16T16:52:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.915+0000 7fb463cabb80 1 -- 172.21.15.2:0/605281980 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5631fe1cbd80 con 0x5631fe452b90 2023-11-16T16:52:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.915+0000 7fb431ffb700 1 -- 172.21.15.2:0/605281980 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb43809df40 con 0x5631fe452b90 2023-11-16T16:52:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.916+0000 7fb431ffb700 1 -- 172.21.15.2:0/605281980 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 31) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fb44002d020 con 0x5631fe452b90 2023-11-16T16:52:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.916+0000 7fb463cabb80 10 client.15627 fetch_fsmap finished waiting for FSMap version 31 2023-11-16T16:52:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.916+0000 7fb463cabb80 10 client.15627 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.916+0000 7fb463cabb80 20 client.15627 populate_metadata read hostname 'smithi002' 2023-11-16T16:52:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.916+0000 7fb463cabb80 1 --2- 172.21.15.2:0/605281980 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5631fe4ba500 0x5631fe4bc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.916+0000 7fb463cabb80 4 client.15627 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:52:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.916+0000 7fb463cabb80 1 -- 172.21.15.2:0/605281980 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5631fe36c600 con 0x5631fe4ba500 2023-11-16T16:52:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.916+0000 7fb44e19b700 1 --2- 172.21.15.2:0/605281980 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5631fe4ba500 0x5631fe4bc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.917+0000 7fb44e19b700 1 --2- 172.21.15.2:0/605281980 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5631fe4ba500 0x5631fe4bc8e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2486894ce7267e60 server_cookie=3ae31f9a3ed3f1dd in_seq=0 out_seq=0 2023-11-16T16:52:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.917+0000 7fb431ffb700 10 client.15627 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:52:23.931 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:23.932 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:23.933 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.924+0000 7fb431ffb700 1 -- 172.21.15.2:0/605281980 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5631fe36c600 con 0x5631fe4ba500 2023-11-16T16:52:23.933 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.924+0000 7fb431ffb700 10 client.15627 handle_command_reply: tid=0 2023-11-16T16:52:23.933 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.924+0000 7fb463cabb80 1 client.15627 shutdown 2023-11-16T16:52:23.933 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.924+0000 7fb463cabb80 1 -- 172.21.15.2:0/605281980 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe452b90 msgr2=0x5631fe4b88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:23.933 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.924+0000 7fb463cabb80 1 --2- 172.21.15.2:0/605281980 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe452b90 0x5631fe4b88d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fb440006250 tx=0x7fb44000bd50 comp rx=0 tx=0).stop 2023-11-16T16:52:23.933 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.924+0000 7fb463cabb80 1 -- 172.21.15.2:0/605281980 shutdown_connections 2023-11-16T16:52:23.933 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.924+0000 7fb463cabb80 1 --2- 172.21.15.2:0/605281980 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5631fe452b90 0x5631fe4b88d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.934 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.924+0000 7fb463cabb80 1 --2- 172.21.15.2:0/605281980 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5631fe2a86b0 0x5631fe4ada40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.934 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.924+0000 7fb463cabb80 1 --2- 172.21.15.2:0/605281980 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5631fe4ba500 0x5631fe4bc8e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.934 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.924+0000 7fb463cabb80 1 --2- 172.21.15.2:0/605281980 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5631fe452720 0x5631fe4adf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.934 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.924+0000 7fb463cabb80 1 -- 172.21.15.2:0/605281980 >> 172.21.15.2:0/605281980 conn(0x5631fe441f50 msgr2=0x5631fe451790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:23.934 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.924+0000 7fb431ffb700 0 client.15627 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:52:23.934 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.924+0000 7fb463cabb80 1 -- 172.21.15.2:0/605281980 shutdown_connections 2023-11-16T16:52:23.934 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.924+0000 7fb463cabb80 1 -- 172.21.15.2:0/605281980 wait complete. 2023-11-16T16:52:23.935 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.925+0000 7fb463cabb80 20 client.15627 trim_cache size 0 max 16384 2023-11-16T16:52:23.935 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:52:23.935 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:52:23.935 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:52:23.935 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:52:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.925+0000 7fb4503ff700 1 -- 172.21.15.2:0/2584973746 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fb4340a1340 msgr2=0x7fb4340a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.925+0000 7fb4503ff700 1 --2- 172.21.15.2:0/2584973746 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fb4340a1340 0x7fb4340a37f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb444011550 tx=0x7fb444009260 comp rx=0 tx=0).stop 2023-11-16T16:52:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.925+0000 7fb4503ff700 1 -- 172.21.15.2:0/2584973746 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4481015f0 msgr2=0x7fb448073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.925+0000 7fb4503ff700 1 --2- 172.21.15.2:0/2584973746 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4481015f0 0x7fb448073330 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb4481019c0 tx=0x7fb43800eca0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.925+0000 7fb4503ff700 1 -- 172.21.15.2:0/2584973746 shutdown_connections 2023-11-16T16:52:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.925+0000 7fb4503ff700 1 --2- 172.21.15.2:0/2584973746 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fb4340a1340 0x7fb4340a37f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.925+0000 7fb4503ff700 1 --2- 172.21.15.2:0/2584973746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb448105560 0x7fb4481a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.925+0000 7fb4503ff700 1 --2- 172.21.15.2:0/2584973746 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb448100c50 0x7fb448072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.925+0000 7fb4503ff700 1 --2- 172.21.15.2:0/2584973746 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4481015f0 0x7fb448073330 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.925+0000 7fb4503ff700 1 -- 172.21.15.2:0/2584973746 >> 172.21.15.2:0/2584973746 conn(0x7fb448077560 msgr2=0x7fb4480fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.926+0000 7fb4503ff700 1 -- 172.21.15.2:0/2584973746 shutdown_connections 2023-11-16T16:52:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:23.926+0000 7fb4503ff700 1 -- 172.21.15.2:0/2584973746 wait complete. 2023-11-16T16:52:23.943 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:23.943 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:23.943 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 98 -lt 100 ']' 2023-11-16T16:52:23.944 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 98 + 1 2023-11-16T16:52:23.969 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=99 2023-11-16T16:52:23.969 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:23.990 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:23.991 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:23.993 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:23.994 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:23.994 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:24.399 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:24.399 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:24.493 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:24.493 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:24.509 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:24.510 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:24.510 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 99 -lt 100 ']' 2023-11-16T16:52:24.510 INFO:tasks.workunit.client.0.smithi002.stderr:++ expr 99 + 1 2023-11-16T16:52:24.539 INFO:tasks.workunit.client.0.smithi002.stderr:+ c=100 2023-11-16T16:52:24.539 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:24.566 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:24.566 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:24.569 INFO:tasks.workunit.client.0.smithi002.stderr:+ c1=2 2023-11-16T16:52:24.569 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo 3 2023-11-16T16:52:24.570 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-11-16T16:52:24.940 INFO:tasks.workunit.client.0.smithi002.stderr:++ getfacl d1 2023-11-16T16:52:24.940 INFO:tasks.workunit.client.0.smithi002.stderr:++ grep -c nobody:rw 2023-11-16T16:52:25.068 INFO:tasks.workunit.client.0.smithi002.stderr:+ c2=2 2023-11-16T16:52:25.068 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:25.085 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:25.085 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 2 -ne 2 ']' 2023-11-16T16:52:25.085 INFO:tasks.workunit.client.0.smithi002.stderr:+ '[' 100 -lt 100 ']' 2023-11-16T16:52:25.092 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir d1 2023-11-16T16:52:25.113 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n system.posix_acl_access -v 0x02000000 d1 2023-11-16T16:52:25.121 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n system.posix_acl_default -v 0x02000000 . 2023-11-16T16:52:25.125 INFO:tasks.workunit.client.0.smithi002.stderr:+ expect_failure getfattr -n system.posix_acl_access d1 2023-11-16T16:52:25.125 INFO:tasks.workunit.client.0.smithi002.stderr:+ getfattr -n system.posix_acl_access d1 2023-11-16T16:52:25.130 INFO:tasks.workunit.client.0.smithi002.stderr:d1: system.posix_acl_access: No such attribute 2023-11-16T16:52:25.131 INFO:tasks.workunit.client.0.smithi002.stderr:+ return 0 2023-11-16T16:52:25.131 INFO:tasks.workunit.client.0.smithi002.stderr:+ expect_failure getfattr -n system.posix_acl_default . 2023-11-16T16:52:25.131 INFO:tasks.workunit.client.0.smithi002.stderr:+ getfattr -n system.posix_acl_default . 2023-11-16T16:52:25.133 INFO:tasks.workunit.client.0.smithi002.stderr:.: system.posix_acl_default: No such attribute 2023-11-16T16:52:25.133 INFO:tasks.workunit.client.0.smithi002.stderr:+ return 0 2023-11-16T16:52:25.133 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir d1 2023-11-16T16:52:25.138 INFO:tasks.workunit.client.0.smithi002.stderr:+ cd .. 2023-11-16T16:52:25.138 INFO:tasks.workunit.client.0.smithi002.stderr:+ rmdir testdir 2023-11-16T16:52:25.154 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo OK 2023-11-16T16:52:25.155 INFO:tasks.workunit.client.0.smithi002.stdout:OK 2023-11-16T16:52:25.156 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T16:52:25.156 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T16:52:25.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:24 smithi002 ceph-mon[128712]: pgmap v78: 129 pgs: 129 active+clean; 2.4 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-11-16T16:52:25.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:24 smithi141 ceph-mon[131184]: pgmap v78: 129 pgs: 129 active+clean; 2.4 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-11-16T16:52:25.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:24 smithi155 ceph-mon[131607]: pgmap v78: 129 pgs: 129 active+clean; 2.4 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-11-16T16:52:25.522 INFO:tasks.workunit:Running workunit fs/misc/chmod.sh... 2023-11-16T16:52:25.522 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/chmod.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/chmod.sh 2023-11-16T16:52:25.627 INFO:tasks.workunit.client.0.smithi002.stderr:+ set -e 2023-11-16T16:52:25.627 INFO:tasks.workunit.client.0.smithi002.stderr:+ file=test_chmod.183213 2023-11-16T16:52:25.627 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo foo 2023-11-16T16:52:25.632 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2023-11-16T16:52:25.633 INFO:tasks.workunit.client.0.smithi002.stderr:+ chmod 400 test_chmod.183213 2023-11-16T16:52:25.637 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2023-11-16T16:52:25.637 INFO:tasks.workunit.client.0.smithi002.stderr:+ check_perms test_chmod.183213 -r-------- 2023-11-16T16:52:25.637 INFO:tasks.workunit.client.0.smithi002.stderr:+ file=test_chmod.183213 2023-11-16T16:52:25.637 INFO:tasks.workunit.client.0.smithi002.stderr:++ ls -la test_chmod.183213 2023-11-16T16:52:25.642 INFO:tasks.workunit.client.0.smithi002.stderr:+ r='-r--------. 1 ubuntu ubuntu 4 Nov 16 16:52 test_chmod.183213' 2023-11-16T16:52:25.642 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2023-11-16T16:52:25.643 INFO:tasks.workunit.client.0.smithi002.stderr:+ perms=-r-------- 2023-11-16T16:52:25.644 INFO:tasks.workunit.client.0.smithi002.stderr:++ echo -r--------. 1 ubuntu ubuntu 4 Nov 16 16:52 test_chmod.183213 2023-11-16T16:52:25.644 INFO:tasks.workunit.client.0.smithi002.stderr:++ awk '{print $1}' 2023-11-16T16:52:25.771 INFO:tasks.workunit.client.0.smithi002.stderr:+ test -r-------- '!=' -r--------. 2023-11-16T16:52:25.772 INFO:tasks.workunit.client.0.smithi002.stderr:++ echo -r--------. 1 ubuntu ubuntu 4 Nov 16 16:52 test_chmod.183213 2023-11-16T16:52:25.773 INFO:tasks.workunit.client.0.smithi002.stderr:++ awk '{print $1}' 2023-11-16T16:52:25.777 INFO:tasks.workunit.client.0.smithi002.stderr:+ test -r--------. '!=' -r--------. 2023-11-16T16:52:25.777 INFO:tasks.workunit.client.0.smithi002.stderr:+ set +e 2023-11-16T16:52:25.778 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo bar 2023-11-16T16:52:25.778 INFO:tasks.workunit.client.0.smithi002.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/chmod.sh: line 40: test_chmod.183213: Permission denied 2023-11-16T16:52:25.778 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 1 = 0 2023-11-16T16:52:25.778 INFO:tasks.workunit.client.0.smithi002.stderr:+ set -e 2023-11-16T16:52:25.779 INFO:tasks.workunit.client.0.smithi002.stderr:+ chmod 600 test_chmod.183213 2023-11-16T16:52:25.780 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo bar 2023-11-16T16:52:25.780 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2023-11-16T16:52:25.781 INFO:tasks.workunit.client.0.smithi002.stderr:+ check_perms test_chmod.183213 -rw------- 2023-11-16T16:52:25.781 INFO:tasks.workunit.client.0.smithi002.stderr:+ file=test_chmod.183213 2023-11-16T16:52:25.781 INFO:tasks.workunit.client.0.smithi002.stderr:++ ls -la test_chmod.183213 2023-11-16T16:52:25.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ r='-rw-------. 1 ubuntu ubuntu 8 Nov 16 16:52 test_chmod.183213' 2023-11-16T16:52:25.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2023-11-16T16:52:25.783 INFO:tasks.workunit.client.0.smithi002.stderr:+ perms=-rw------- 2023-11-16T16:52:25.784 INFO:tasks.workunit.client.0.smithi002.stderr:++ echo -rw-------. 1 ubuntu ubuntu 8 Nov 16 16:52 test_chmod.183213 2023-11-16T16:52:25.784 INFO:tasks.workunit.client.0.smithi002.stderr:++ awk '{print $1}' 2023-11-16T16:52:25.788 INFO:tasks.workunit.client.0.smithi002.stderr:+ test -rw------- '!=' -rw-------. 2023-11-16T16:52:25.789 INFO:tasks.workunit.client.0.smithi002.stderr:++ echo -rw-------. 1 ubuntu ubuntu 8 Nov 16 16:52 test_chmod.183213 2023-11-16T16:52:25.790 INFO:tasks.workunit.client.0.smithi002.stderr:++ awk '{print $1}' 2023-11-16T16:52:25.794 INFO:tasks.workunit.client.0.smithi002.stderr:+ test -rw-------. '!=' -rw-------. 2023-11-16T16:52:25.794 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo foo 2023-11-16T16:52:25.794 INFO:tasks.workunit.client.0.smithi002.stderr:+ test 0 '!=' 0 2023-11-16T16:52:25.795 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T16:52:25.795 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T16:52:25.884 INFO:tasks.workunit:Running workunit fs/misc/dac_override.sh... 2023-11-16T16:52:25.884 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/dac_override.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dac_override.sh 2023-11-16T16:52:25.942 INFO:tasks.workunit.client.0.smithi002.stderr:+ set -e 2023-11-16T16:52:25.943 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir -p testdir 2023-11-16T16:52:25.947 INFO:tasks.workunit.client.0.smithi002.stderr:+ file=test_chmod.183272 2023-11-16T16:52:25.947 INFO:tasks.workunit.client.0.smithi002.stderr:+ echo foo 2023-11-16T16:52:25.948 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo chmod 600 testdir 2023-11-16T16:52:25.983 INFO:tasks.workunit.client.0.smithi002.stderr:+ expect_failure cat testdir/test_chmod.183272 2023-11-16T16:52:25.984 INFO:tasks.workunit.client.0.smithi002.stderr:+ cat testdir/test_chmod.183272 2023-11-16T16:52:25.986 INFO:tasks.workunit.client.0.smithi002.stderr:cat: testdir/test_chmod.183272: Permission denied 2023-11-16T16:52:25.987 INFO:tasks.workunit.client.0.smithi002.stderr:+ return 0 2023-11-16T16:52:25.987 INFO:tasks.workunit.client.0.smithi002.stderr:+ sudo cat testdir/test_chmod.183272 2023-11-16T16:52:26.012 INFO:tasks.workunit.client.0.smithi002.stdout:foo 2023-11-16T16:52:26.016 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T16:52:26.016 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T16:52:26.112 INFO:tasks.workunit:Running workunit fs/misc/direct_io.py... 2023-11-16T16:52:26.112 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/direct_io.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/direct_io.py 2023-11-16T16:52:26.932 INFO:tasks.workunit.client.0.smithi002.stdout:ok 2023-11-16T16:52:26.938 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T16:52:26.938 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T16:52:27.024 INFO:tasks.workunit:Running workunit fs/misc/dirfrag.sh... 2023-11-16T16:52:27.024 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/dirfrag.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dirfrag.sh 2023-11-16T16:52:27.102 INFO:tasks.workunit.client.0.smithi002.stdout:creating folder hierarchy 2023-11-16T16:52:27.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:26 smithi002 ceph-mon[128712]: pgmap v79: 129 pgs: 129 active+clean; 2.4 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-11-16T16:52:27.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:26 smithi141 ceph-mon[131184]: pgmap v79: 129 pgs: 129 active+clean; 2.4 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-11-16T16:52:27.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:26 smithi155 ceph-mon[131607]: pgmap v79: 129 pgs: 129 active+clean; 2.4 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-11-16T16:52:29.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:28 smithi002 ceph-mon[128712]: pgmap v80: 129 pgs: 129 active+clean; 2.6 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 12 op/s 2023-11-16T16:52:29.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:28 smithi141 ceph-mon[131184]: pgmap v80: 129 pgs: 129 active+clean; 2.6 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 12 op/s 2023-11-16T16:52:29.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:28 smithi155 ceph-mon[131607]: pgmap v80: 129 pgs: 129 active+clean; 2.6 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 12 op/s 2023-11-16T16:52:31.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:30 smithi002 ceph-mon[128712]: pgmap v81: 129 pgs: 129 active+clean; 2.6 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-11-16T16:52:31.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:30 smithi141 ceph-mon[131184]: pgmap v81: 129 pgs: 129 active+clean; 2.6 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-11-16T16:52:31.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:30 smithi155 ceph-mon[131607]: pgmap v81: 129 pgs: 129 active+clean; 2.6 MiB data, 88 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-11-16T16:52:33.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:32 smithi141 ceph-mon[131184]: pgmap v82: 129 pgs: 129 active+clean; 2.7 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 KiB/s wr, 13 op/s 2023-11-16T16:52:33.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:32 smithi155 ceph-mon[131607]: pgmap v82: 129 pgs: 129 active+clean; 2.7 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 KiB/s wr, 13 op/s 2023-11-16T16:52:33.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:32 smithi002 ceph-mon[128712]: pgmap v82: 129 pgs: 129 active+clean; 2.7 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 KiB/s wr, 13 op/s 2023-11-16T16:52:35.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:34 smithi141 ceph-mon[131184]: pgmap v83: 129 pgs: 129 active+clean; 2.7 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-11-16T16:52:35.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:34 smithi155 ceph-mon[131607]: pgmap v83: 129 pgs: 129 active+clean; 2.7 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-11-16T16:52:35.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:34 smithi002 ceph-mon[128712]: pgmap v83: 129 pgs: 129 active+clean; 2.7 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-11-16T16:52:37.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:36 smithi141 ceph-mon[131184]: pgmap v84: 129 pgs: 129 active+clean; 2.7 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-11-16T16:52:37.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:36 smithi155 ceph-mon[131607]: pgmap v84: 129 pgs: 129 active+clean; 2.7 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-11-16T16:52:37.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:36 smithi002 ceph-mon[128712]: pgmap v84: 129 pgs: 129 active+clean; 2.7 MiB data, 89 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-11-16T16:52:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.449+0000 7f7b6ffff700 1 -- 172.21.15.2:0/2297597055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b680ff230 msgr2=0x7f7b680ff600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.449+0000 7f7b6ffff700 1 --2- 172.21.15.2:0/2297597055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b680ff230 0x7f7b680ff600 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7b640099c0 tx=0x7f7b64009cd0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.449+0000 7f7b6ffff700 1 -- 172.21.15.2:0/2297597055 shutdown_connections 2023-11-16T16:52:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.449+0000 7f7b6ffff700 1 --2- 172.21.15.2:0/2297597055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b680ff230 0x7f7b680ff600 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.449+0000 7f7b6ffff700 1 --2- 172.21.15.2:0/2297597055 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b68104630 0x7f7b68104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.449+0000 7f7b6ffff700 1 --2- 172.21.15.2:0/2297597055 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b680ffb40 0x7f7b68104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.449+0000 7f7b6ffff700 1 -- 172.21.15.2:0/2297597055 >> 172.21.15.2:0/2297597055 conn(0x7f7b680763a0 msgr2=0x7f7b680767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.450+0000 7f7b6ffff700 1 -- 172.21.15.2:0/2297597055 shutdown_connections 2023-11-16T16:52:37.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.450+0000 7f7b6ffff700 1 -- 172.21.15.2:0/2297597055 wait complete. 2023-11-16T16:52:37.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.452+0000 7f7b6ffff700 1 Processor -- start 2023-11-16T16:52:37.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.452+0000 7f7b6ffff700 1 -- start start 2023-11-16T16:52:37.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.453+0000 7f7b6ffff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b680ff230 0x7f7b6806fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:37.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.453+0000 7f7b6ffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b680ffb40 0x7f7b680701a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:37.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.453+0000 7f7b6ffff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b68104630 0x7f7b6806ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:37.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.453+0000 7f7b6ffff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6806b420 con 0x7f7b680ff230 2023-11-16T16:52:37.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.453+0000 7f7b6ffff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6806b590 con 0x7f7b68104630 2023-11-16T16:52:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.453+0000 7f7b6ffff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6806b750 con 0x7f7b680ffb40 2023-11-16T16:52:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.453+0000 7f7b6d59a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b680ffb40 0x7f7b680701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.453+0000 7f7b6e59c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b68104630 0x7f7b6806ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.453+0000 7f7b6e59c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b68104630 0x7f7b6806ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:60802/0 (socket says 172.21.15.2:60802) 2023-11-16T16:52:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.453+0000 7f7b6d59a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b680ffb40 0x7f7b680701a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:49620/0 (socket says 172.21.15.2:49620) 2023-11-16T16:52:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.453+0000 7f7b6e59c700 1 -- 172.21.15.2:0/991238202 learned_addr learned my addr 172.21.15.2:0/991238202 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.454+0000 7f7b6dd9b700 1 --2- 172.21.15.2:0/991238202 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b680ff230 0x7f7b6806fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.454+0000 7f7b6d59a700 1 -- 172.21.15.2:0/991238202 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b68104630 msgr2=0x7f7b6806ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.454+0000 7f7b6d59a700 1 --2- 172.21.15.2:0/991238202 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b68104630 0x7f7b6806ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.454+0000 7f7b6d59a700 1 -- 172.21.15.2:0/991238202 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b680ff230 msgr2=0x7f7b6806fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.454+0000 7f7b6d59a700 1 --2- 172.21.15.2:0/991238202 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b680ff230 0x7f7b6806fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.454+0000 7f7b6d59a700 1 -- 172.21.15.2:0/991238202 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b58009520 con 0x7f7b680ffb40 2023-11-16T16:52:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.454+0000 7f7b6d59a700 1 --2- 172.21.15.2:0/991238202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b680ffb40 0x7f7b680701a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800e9f0 tx=0x7f7b5800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.455+0000 7f7b5effd700 1 -- 172.21.15.2:0/991238202 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5800cb70 con 0x7f7b680ffb40 2023-11-16T16:52:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.455+0000 7f7b5effd700 1 -- 172.21.15.2:0/991238202 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b5800ee90 con 0x7f7b680ffb40 2023-11-16T16:52:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.455+0000 7f7b5effd700 1 -- 172.21.15.2:0/991238202 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b58003c80 con 0x7f7b680ffb40 2023-11-16T16:52:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.456+0000 7f7b6ffff700 1 -- 172.21.15.2:0/991238202 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b640095f0 con 0x7f7b680ffb40 2023-11-16T16:52:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.456+0000 7f7b6ffff700 1 -- 172.21.15.2:0/991238202 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b681168c0 con 0x7f7b680ffb40 2023-11-16T16:52:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.461+0000 7f7b5effd700 1 -- 172.21.15.2:0/991238202 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f7b58003de0 con 0x7f7b680ffb40 2023-11-16T16:52:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.463+0000 7f7b5effd700 1 --2- 172.21.15.2:0/991238202 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f7b540a1410 0x7f7b540a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.463+0000 7f7b5effd700 1 -- 172.21.15.2:0/991238202 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f7b58014070 con 0x7f7b680ffb40 2023-11-16T16:52:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.464+0000 7f7b6dd9b700 1 --2- 172.21.15.2:0/991238202 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f7b540a1410 0x7f7b540a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.464+0000 7f7b6dd9b700 1 --2- 172.21.15.2:0/991238202 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f7b540a1410 0x7f7b540a38c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7b64000c00 tx=0x7f7b64019040 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:37.518 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.516+0000 7f7b83938b80 1 Processor -- start 2023-11-16T16:52:37.519 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.517+0000 7f7b83938b80 1 -- start start 2023-11-16T16:52:37.519 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.517+0000 7f7b83938b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563199ef86b0 0x56319a0a21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:37.519 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.517+0000 7f7b83938b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2720 0x56319a09c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:37.519 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.517+0000 7f7b83938b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56319a0a2b90 0x56319a09cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:37.519 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.517+0000 7f7b83938b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x563199fa2000 con 0x56319a0a2720 2023-11-16T16:52:37.519 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.517+0000 7f7b83938b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x563199ff4840 con 0x563199ef86b0 2023-11-16T16:52:37.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.517+0000 7f7b83938b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x563199fedcd0 con 0x56319a0a2b90 2023-11-16T16:52:37.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.518+0000 7f7b6dd9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563199ef86b0 0x56319a0a21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.518+0000 7f7b6dd9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563199ef86b0 0x56319a0a21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:60812/0 (socket says 172.21.15.2:60812) 2023-11-16T16:52:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.518+0000 7f7b6dd9b700 1 -- 172.21.15.2:0/690147804 learned_addr learned my addr 172.21.15.2:0/690147804 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.518+0000 7f7b6e59c700 1 --2- 172.21.15.2:0/690147804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56319a0a2b90 0x56319a09cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.518+0000 7f7b6d59a700 1 --2- 172.21.15.2:0/690147804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2720 0x56319a09c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.518+0000 7f7b6dd9b700 1 -- 172.21.15.2:0/690147804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56319a0a2b90 msgr2=0x56319a09cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.518+0000 7f7b6dd9b700 1 --2- 172.21.15.2:0/690147804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56319a0a2b90 0x56319a09cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.518+0000 7f7b6dd9b700 1 -- 172.21.15.2:0/690147804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2720 msgr2=0x56319a09c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.518+0000 7f7b6dd9b700 1 --2- 172.21.15.2:0/690147804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2720 0x56319a09c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.518+0000 7f7b6dd9b700 1 -- 172.21.15.2:0/690147804 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563199e6f5d0 con 0x563199ef86b0 2023-11-16T16:52:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.519+0000 7f7b6d59a700 1 --2- 172.21.15.2:0/690147804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2720 0x56319a09c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:52:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.519+0000 7f7b6dd9b700 1 --2- 172.21.15.2:0/690147804 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563199ef86b0 0x56319a0a21e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7b640099c0 tx=0x7f7b6401f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4deeb7222eb8ac5d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:37.522 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.519+0000 7f7b53fff700 1 -- 172.21.15.2:0/690147804 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6400f8c0 con 0x563199ef86b0 2023-11-16T16:52:37.522 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.519+0000 7f7b53fff700 1 -- 172.21.15.2:0/690147804 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b6401d040 con 0x563199ef86b0 2023-11-16T16:52:37.522 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.519+0000 7f7b53fff700 1 -- 172.21.15.2:0/690147804 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b64023e80 con 0x563199ef86b0 2023-11-16T16:52:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.520+0000 7f7b83938b80 1 -- 172.21.15.2:0/690147804 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563199ef86b0 msgr2=0x56319a0a21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.520+0000 7f7b83938b80 1 --2- 172.21.15.2:0/690147804 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563199ef86b0 0x56319a0a21e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7b640099c0 tx=0x7f7b6401f000 comp rx=0 tx=0).stop 2023-11-16T16:52:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.521+0000 7f7b83938b80 1 -- 172.21.15.2:0/690147804 shutdown_connections 2023-11-16T16:52:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.521+0000 7f7b83938b80 1 --2- 172.21.15.2:0/690147804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56319a0a2b90 0x56319a09cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.521+0000 7f7b83938b80 1 --2- 172.21.15.2:0/690147804 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563199ef86b0 0x56319a0a21e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.521+0000 7f7b83938b80 1 --2- 172.21.15.2:0/690147804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2720 0x56319a09c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.521+0000 7f7b83938b80 1 -- 172.21.15.2:0/690147804 >> 172.21.15.2:0/690147804 conn(0x56319a091f50 msgr2=0x56319a097270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.521+0000 7f7b83938b80 1 -- 172.21.15.2:0/690147804 shutdown_connections 2023-11-16T16:52:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.521+0000 7f7b83938b80 1 -- 172.21.15.2:0/690147804 wait complete. 2023-11-16T16:52:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.521+0000 7f7b83938b80 1 -- start start 2023-11-16T16:52:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.522+0000 7f7b83938b80 1 Processor -- start 2023-11-16T16:52:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.522+0000 7f7b83938b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563199ef86b0 0x56319a0a5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.522+0000 7f7b83938b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2720 0x56319a0a6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.523+0000 7f7b83938b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56319a0a2b90 0x56319a0a6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.523+0000 7f7b83938b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x563199f8f880 con 0x56319a0a2720 2023-11-16T16:52:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.523+0000 7f7b83938b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x563199f8f520 con 0x563199ef86b0 2023-11-16T16:52:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.523+0000 7f7b83938b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x563199e718c0 con 0x56319a0a2b90 2023-11-16T16:52:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.523+0000 7f7b83938b80 10 client.0 _ll_register_callbacks cb 0x563199fad720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.523+0000 7f7b6dd9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563199ef86b0 0x56319a0a5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.523+0000 7f7b6dd9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563199ef86b0 0x56319a0a5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:60818/0 (socket says 172.21.15.2:60818) 2023-11-16T16:52:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.523+0000 7f7b6dd9b700 1 -- 172.21.15.2:0/579783452 learned_addr learned my addr 172.21.15.2:0/579783452 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.523+0000 7f7b6e59c700 1 --2- 172.21.15.2:0/579783452 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56319a0a2b90 0x56319a0a6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.523+0000 7f7b6d59a700 1 --2- 172.21.15.2:0/579783452 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2720 0x56319a0a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.524+0000 7f7b6e59c700 1 -- 172.21.15.2:0/579783452 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563199ef86b0 msgr2=0x56319a0a5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.524+0000 7f7b6e59c700 1 --2- 172.21.15.2:0/579783452 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563199ef86b0 0x56319a0a5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.524+0000 7f7b6e59c700 1 -- 172.21.15.2:0/579783452 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2720 msgr2=0x56319a0a6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.524+0000 7f7b6e59c700 1 --2- 172.21.15.2:0/579783452 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2720 0x56319a0a6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.524+0000 7f7b6e59c700 1 -- 172.21.15.2:0/579783452 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b68113f60 con 0x56319a0a2b90 2023-11-16T16:52:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.524+0000 7f7b6d59a700 1 --2- 172.21.15.2:0/579783452 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2720 0x56319a0a6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:52:37.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.524+0000 7f7b6e59c700 1 --2- 172.21.15.2:0/579783452 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56319a0a2b90 0x56319a0a6850 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7b60009fd0 tx=0x7f7b60014600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=145f5cfa30256676 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:37.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.524+0000 7f7b517fa700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:52:37.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.525+0000 7f7b517fa700 1 -- 172.21.15.2:0/579783452 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6000e040 con 0x56319a0a2b90 2023-11-16T16:52:37.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.525+0000 7f7b517fa700 1 -- 172.21.15.2:0/579783452 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f7b6001a070 con 0x56319a0a2b90 2023-11-16T16:52:37.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.525+0000 7f7b83938b80 1 -- 172.21.15.2:0/579783452 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56319a08f710 con 0x56319a0a2b90 2023-11-16T16:52:37.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.526+0000 7f7b517fa700 1 -- 172.21.15.2:0/579783452 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b600093a0 con 0x56319a0a2b90 2023-11-16T16:52:37.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.526+0000 7f7b83938b80 10 client.25663 fetch_fsmap learned FSMap version 32 2023-11-16T16:52:37.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.526+0000 7f7b83938b80 1 -- 172.21.15.2:0/579783452 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x563199fb4130 con 0x56319a0a2b90 2023-11-16T16:52:37.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.526+0000 7f7b517fa700 1 -- 172.21.15.2:0/579783452 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56319a08f710 con 0x56319a0a2b90 2023-11-16T16:52:37.529 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.527+0000 7f7b517fa700 1 -- 172.21.15.2:0/579783452 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 32) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f7b60004230 con 0x56319a0a2b90 2023-11-16T16:52:37.529 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.527+0000 7f7b83938b80 10 client.25663 fetch_fsmap finished waiting for FSMap version 32 2023-11-16T16:52:37.529 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.527+0000 7f7b83938b80 10 client.25663 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:37.529 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.527+0000 7f7b83938b80 20 client.25663 populate_metadata read hostname 'smithi002' 2023-11-16T16:52:37.529 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.527+0000 7f7b83938b80 1 --2- 172.21.15.2:0/579783452 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x56319a0b6020 0x56319a0b8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.527+0000 7f7b83938b80 4 client.25663 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.527+0000 7f7b83938b80 1 -- 172.21.15.2:0/579783452 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563199fb0040 con 0x56319a0b6020 2023-11-16T16:52:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.527+0000 7f7b6dd9b700 1 --2- 172.21.15.2:0/579783452 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x56319a0b6020 0x56319a0b8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.528+0000 7f7b6dd9b700 1 --2- 172.21.15.2:0/579783452 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x56319a0b6020 0x56319a0b8430 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9765cfe8dcd4bb31 server_cookie=58699cbe19bdea35 in_seq=0 out_seq=0 2023-11-16T16:52:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.528+0000 7f7b517fa700 10 client.25663 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:52:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.529+0000 7f7b517fa700 1 -- 172.21.15.2:0/579783452 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x563199fb0040 con 0x56319a0b6020 2023-11-16T16:52:37.531 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.529+0000 7f7b517fa700 10 client.25663 handle_command_reply: tid=0 2023-11-16T16:52:37.531 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.529+0000 7f7b83938b80 1 client.25663 shutdown 2023-11-16T16:52:37.532 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.529+0000 7f7b83938b80 1 -- 172.21.15.2:0/579783452 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56319a0a2b90 msgr2=0x56319a0a6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:37.532 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.529+0000 7f7b83938b80 1 --2- 172.21.15.2:0/579783452 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56319a0a2b90 0x56319a0a6850 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7b60009fd0 tx=0x7f7b60014600 comp rx=0 tx=0).stop 2023-11-16T16:52:37.532 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.530+0000 7f7b83938b80 1 -- 172.21.15.2:0/579783452 shutdown_connections 2023-11-16T16:52:37.532 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.530+0000 7f7b83938b80 1 --2- 172.21.15.2:0/579783452 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x56319a0b6020 0x56319a0b8430 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.530+0000 7f7b83938b80 1 --2- 172.21.15.2:0/579783452 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56319a0a2b90 0x56319a0a6850 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.530+0000 7f7b83938b80 1 --2- 172.21.15.2:0/579783452 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x563199ef86b0 0x56319a0a5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.530+0000 7f7b517fa700 0 client.25663 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:52:37.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.530+0000 7f7b83938b80 1 --2- 172.21.15.2:0/579783452 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2720 0x56319a0a6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.530+0000 7f7b83938b80 1 -- 172.21.15.2:0/579783452 >> 172.21.15.2:0/579783452 conn(0x56319a091f50 msgr2=0x56319a095f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:37.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.531+0000 7f7b83938b80 1 -- 172.21.15.2:0/579783452 shutdown_connections 2023-11-16T16:52:37.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.531+0000 7f7b83938b80 1 -- 172.21.15.2:0/579783452 wait complete. 2023-11-16T16:52:37.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.531+0000 7f7b83938b80 20 client.25663 trim_cache size 0 max 16384 2023-11-16T16:52:37.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.556+0000 7f7b83938b80 1 Processor -- start 2023-11-16T16:52:37.558 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.556+0000 7f7b83938b80 1 -- start start 2023-11-16T16:52:37.558 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.556+0000 7f7b83938b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563199ef86b0 0x56319a0ad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:37.558 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.556+0000 7f7b83938b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56319a0a2720 0x56319a0a66d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.557+0000 7f7b6dd9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563199ef86b0 0x56319a0ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.557+0000 7f7b83938b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2b90 0x56319a0a4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.557+0000 7f7b83938b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56319a099060 con 0x56319a0a2b90 2023-11-16T16:52:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.557+0000 7f7b83938b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x56319a0b29b0 con 0x56319a0a2720 2023-11-16T16:52:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.557+0000 7f7b83938b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x56319a0a8df0 con 0x563199ef86b0 2023-11-16T16:52:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.557+0000 7f7b6dd9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563199ef86b0 0x56319a0ad090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:49648/0 (socket says 172.21.15.2:49648) 2023-11-16T16:52:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.557+0000 7f7b6dd9b700 1 -- 172.21.15.2:0/2783575319 learned_addr learned my addr 172.21.15.2:0/2783575319 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.557+0000 7f7b6dd9b700 1 -- 172.21.15.2:0/2783575319 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56319a0a2720 msgr2=0x56319a0a66d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T16:52:37.559 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.557+0000 7f7b6dd9b700 1 --2- 172.21.15.2:0/2783575319 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56319a0a2720 0x56319a0a66d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.557+0000 7f7b6dd9b700 1 -- 172.21.15.2:0/2783575319 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2b90 msgr2=0x56319a0a4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.557+0000 7f7b6dd9b700 1 --2- 172.21.15.2:0/2783575319 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2b90 0x56319a0a4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.560 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.557+0000 7f7b6dd9b700 1 -- 172.21.15.2:0/2783575319 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563199fb3e80 con 0x563199ef86b0 2023-11-16T16:52:37.561 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.558+0000 7f7b6dd9b700 1 --2- 172.21.15.2:0/2783575319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563199ef86b0 0x56319a0ad090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7b64023a00 tx=0x7f7b64023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=594b36c8ccc8134b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:37.561 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.558+0000 7f7b5cff9700 1 -- 172.21.15.2:0/2783575319 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6403e250 con 0x563199ef86b0 2023-11-16T16:52:37.561 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.558+0000 7f7b5cff9700 1 -- 172.21.15.2:0/2783575319 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b64042070 con 0x563199ef86b0 2023-11-16T16:52:37.561 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.558+0000 7f7b5cff9700 1 -- 172.21.15.2:0/2783575319 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6400f8c0 con 0x563199ef86b0 2023-11-16T16:52:37.561 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.559+0000 7f7b83938b80 1 -- 172.21.15.2:0/2783575319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563199ef86b0 msgr2=0x56319a0ad090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:37.561 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.559+0000 7f7b83938b80 1 --2- 172.21.15.2:0/2783575319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563199ef86b0 0x56319a0ad090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7b64023a00 tx=0x7f7b64023000 comp rx=0 tx=0).stop 2023-11-16T16:52:37.561 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.559+0000 7f7b83938b80 1 -- 172.21.15.2:0/2783575319 shutdown_connections 2023-11-16T16:52:37.561 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.559+0000 7f7b83938b80 1 --2- 172.21.15.2:0/2783575319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563199ef86b0 0x56319a0ad090 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.561 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.559+0000 7f7b83938b80 1 --2- 172.21.15.2:0/2783575319 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2b90 0x56319a0a4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.562 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.559+0000 7f7b83938b80 1 --2- 172.21.15.2:0/2783575319 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56319a0a2720 0x56319a0a66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.562 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.559+0000 7f7b83938b80 1 -- 172.21.15.2:0/2783575319 >> 172.21.15.2:0/2783575319 conn(0x56319a091f50 msgr2=0x56319a0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:37.562 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.559+0000 7f7b83938b80 1 -- 172.21.15.2:0/2783575319 shutdown_connections 2023-11-16T16:52:37.562 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.559+0000 7f7b83938b80 1 -- 172.21.15.2:0/2783575319 wait complete. 2023-11-16T16:52:37.562 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.560+0000 7f7b83938b80 1 -- start start 2023-11-16T16:52:37.562 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.560+0000 7f7b83938b80 1 Processor -- start 2023-11-16T16:52:37.563 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.561+0000 7f7b83938b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563199ef86b0 0x56319a0ad020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:37.563 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.561+0000 7f7b83938b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56319a0a2720 0x56319a0a8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:37.563 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.561+0000 7f7b83938b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2b90 0x56319a0a9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:37.563 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.561+0000 7f7b83938b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56319a0ad6d0 con 0x56319a0a2b90 2023-11-16T16:52:37.563 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.561+0000 7f7b83938b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x56319a0a98a0 con 0x56319a0a2720 2023-11-16T16:52:37.563 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.561+0000 7f7b83938b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x56319a0a9a60 con 0x563199ef86b0 2023-11-16T16:52:37.563 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.561+0000 7f7b83938b80 10 client.0 _ll_register_callbacks cb 0x7f7b6000e7e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.561+0000 7f7b6e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2b90 0x56319a0a9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.561+0000 7f7b6e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2b90 0x56319a0a9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32800/0 (socket says 172.21.15.2:32800) 2023-11-16T16:52:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.561+0000 7f7b6e59c700 1 -- 172.21.15.2:0/3602301271 learned_addr learned my addr 172.21.15.2:0/3602301271 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.561+0000 7f7b6d59a700 1 --2- 172.21.15.2:0/3602301271 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56319a0a2720 0x56319a0a8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.562+0000 7f7b6dd9b700 1 --2- 172.21.15.2:0/3602301271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563199ef86b0 0x56319a0ad020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.562+0000 7f7b6e59c700 1 -- 172.21.15.2:0/3602301271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563199ef86b0 msgr2=0x56319a0ad020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:37.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.562+0000 7f7b6e59c700 1 --2- 172.21.15.2:0/3602301271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563199ef86b0 0x56319a0ad020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.562+0000 7f7b6e59c700 1 -- 172.21.15.2:0/3602301271 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56319a0a2720 msgr2=0x56319a0a8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.562+0000 7f7b6e59c700 1 --2- 172.21.15.2:0/3602301271 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56319a0a2720 0x56319a0a8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.562+0000 7f7b6e59c700 1 -- 172.21.15.2:0/3602301271 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563199e6f5d0 con 0x56319a0a2b90 2023-11-16T16:52:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.562+0000 7f7b6dd9b700 1 --2- 172.21.15.2:0/3602301271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563199ef86b0 0x56319a0ad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:52:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.562+0000 7f7b6e59c700 1 --2- 172.21.15.2:0/3602301271 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2b90 0x56319a0a9330 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7b60004610 tx=0x7f7b60005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=279da5439f23fbf2 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:37.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.562+0000 7f7b517fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:52:37.566 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.563+0000 7f7b517fa700 1 -- 172.21.15.2:0/3602301271 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6000e040 con 0x56319a0a2b90 2023-11-16T16:52:37.566 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.563+0000 7f7b517fa700 1 -- 172.21.15.2:0/3602301271 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f7b6001a040 con 0x56319a0a2b90 2023-11-16T16:52:37.566 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.564+0000 7f7b83938b80 1 -- 172.21.15.2:0/3602301271 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b6401d040 con 0x56319a0a2b90 2023-11-16T16:52:37.566 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.564+0000 7f7b517fa700 1 -- 172.21.15.2:0/3602301271 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b600093a0 con 0x56319a0a2b90 2023-11-16T16:52:37.566 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.564+0000 7f7b83938b80 10 client.15657 fetch_fsmap learned FSMap version 32 2023-11-16T16:52:37.566 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.564+0000 7f7b83938b80 1 -- 172.21.15.2:0/3602301271 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x563199e04c00 con 0x56319a0a2b90 2023-11-16T16:52:37.566 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.564+0000 7f7b517fa700 1 -- 172.21.15.2:0/3602301271 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b6401d040 con 0x56319a0a2b90 2023-11-16T16:52:37.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.565+0000 7f7b517fa700 1 -- 172.21.15.2:0/3602301271 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 32) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f7b60031070 con 0x56319a0a2b90 2023-11-16T16:52:37.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.565+0000 7f7b83938b80 10 client.15657 fetch_fsmap finished waiting for FSMap version 32 2023-11-16T16:52:37.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.565+0000 7f7b83938b80 10 client.15657 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:37.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.565+0000 7f7b83938b80 20 client.15657 populate_metadata read hostname 'smithi002' 2023-11-16T16:52:37.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.565+0000 7f7b83938b80 1 --2- 172.21.15.2:0/3602301271 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x56319a10a250 0x56319a10c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:37.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.565+0000 7f7b83938b80 4 client.15657 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:52:37.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.565+0000 7f7b83938b80 1 -- 172.21.15.2:0/3602301271 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563199fbc600 con 0x56319a10a250 2023-11-16T16:52:37.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.565+0000 7f7b6dd9b700 1 --2- 172.21.15.2:0/3602301271 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x56319a10a250 0x56319a10c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:37.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.566+0000 7f7b6dd9b700 1 --2- 172.21.15.2:0/3602301271 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x56319a10a250 0x56319a10c660 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88276144022a40dc server_cookie=d4894bee318e68f1 in_seq=0 out_seq=0 2023-11-16T16:52:37.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:37.566+0000 7f7b517fa700 10 client.15657 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:52:38.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:37 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:52:38.373 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:37 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:52:38.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:37 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:52:39.059 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.057+0000 7f7b517fa700 1 -- 172.21.15.2:0/3602301271 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563199fbc600 con 0x56319a10a250 2023-11-16T16:52:39.059 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.057+0000 7f7b517fa700 10 client.15657 handle_command_reply: tid=0 2023-11-16T16:52:39.059 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.057+0000 7f7b83938b80 1 client.15657 shutdown 2023-11-16T16:52:39.059 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.058+0000 7f7b83938b80 1 -- 172.21.15.2:0/3602301271 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2b90 msgr2=0x56319a0a9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:39.059 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.058+0000 7f7b83938b80 1 --2- 172.21.15.2:0/3602301271 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2b90 0x56319a0a9330 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7b60004610 tx=0x7f7b60005b20 comp rx=0 tx=0).stop 2023-11-16T16:52:39.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.058+0000 7f7b83938b80 1 -- 172.21.15.2:0/3602301271 shutdown_connections 2023-11-16T16:52:39.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.058+0000 7f7b83938b80 1 --2- 172.21.15.2:0/3602301271 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x56319a10a250 0x56319a10c660 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.058+0000 7f7b83938b80 1 --2- 172.21.15.2:0/3602301271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563199ef86b0 0x56319a0ad020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.058+0000 7f7b83938b80 1 --2- 172.21.15.2:0/3602301271 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56319a0a2b90 0x56319a0a9330 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.058+0000 7f7b517fa700 0 client.15657 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:52:39.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.058+0000 7f7b83938b80 1 --2- 172.21.15.2:0/3602301271 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56319a0a2720 0x56319a0a8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.058+0000 7f7b83938b80 1 -- 172.21.15.2:0/3602301271 >> 172.21.15.2:0/3602301271 conn(0x56319a091f50 msgr2=0x56319a0a4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:39.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.058+0000 7f7b83938b80 1 -- 172.21.15.2:0/3602301271 shutdown_connections 2023-11-16T16:52:39.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.058+0000 7f7b83938b80 1 -- 172.21.15.2:0/3602301271 wait complete. 2023-11-16T16:52:39.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.059+0000 7f7b83938b80 20 client.15657 trim_cache size 0 max 16384 2023-11-16T16:52:39.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:52:39.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:52:39.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:52:39.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:52:39.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.061+0000 7f7b6ffff700 1 -- 172.21.15.2:0/991238202 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f7b540a1410 msgr2=0x7f7b540a38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:39.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.061+0000 7f7b6ffff700 1 --2- 172.21.15.2:0/991238202 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f7b540a1410 0x7f7b540a38c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7b64000c00 tx=0x7f7b64019040 comp rx=0 tx=0).stop 2023-11-16T16:52:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.061+0000 7f7b6ffff700 1 -- 172.21.15.2:0/991238202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b680ffb40 msgr2=0x7f7b680701a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.061+0000 7f7b6ffff700 1 --2- 172.21.15.2:0/991238202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b680ffb40 0x7f7b680701a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800e9f0 tx=0x7f7b5800ed00 comp rx=0 tx=0).stop 2023-11-16T16:52:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.061+0000 7f7b6ffff700 1 -- 172.21.15.2:0/991238202 shutdown_connections 2023-11-16T16:52:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.061+0000 7f7b6ffff700 1 --2- 172.21.15.2:0/991238202 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f7b540a1410 0x7f7b540a38c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.061+0000 7f7b6ffff700 1 --2- 172.21.15.2:0/991238202 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7b68104630 0x7f7b6806ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.061+0000 7f7b6ffff700 1 --2- 172.21.15.2:0/991238202 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b680ff230 0x7f7b6806fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.061+0000 7f7b6ffff700 1 --2- 172.21.15.2:0/991238202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b680ffb40 0x7f7b680701a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:39.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.061+0000 7f7b6ffff700 1 -- 172.21.15.2:0/991238202 >> 172.21.15.2:0/991238202 conn(0x7f7b680763a0 msgr2=0x7f7b68102050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:39.064 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.062+0000 7f7b6ffff700 1 -- 172.21.15.2:0/991238202 shutdown_connections 2023-11-16T16:52:39.064 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:39.062+0000 7f7b6ffff700 1 -- 172.21.15.2:0/991238202 wait complete. 2023-11-16T16:52:39.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:38 smithi141 ceph-mon[131184]: pgmap v85: 129 pgs: 129 active+clean; 14 MiB data, 96 MiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 1.0 MiB/s wr, 24 op/s 2023-11-16T16:52:39.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:38 smithi155 ceph-mon[131607]: pgmap v85: 129 pgs: 129 active+clean; 14 MiB data, 96 MiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 1.0 MiB/s wr, 24 op/s 2023-11-16T16:52:39.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:38 smithi002 ceph-mon[128712]: pgmap v85: 129 pgs: 129 active+clean; 14 MiB data, 96 MiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 1.0 MiB/s wr, 24 op/s 2023-11-16T16:52:41.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:40 smithi141 ceph-mon[131184]: pgmap v86: 129 pgs: 129 active+clean; 14 MiB data, 96 MiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 1008 KiB/s wr, 19 op/s 2023-11-16T16:52:41.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:40 smithi155 ceph-mon[131607]: pgmap v86: 129 pgs: 129 active+clean; 14 MiB data, 96 MiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 1008 KiB/s wr, 19 op/s 2023-11-16T16:52:41.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:40 smithi002 ceph-mon[128712]: pgmap v86: 129 pgs: 129 active+clean; 14 MiB data, 96 MiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 1008 KiB/s wr, 19 op/s 2023-11-16T16:52:43.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:42 smithi141 ceph-mon[131184]: pgmap v87: 129 pgs: 129 active+clean; 24 MiB data, 182 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 795 op/s 2023-11-16T16:52:43.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:42 smithi155 ceph-mon[131607]: pgmap v87: 129 pgs: 129 active+clean; 24 MiB data, 182 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 795 op/s 2023-11-16T16:52:43.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:42 smithi002 ceph-mon[128712]: pgmap v87: 129 pgs: 129 active+clean; 24 MiB data, 182 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 795 op/s 2023-11-16T16:52:45.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:44 smithi141 ceph-mon[131184]: pgmap v88: 129 pgs: 129 active+clean; 24 MiB data, 182 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 790 op/s 2023-11-16T16:52:45.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:44 smithi155 ceph-mon[131607]: pgmap v88: 129 pgs: 129 active+clean; 24 MiB data, 182 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 790 op/s 2023-11-16T16:52:45.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:44 smithi002 ceph-mon[128712]: pgmap v88: 129 pgs: 129 active+clean; 24 MiB data, 182 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 790 op/s 2023-11-16T16:52:47.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:46 smithi141 ceph-mon[131184]: pgmap v89: 129 pgs: 129 active+clean; 24 MiB data, 182 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 790 op/s 2023-11-16T16:52:47.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:46 smithi155 ceph-mon[131607]: pgmap v89: 129 pgs: 129 active+clean; 24 MiB data, 182 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 790 op/s 2023-11-16T16:52:47.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:46 smithi002 ceph-mon[128712]: pgmap v89: 129 pgs: 129 active+clean; 24 MiB data, 182 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 790 op/s 2023-11-16T16:52:49.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:48 smithi141 ceph-mon[131184]: pgmap v90: 129 pgs: 129 active+clean; 39 MiB data, 191 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 801 op/s 2023-11-16T16:52:49.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:48 smithi155 ceph-mon[131607]: pgmap v90: 129 pgs: 129 active+clean; 39 MiB data, 191 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 801 op/s 2023-11-16T16:52:49.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:48 smithi002 ceph-mon[128712]: pgmap v90: 129 pgs: 129 active+clean; 39 MiB data, 191 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 801 op/s 2023-11-16T16:52:51.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:50 smithi141 ceph-mon[131184]: pgmap v91: 129 pgs: 129 active+clean; 39 MiB data, 191 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 786 op/s 2023-11-16T16:52:51.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:50 smithi155 ceph-mon[131607]: pgmap v91: 129 pgs: 129 active+clean; 39 MiB data, 191 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 786 op/s 2023-11-16T16:52:51.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:50 smithi002 ceph-mon[128712]: pgmap v91: 129 pgs: 129 active+clean; 39 MiB data, 191 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 786 op/s 2023-11-16T16:52:52.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.283+0000 7fc100b9d700 1 -- 172.21.15.2:0/2876755736 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0fc07aa80 msgr2=0x7fc0fc1134e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:52.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.283+0000 7fc100b9d700 1 --2- 172.21.15.2:0/2876755736 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0fc07aa80 0x7fc0fc1134e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec009f90 tx=0x7fc0ec00b330 comp rx=0 tx=0).stop 2023-11-16T16:52:52.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.283+0000 7fc100b9d700 1 -- 172.21.15.2:0/2876755736 shutdown_connections 2023-11-16T16:52:52.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.283+0000 7fc100b9d700 1 --2- 172.21.15.2:0/2876755736 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0fc07aa80 0x7fc0fc1134e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.283+0000 7fc100b9d700 1 --2- 172.21.15.2:0/2876755736 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc0fc10c6b0 0x7fc0fc10ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.283+0000 7fc100b9d700 1 --2- 172.21.15.2:0/2876755736 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0fc07a0d0 0x7fc0fc07a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.283+0000 7fc100b9d700 1 -- 172.21.15.2:0/2876755736 >> 172.21.15.2:0/2876755736 conn(0x7fc0fc075bd0 msgr2=0x7fc0fc077fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:52.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.283+0000 7fc100b9d700 1 -- 172.21.15.2:0/2876755736 shutdown_connections 2023-11-16T16:52:52.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.283+0000 7fc100b9d700 1 -- 172.21.15.2:0/2876755736 wait complete. 2023-11-16T16:52:52.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.285+0000 7fc100b9d700 1 Processor -- start 2023-11-16T16:52:52.288 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.285+0000 7fc100b9d700 1 -- start start 2023-11-16T16:52:52.288 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.286+0000 7fc100b9d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc0fc07a0d0 0x7fc0fc1a2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:52.288 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.286+0000 7fc100b9d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0fc07aa80 0x7fc0fc1a2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:52.288 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.286+0000 7fc100b9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0fc10c6b0 0x7fc0fc19c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:52.289 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.286+0000 7fc100b9d700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0fc19cda0 con 0x7fc0fc10c6b0 2023-11-16T16:52:52.289 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.286+0000 7fc100b9d700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0fc19cf10 con 0x7fc0fc07a0d0 2023-11-16T16:52:52.289 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.286+0000 7fc0faffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0fc07aa80 0x7fc0fc1a2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:52.289 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.286+0000 7fc0fb7fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc0fc07a0d0 0x7fc0fc1a2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:52.289 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.286+0000 7fc0faffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0fc07aa80 0x7fc0fc1a2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:56840/0 (socket says 172.21.15.2:56840) 2023-11-16T16:52:52.289 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.286+0000 7fc0faffd700 1 -- 172.21.15.2:0/242229986 learned_addr learned my addr 172.21.15.2:0/242229986 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:52.289 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.287+0000 7fc100b9d700 1 -- 172.21.15.2:0/242229986 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0fc19d0d0 con 0x7fc0fc07aa80 2023-11-16T16:52:52.289 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.287+0000 7fc0fbfff700 1 --2- 172.21.15.2:0/242229986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0fc10c6b0 0x7fc0fc19c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:52.290 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.287+0000 7fc0faffd700 1 -- 172.21.15.2:0/242229986 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc0fc07a0d0 msgr2=0x7fc0fc1a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:52.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.287+0000 7fc0faffd700 1 --2- 172.21.15.2:0/242229986 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc0fc07a0d0 0x7fc0fc1a2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.287+0000 7fc0faffd700 1 -- 172.21.15.2:0/242229986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0fc10c6b0 msgr2=0x7fc0fc19c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:52.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.287+0000 7fc0faffd700 1 --2- 172.21.15.2:0/242229986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0fc10c6b0 0x7fc0fc19c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.287+0000 7fc0faffd700 1 -- 172.21.15.2:0/242229986 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ec009c40 con 0x7fc0fc07aa80 2023-11-16T16:52:52.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.287+0000 7fc0faffd700 1 --2- 172.21.15.2:0/242229986 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0fc07aa80 0x7fc0fc1a2bb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400e9a0 tx=0x7fc0e400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:52.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.288+0000 7fc0f8ff9700 1 -- 172.21.15.2:0/242229986 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e400c9c0 con 0x7fc0fc07aa80 2023-11-16T16:52:52.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.288+0000 7fc0f8ff9700 1 -- 172.21.15.2:0/242229986 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0e4010430 con 0x7fc0fc07aa80 2023-11-16T16:52:52.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.288+0000 7fc100b9d700 1 -- 172.21.15.2:0/242229986 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0fc116700 con 0x7fc0fc07aa80 2023-11-16T16:52:52.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.288+0000 7fc0f8ff9700 1 -- 172.21.15.2:0/242229986 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e4018680 con 0x7fc0fc07aa80 2023-11-16T16:52:52.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.288+0000 7fc100b9d700 1 -- 172.21.15.2:0/242229986 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0fc116570 con 0x7fc0fc07aa80 2023-11-16T16:52:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.294+0000 7fc0f8ff9700 1 -- 172.21.15.2:0/242229986 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fc0e4010860 con 0x7fc0fc07aa80 2023-11-16T16:52:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.297+0000 7fc115704b80 1 Processor -- start 2023-11-16T16:52:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.297+0000 7fc0f8ff9700 1 --2- 172.21.15.2:0/242229986 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fc0e80a1340 0x7fc0e80a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:52.301 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.298+0000 7fc115704b80 1 -- start start 2023-11-16T16:52:52.301 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.298+0000 7fc0f8ff9700 1 -- 172.21.15.2:0/242229986 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fc0e4014070 con 0x7fc0fc07aa80 2023-11-16T16:52:52.301 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.298+0000 7fc115704b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f42e6b0 0x55633f5d8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:52.301 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.298+0000 7fc115704b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f5d8990 0x55633f5d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:52.301 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.298+0000 7fc115704b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f5d2cf0 0x55633f5d3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:52.301 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.298+0000 7fc115704b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633f4d8000 con 0x55633f42e6b0 2023-11-16T16:52:52.301 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.298+0000 7fc115704b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633f52a840 con 0x55633f5d2cf0 2023-11-16T16:52:52.301 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.298+0000 7fc115704b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633f523cd0 con 0x55633f5d8990 2023-11-16T16:52:52.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.298+0000 7fc0faffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f5d8990 0x55633f5d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:52.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.299+0000 7fc0faffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f5d8990 0x55633f5d27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:56850/0 (socket says 172.21.15.2:56850) 2023-11-16T16:52:52.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.299+0000 7fc0faffd700 1 -- 172.21.15.2:0/2931488692 learned_addr learned my addr 172.21.15.2:0/2931488692 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:52.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.299+0000 7fc0fb7fe700 1 --2- 172.21.15.2:0/2931488692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f42e6b0 0x55633f5d8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:52.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.299+0000 7fc0faffd700 1 -- 172.21.15.2:0/2931488692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f5d2cf0 msgr2=0x55633f5d3120 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T16:52:52.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.299+0000 7fc0faffd700 1 --2- 172.21.15.2:0/2931488692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f5d2cf0 0x55633f5d3120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.299+0000 7fc0faffd700 1 -- 172.21.15.2:0/2931488692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f42e6b0 msgr2=0x55633f5d8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:52.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.299+0000 7fc0faffd700 1 --2- 172.21.15.2:0/2931488692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f42e6b0 0x55633f5d8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.299+0000 7fc0faffd700 1 -- 172.21.15.2:0/2931488692 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55633f3a55d0 con 0x55633f5d8990 2023-11-16T16:52:52.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.299+0000 7fc0faffd700 1 --2- 172.21.15.2:0/2931488692 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f5d8990 0x55633f5d27b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc0e40964e0 tx=0x7fc0e4099c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17a38e54871cbbd3 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:52.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.299+0000 7fc0fbfff700 1 --2- 172.21.15.2:0/242229986 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fc0e80a1340 0x7fc0e80a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:52.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.300+0000 7fc0f1ffb700 1 -- 172.21.15.2:0/2931488692 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e40a2430 con 0x55633f5d8990 2023-11-16T16:52:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.300+0000 7fc0f1ffb700 1 -- 172.21.15.2:0/2931488692 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0e40105a0 con 0x55633f5d8990 2023-11-16T16:52:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.300+0000 7fc0f1ffb700 1 -- 172.21.15.2:0/2931488692 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e40a1fa0 con 0x55633f5d8990 2023-11-16T16:52:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.301+0000 7fc115704b80 1 -- 172.21.15.2:0/2931488692 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f5d8990 msgr2=0x55633f5d27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.301+0000 7fc115704b80 1 --2- 172.21.15.2:0/2931488692 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f5d8990 0x55633f5d27b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc0e40964e0 tx=0x7fc0e4099c30 comp rx=0 tx=0).stop 2023-11-16T16:52:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.301+0000 7fc0fbfff700 1 --2- 172.21.15.2:0/242229986 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fc0e80a1340 0x7fc0e80a37f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc1a3370 tx=0x7fc0ec009700 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.301+0000 7fc115704b80 1 -- 172.21.15.2:0/2931488692 shutdown_connections 2023-11-16T16:52:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.301+0000 7fc115704b80 1 --2- 172.21.15.2:0/2931488692 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f5d2cf0 0x55633f5d3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.301+0000 7fc115704b80 1 --2- 172.21.15.2:0/2931488692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f42e6b0 0x55633f5d8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.301+0000 7fc115704b80 1 --2- 172.21.15.2:0/2931488692 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f5d8990 0x55633f5d27b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.304 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.301+0000 7fc115704b80 1 -- 172.21.15.2:0/2931488692 >> 172.21.15.2:0/2931488692 conn(0x55633f5c81a0 msgr2=0x55633f5cd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:52.305 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.301+0000 7fc115704b80 1 -- 172.21.15.2:0/2931488692 shutdown_connections 2023-11-16T16:52:52.305 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.301+0000 7fc115704b80 1 -- 172.21.15.2:0/2931488692 wait complete. 2023-11-16T16:52:52.305 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.302+0000 7fc115704b80 1 -- start start 2023-11-16T16:52:52.305 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.302+0000 7fc115704b80 1 Processor -- start 2023-11-16T16:52:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.303+0000 7fc115704b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f42e6b0 0x55633f5dc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.303+0000 7fc115704b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f5d2cf0 0x55633f5dc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.303+0000 7fc115704b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f5d8990 0x55633f5dcac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.303+0000 7fc115704b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633f4c5880 con 0x55633f42e6b0 2023-11-16T16:52:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.303+0000 7fc115704b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633f4c5520 con 0x55633f5d8990 2023-11-16T16:52:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.303+0000 7fc115704b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633f3a78c0 con 0x55633f5d2cf0 2023-11-16T16:52:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.303+0000 7fc115704b80 10 client.0 _ll_register_callbacks cb 0x55633f4e3720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.303+0000 7fc0fbfff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f5d8990 0x55633f5dcac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.303+0000 7fc0fbfff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f5d8990 0x55633f5dcac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:48406/0 (socket says 172.21.15.2:48406) 2023-11-16T16:52:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.303+0000 7fc0fbfff700 1 -- 172.21.15.2:0/1304242488 learned_addr learned my addr 172.21.15.2:0/1304242488 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:52.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.304+0000 7fc0fbfff700 1 -- 172.21.15.2:0/1304242488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f5d2cf0 msgr2=0x55633f5dc580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T16:52:52.308 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.304+0000 7fc0fbfff700 1 --2- 172.21.15.2:0/1304242488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f5d2cf0 0x55633f5dc580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.308 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.304+0000 7fc0fbfff700 1 -- 172.21.15.2:0/1304242488 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f42e6b0 msgr2=0x55633f5dc040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T16:52:52.308 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.304+0000 7fc0fbfff700 1 --2- 172.21.15.2:0/1304242488 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f42e6b0 0x55633f5dc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.308 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.304+0000 7fc0fbfff700 1 -- 172.21.15.2:0/1304242488 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0fc116570 con 0x55633f5d8990 2023-11-16T16:52:52.308 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.304+0000 7fc0fbfff700 1 --2- 172.21.15.2:0/1304242488 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f5d8990 0x55633f5dcac0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec007f90 tx=0x7fc0ec009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1995e7a29addba04 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:52.308 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.304+0000 7fc0daffd700 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2023-11-16T16:52:52.308 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.305+0000 7fc0daffd700 1 -- 172.21.15.2:0/1304242488 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec00e040 con 0x55633f5d8990 2023-11-16T16:52:52.308 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.305+0000 7fc0daffd700 1 -- 172.21.15.2:0/1304242488 <== mon.2 v2:172.21.15.141:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fc0ec003d90 con 0x55633f5d8990 2023-11-16T16:52:52.308 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.305+0000 7fc115704b80 1 -- 172.21.15.2:0/1304242488 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55633f5c5960 con 0x55633f5d8990 2023-11-16T16:52:52.308 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.305+0000 7fc0daffd700 1 -- 172.21.15.2:0/1304242488 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0ec01c440 con 0x55633f5d8990 2023-11-16T16:52:52.308 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.305+0000 7fc115704b80 10 client.25703 fetch_fsmap learned FSMap version 32 2023-11-16T16:52:52.309 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.305+0000 7fc115704b80 1 -- 172.21.15.2:0/1304242488 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55633f4ea130 con 0x55633f5d8990 2023-11-16T16:52:52.309 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.306+0000 7fc0daffd700 1 -- 172.21.15.2:0/1304242488 <== mon.2 v2:172.21.15.141:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55633f5c5960 con 0x55633f5d8990 2023-11-16T16:52:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.306+0000 7fc0daffd700 1 -- 172.21.15.2:0/1304242488 <== mon.2 v2:172.21.15.141:3300/0 5 ==== fsmap(e 32) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fc0ec033070 con 0x55633f5d8990 2023-11-16T16:52:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.306+0000 7fc115704b80 10 client.25703 fetch_fsmap finished waiting for FSMap version 32 2023-11-16T16:52:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.307+0000 7fc115704b80 10 client.25703 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.307+0000 7fc115704b80 20 client.25703 populate_metadata read hostname 'smithi002' 2023-11-16T16:52:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.307+0000 7fc115704b80 1 --2- 172.21.15.2:0/1304242488 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55633f5ec260 0x55633f5ee670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.307+0000 7fc115704b80 4 client.25703 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:52:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.307+0000 7fc115704b80 1 -- 172.21.15.2:0/1304242488 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55633f4e6040 con 0x55633f5ec260 2023-11-16T16:52:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.307+0000 7fc0fb7fe700 1 --2- 172.21.15.2:0/1304242488 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55633f5ec260 0x55633f5ee670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.308+0000 7fc0daffd700 10 client.25703 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:52:52.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.308+0000 7fc0fb7fe700 1 --2- 172.21.15.2:0/1304242488 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55633f5ec260 0x55633f5ee670 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d62a373102757355 server_cookie=23116fa98f2eaac4 in_seq=0 out_seq=0 2023-11-16T16:52:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.309+0000 7fc0daffd700 1 -- 172.21.15.2:0/1304242488 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55633f4e6040 con 0x55633f5ec260 2023-11-16T16:52:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.309+0000 7fc0daffd700 10 client.25703 handle_command_reply: tid=0 2023-11-16T16:52:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.309+0000 7fc115704b80 1 client.25703 shutdown 2023-11-16T16:52:52.311 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.310+0000 7fc115704b80 1 -- 172.21.15.2:0/1304242488 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f5d8990 msgr2=0x55633f5dcac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.310+0000 7fc115704b80 1 --2- 172.21.15.2:0/1304242488 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f5d8990 0x55633f5dcac0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec007f90 tx=0x7fc0ec009920 comp rx=0 tx=0).stop 2023-11-16T16:52:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.310+0000 7fc115704b80 1 -- 172.21.15.2:0/1304242488 shutdown_connections 2023-11-16T16:52:52.312 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.310+0000 7fc115704b80 1 --2- 172.21.15.2:0/1304242488 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f5d8990 0x55633f5dcac0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.310+0000 7fc115704b80 1 --2- 172.21.15.2:0/1304242488 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f42e6b0 0x55633f5dc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.310+0000 7fc115704b80 1 --2- 172.21.15.2:0/1304242488 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55633f5ec260 0x55633f5ee670 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.310+0000 7fc115704b80 1 --2- 172.21.15.2:0/1304242488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f5d2cf0 0x55633f5dc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.310+0000 7fc115704b80 1 -- 172.21.15.2:0/1304242488 >> 172.21.15.2:0/1304242488 conn(0x55633f5c81a0 msgr2=0x55633f5cca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.311+0000 7fc0daffd700 0 client.25703 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:52:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.311+0000 7fc115704b80 1 -- 172.21.15.2:0/1304242488 shutdown_connections 2023-11-16T16:52:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.311+0000 7fc115704b80 1 -- 172.21.15.2:0/1304242488 wait complete. 2023-11-16T16:52:52.313 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.311+0000 7fc115704b80 20 client.25703 trim_cache size 0 max 16384 2023-11-16T16:52:52.340 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.338+0000 7fc115704b80 1 Processor -- start 2023-11-16T16:52:52.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.338+0000 7fc115704b80 1 -- start start 2023-11-16T16:52:52.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.339+0000 7fc115704b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f42e6b0 0x55633f5d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:52.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.339+0000 7fc115704b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f5d2cf0 0x55633f5e3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:52.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.339+0000 7fc115704b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f5d8990 0x55633f5dc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:52.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.339+0000 7fc115704b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633f5ee980 con 0x55633f5d2cf0 2023-11-16T16:52:52.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.339+0000 7fc115704b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633f5e80a0 con 0x55633f5d8990 2023-11-16T16:52:52.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.339+0000 7fc115704b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633f5e37b0 con 0x55633f42e6b0 2023-11-16T16:52:52.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.339+0000 7fc0fbfff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f5d8990 0x55633f5dc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:52.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.339+0000 7fc0fbfff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f5d8990 0x55633f5dc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:48414/0 (socket says 172.21.15.2:48414) 2023-11-16T16:52:52.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.339+0000 7fc0fbfff700 1 -- 172.21.15.2:0/1834984219 learned_addr learned my addr 172.21.15.2:0/1834984219 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:52.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.340+0000 7fc0faffd700 1 --2- 172.21.15.2:0/1834984219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f5d2cf0 0x55633f5e3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:52.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.340+0000 7fc0fb7fe700 1 --2- 172.21.15.2:0/1834984219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f42e6b0 0x55633f5d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:52.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.340+0000 7fc0fbfff700 1 -- 172.21.15.2:0/1834984219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f42e6b0 msgr2=0x55633f5d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:52.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.340+0000 7fc0fbfff700 1 --2- 172.21.15.2:0/1834984219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f42e6b0 0x55633f5d27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.340+0000 7fc0fbfff700 1 -- 172.21.15.2:0/1834984219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f5d2cf0 msgr2=0x55633f5e3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:52.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.340+0000 7fc0fbfff700 1 --2- 172.21.15.2:0/1834984219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f5d2cf0 0x55633f5e3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.340+0000 7fc0fbfff700 1 -- 172.21.15.2:0/1834984219 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55633f5f32a0 con 0x55633f5d8990 2023-11-16T16:52:52.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.340+0000 7fc0faffd700 1 --2- 172.21.15.2:0/1834984219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f5d2cf0 0x55633f5e3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:52:52.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.341+0000 7fc0fbfff700 1 --2- 172.21.15.2:0/1834984219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f5d8990 0x55633f5dc8f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0299b0 tx=0x7fc0ec0299e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b809387beaaa1c5 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.341+0000 7fc0f27fc700 1 -- 172.21.15.2:0/1834984219 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec00e070 con 0x55633f5d8990 2023-11-16T16:52:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.342+0000 7fc0f27fc700 1 -- 172.21.15.2:0/1834984219 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0ec029da0 con 0x55633f5d8990 2023-11-16T16:52:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.342+0000 7fc0f27fc700 1 -- 172.21.15.2:0/1834984219 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec0050a0 con 0x55633f5d8990 2023-11-16T16:52:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.343+0000 7fc115704b80 1 -- 172.21.15.2:0/1834984219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f5d8990 msgr2=0x55633f5dc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.343+0000 7fc115704b80 1 --2- 172.21.15.2:0/1834984219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f5d8990 0x55633f5dc8f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0299b0 tx=0x7fc0ec0299e0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.343+0000 7fc115704b80 1 -- 172.21.15.2:0/1834984219 shutdown_connections 2023-11-16T16:52:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.343+0000 7fc115704b80 1 --2- 172.21.15.2:0/1834984219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f42e6b0 0x55633f5d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.343+0000 7fc115704b80 1 --2- 172.21.15.2:0/1834984219 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f5d8990 0x55633f5dc8f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.343+0000 7fc115704b80 1 --2- 172.21.15.2:0/1834984219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f5d2cf0 0x55633f5e3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.343+0000 7fc115704b80 1 -- 172.21.15.2:0/1834984219 >> 172.21.15.2:0/1834984219 conn(0x55633f5c81a0 msgr2=0x55633f5ee270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.343+0000 7fc115704b80 1 -- 172.21.15.2:0/1834984219 shutdown_connections 2023-11-16T16:52:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.343+0000 7fc115704b80 1 -- 172.21.15.2:0/1834984219 wait complete. 2023-11-16T16:52:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.344+0000 7fc115704b80 1 -- start start 2023-11-16T16:52:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.344+0000 7fc115704b80 1 Processor -- start 2023-11-16T16:52:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.344+0000 7fc115704b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f42e6b0 0x55633f5e2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.344+0000 7fc115704b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f5d2cf0 0x55633f5e3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.344+0000 7fc115704b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f5d8990 0x55633f5e01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:52.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.345+0000 7fc115704b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633f5e08f0 con 0x55633f5d2cf0 2023-11-16T16:52:52.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.345+0000 7fc115704b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633f5e0a60 con 0x55633f42e6b0 2023-11-16T16:52:52.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.345+0000 7fc115704b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55633f5e0c20 con 0x55633f5d8990 2023-11-16T16:52:52.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.345+0000 7fc115704b80 10 client.0 _ll_register_callbacks cb 0x7fc0ec007730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:52:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.345+0000 7fc0fb7fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f42e6b0 0x55633f5e2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.345+0000 7fc0fb7fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f42e6b0 0x55633f5e2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:48422/0 (socket says 172.21.15.2:48422) 2023-11-16T16:52:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.345+0000 7fc0fb7fe700 1 -- 172.21.15.2:0/1093253370 learned_addr learned my addr 172.21.15.2:0/1093253370 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:52:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.345+0000 7fc0faffd700 1 --2- 172.21.15.2:0/1093253370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f5d2cf0 0x55633f5e3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.345+0000 7fc0fbfff700 1 --2- 172.21.15.2:0/1093253370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f5d8990 0x55633f5e01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.345+0000 7fc0fb7fe700 1 -- 172.21.15.2:0/1093253370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f5d8990 msgr2=0x55633f5e01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.346+0000 7fc0fb7fe700 1 --2- 172.21.15.2:0/1093253370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f5d8990 0x55633f5e01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.346+0000 7fc0fb7fe700 1 -- 172.21.15.2:0/1093253370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f5d2cf0 msgr2=0x55633f5e3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:52:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.346+0000 7fc0fb7fe700 1 --2- 172.21.15.2:0/1093253370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f5d2cf0 0x55633f5e3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.346+0000 7fc0fb7fe700 1 -- 172.21.15.2:0/1093253370 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55633f4ea130 con 0x55633f42e6b0 2023-11-16T16:52:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.346+0000 7fc0fb7fe700 1 --2- 172.21.15.2:0/1093253370 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f42e6b0 0x55633f5e2dd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4005fd0 tx=0x7fc0f4009fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2627d24be8205868 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:52:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.346+0000 7fc0daffd700 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2023-11-16T16:52:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.346+0000 7fc0daffd700 1 -- 172.21.15.2:0/1093253370 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f4017040 con 0x55633f42e6b0 2023-11-16T16:52:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.346+0000 7fc0daffd700 1 -- 172.21.15.2:0/1093253370 <== mon.2 v2:172.21.15.141:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fc0f4004230 con 0x55633f42e6b0 2023-11-16T16:52:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.346+0000 7fc115704b80 1 -- 172.21.15.2:0/1093253370 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc0e40105a0 con 0x55633f42e6b0 2023-11-16T16:52:52.350 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.347+0000 7fc0daffd700 1 -- 172.21.15.2:0/1093253370 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0f401b780 con 0x55633f42e6b0 2023-11-16T16:52:52.350 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.347+0000 7fc115704b80 10 client.25709 fetch_fsmap learned FSMap version 32 2023-11-16T16:52:52.350 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.347+0000 7fc115704b80 1 -- 172.21.15.2:0/1093253370 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55633f351d80 con 0x55633f42e6b0 2023-11-16T16:52:52.350 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.347+0000 7fc0daffd700 1 -- 172.21.15.2:0/1093253370 <== mon.2 v2:172.21.15.141:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0e40105a0 con 0x55633f42e6b0 2023-11-16T16:52:52.350 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.347+0000 7fc0daffd700 1 -- 172.21.15.2:0/1093253370 <== mon.2 v2:172.21.15.141:3300/0 5 ==== fsmap(e 32) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fc0f4010b70 con 0x55633f42e6b0 2023-11-16T16:52:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.348+0000 7fc115704b80 10 client.25709 fetch_fsmap finished waiting for FSMap version 32 2023-11-16T16:52:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.348+0000 7fc115704b80 10 client.25709 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:52:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.348+0000 7fc115704b80 20 client.25709 populate_metadata read hostname 'smithi002' 2023-11-16T16:52:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.348+0000 7fc115704b80 1 --2- 172.21.15.2:0/1093253370 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55633f6403c0 0x55633f6427d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:52:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.348+0000 7fc115704b80 4 client.25709 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:52:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.348+0000 7fc115704b80 1 -- 172.21.15.2:0/1093253370 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55633f4f2600 con 0x55633f6403c0 2023-11-16T16:52:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.348+0000 7fc0faffd700 1 --2- 172.21.15.2:0/1093253370 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55633f6403c0 0x55633f6427d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:52:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.349+0000 7fc0faffd700 1 --2- 172.21.15.2:0/1093253370 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55633f6403c0 0x55633f6427d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26269f9bcad75fe1 server_cookie=31a1a1e78e92f9a0 in_seq=0 out_seq=0 2023-11-16T16:52:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:52.349+0000 7fc0daffd700 10 client.25709 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:52:53.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:52 smithi141 ceph-mon[131184]: pgmap v92: 129 pgs: 129 active+clean; 51 MiB data, 203 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 794 op/s 2023-11-16T16:52:53.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:52 smithi155 ceph-mon[131607]: pgmap v92: 129 pgs: 129 active+clean; 51 MiB data, 203 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 794 op/s 2023-11-16T16:52:53.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:52 smithi002 ceph-mon[128712]: pgmap v92: 129 pgs: 129 active+clean; 51 MiB data, 203 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.5 MiB/s wr, 794 op/s 2023-11-16T16:52:55.267 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:54 smithi141 ceph-mon[131184]: pgmap v93: 129 pgs: 129 active+clean; 51 MiB data, 203 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 19 op/s 2023-11-16T16:52:55.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:54 smithi155 ceph-mon[131607]: pgmap v93: 129 pgs: 129 active+clean; 51 MiB data, 203 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 19 op/s 2023-11-16T16:52:55.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:54 smithi002 ceph-mon[128712]: pgmap v93: 129 pgs: 129 active+clean; 51 MiB data, 203 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 19 op/s 2023-11-16T16:52:57.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:56 smithi141 ceph-mon[131184]: pgmap v94: 129 pgs: 129 active+clean; 51 MiB data, 203 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 19 op/s 2023-11-16T16:52:57.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:56 smithi155 ceph-mon[131607]: pgmap v94: 129 pgs: 129 active+clean; 51 MiB data, 203 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 19 op/s 2023-11-16T16:52:57.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:56 smithi002 ceph-mon[128712]: pgmap v94: 129 pgs: 129 active+clean; 51 MiB data, 203 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 19 op/s 2023-11-16T16:52:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.978+0000 7fc0daffd700 1 -- 172.21.15.2:0/1093253370 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55633f4f2600 con 0x55633f6403c0 2023-11-16T16:52:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.978+0000 7fc0daffd700 10 client.25709 handle_command_reply: tid=0 2023-11-16T16:52:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.978+0000 7fc115704b80 1 client.25709 shutdown 2023-11-16T16:52:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.980+0000 7fc115704b80 1 -- 172.21.15.2:0/1093253370 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f42e6b0 msgr2=0x55633f5e2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.980+0000 7fc115704b80 1 --2- 172.21.15.2:0/1093253370 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f42e6b0 0x55633f5e2dd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4005fd0 tx=0x7fc0f4009fc0 comp rx=0 tx=0).stop 2023-11-16T16:52:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.980+0000 7fc115704b80 1 -- 172.21.15.2:0/1093253370 shutdown_connections 2023-11-16T16:52:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.980+0000 7fc115704b80 1 --2- 172.21.15.2:0/1093253370 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55633f6403c0 0x55633f6427d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.980+0000 7fc115704b80 1 --2- 172.21.15.2:0/1093253370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55633f5d8990 0x55633f5e01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.980+0000 7fc115704b80 1 --2- 172.21.15.2:0/1093253370 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55633f42e6b0 0x55633f5e2dd0 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4005fd0 tx=0x7fc0f4009fc0 comp rx=0 tx=0).stop 2023-11-16T16:52:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.980+0000 7fc115704b80 1 --2- 172.21.15.2:0/1093253370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55633f5d2cf0 0x55633f5e3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.980+0000 7fc115704b80 1 -- 172.21.15.2:0/1093253370 >> 172.21.15.2:0/1093253370 conn(0x55633f5c81a0 msgr2=0x55633f5d6160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.980+0000 7fc0daffd700 0 client.25709 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:52:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.980+0000 7fc115704b80 1 -- 172.21.15.2:0/1093253370 shutdown_connections 2023-11-16T16:52:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.981+0000 7fc115704b80 1 -- 172.21.15.2:0/1093253370 wait complete. 2023-11-16T16:52:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.981+0000 7fc115704b80 20 client.25709 trim_cache size 0 max 16384 2023-11-16T16:52:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:52:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:52:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:52:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:52:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.983+0000 7fc100b9d700 1 -- 172.21.15.2:0/242229986 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fc0e80a1340 msgr2=0x7fc0e80a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.983+0000 7fc100b9d700 1 --2- 172.21.15.2:0/242229986 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fc0e80a1340 0x7fc0e80a37f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc1a3370 tx=0x7fc0ec009700 comp rx=0 tx=0).stop 2023-11-16T16:52:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.983+0000 7fc100b9d700 1 -- 172.21.15.2:0/242229986 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0fc07aa80 msgr2=0x7fc0fc1a2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:52:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.983+0000 7fc100b9d700 1 --2- 172.21.15.2:0/242229986 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0fc07aa80 0x7fc0fc1a2bb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400e9a0 tx=0x7fc0e400ecb0 comp rx=0 tx=0).stop 2023-11-16T16:52:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.983+0000 7fc100b9d700 1 -- 172.21.15.2:0/242229986 shutdown_connections 2023-11-16T16:52:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.983+0000 7fc100b9d700 1 --2- 172.21.15.2:0/242229986 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fc0e80a1340 0x7fc0e80a37f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.983+0000 7fc100b9d700 1 --2- 172.21.15.2:0/242229986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc0fc10c6b0 0x7fc0fc19c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.983+0000 7fc100b9d700 1 --2- 172.21.15.2:0/242229986 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc0fc07a0d0 0x7fc0fc1a2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:52:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.983+0000 7fc100b9d700 1 --2- 172.21.15.2:0/242229986 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0fc07aa80 0x7fc0fc1a2bb0 secure :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc0e400e9a0 tx=0x7fc0e400ecb0 comp rx=0 tx=0).stop 2023-11-16T16:52:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.983+0000 7fc100b9d700 1 -- 172.21.15.2:0/242229986 >> 172.21.15.2:0/242229986 conn(0x7fc0fc075bd0 msgr2=0x7fc0fc077280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:52:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.983+0000 7fc100b9d700 1 -- 172.21.15.2:0/242229986 shutdown_connections 2023-11-16T16:52:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:52:58.984+0000 7fc100b9d700 1 -- 172.21.15.2:0/242229986 wait complete. 2023-11-16T16:52:59.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:52:58 smithi141 ceph-mon[131184]: pgmap v95: 129 pgs: 129 active+clean; 61 MiB data, 234 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 659 op/s 2023-11-16T16:52:59.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:52:58 smithi155 ceph-mon[131607]: pgmap v95: 129 pgs: 129 active+clean; 61 MiB data, 234 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 659 op/s 2023-11-16T16:52:59.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:52:58 smithi002 ceph-mon[128712]: pgmap v95: 129 pgs: 129 active+clean; 61 MiB data, 234 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 659 op/s 2023-11-16T16:53:01.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:00 smithi141 ceph-mon[131184]: pgmap v96: 129 pgs: 129 active+clean; 61 MiB data, 234 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 648 op/s 2023-11-16T16:53:01.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:00 smithi155 ceph-mon[131607]: pgmap v96: 129 pgs: 129 active+clean; 61 MiB data, 234 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 648 op/s 2023-11-16T16:53:01.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:00 smithi002 ceph-mon[128712]: pgmap v96: 129 pgs: 129 active+clean; 61 MiB data, 234 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 648 op/s 2023-11-16T16:53:03.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:02 smithi141 ceph-mon[131184]: pgmap v97: 129 pgs: 129 active+clean; 47 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.6 MiB/s wr, 2.72k op/s 2023-11-16T16:53:03.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:02 smithi155 ceph-mon[131607]: pgmap v97: 129 pgs: 129 active+clean; 47 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.6 MiB/s wr, 2.72k op/s 2023-11-16T16:53:03.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:02 smithi002 ceph-mon[128712]: pgmap v97: 129 pgs: 129 active+clean; 47 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.6 MiB/s wr, 2.72k op/s 2023-11-16T16:53:05.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:04 smithi141 ceph-mon[131184]: pgmap v98: 129 pgs: 129 active+clean; 47 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 2.71k op/s 2023-11-16T16:53:05.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:04 smithi155 ceph-mon[131607]: pgmap v98: 129 pgs: 129 active+clean; 47 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 2.71k op/s 2023-11-16T16:53:05.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:04 smithi002 ceph-mon[128712]: pgmap v98: 129 pgs: 129 active+clean; 47 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 2.71k op/s 2023-11-16T16:53:07.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:07 smithi141 ceph-mon[131184]: pgmap v99: 129 pgs: 129 active+clean; 47 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 2.71k op/s 2023-11-16T16:53:07.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:07 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:53:07.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:07 smithi155 ceph-mon[131607]: pgmap v99: 129 pgs: 129 active+clean; 47 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 2.71k op/s 2023-11-16T16:53:07.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:07 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:53:07.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:07 smithi002 ceph-mon[128712]: pgmap v99: 129 pgs: 129 active+clean; 47 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 2.71k op/s 2023-11-16T16:53:07.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:07 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:53:09.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:09 smithi141 ceph-mon[131184]: pgmap v100: 129 pgs: 129 active+clean; 87 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 6.0 MiB/s wr, 2.73k op/s 2023-11-16T16:53:09.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:09 smithi155 ceph-mon[131607]: pgmap v100: 129 pgs: 129 active+clean; 87 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 6.0 MiB/s wr, 2.73k op/s 2023-11-16T16:53:09.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:09 smithi002 ceph-mon[128712]: pgmap v100: 129 pgs: 129 active+clean; 87 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 6.0 MiB/s wr, 2.73k op/s 2023-11-16T16:53:11.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:11 smithi141 ceph-mon[131184]: pgmap v101: 129 pgs: 129 active+clean; 87 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 2.10k op/s 2023-11-16T16:53:11.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:11 smithi155 ceph-mon[131607]: pgmap v101: 129 pgs: 129 active+clean; 87 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 2.10k op/s 2023-11-16T16:53:11.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:11 smithi002 ceph-mon[128712]: pgmap v101: 129 pgs: 129 active+clean; 87 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 2.10k op/s 2023-11-16T16:53:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.183+0000 7f3f45dff700 1 -- 172.21.15.2:0/2688671292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f400ffd60 msgr2=0x7f3f40103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.183+0000 7f3f45dff700 1 --2- 172.21.15.2:0/2688671292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f400ffd60 0x7f3f40103d20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3f340099c0 tx=0x7f3f34009cd0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.183+0000 7f3f45dff700 1 -- 172.21.15.2:0/2688671292 shutdown_connections 2023-11-16T16:53:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.183+0000 7f3f45dff700 1 --2- 172.21.15.2:0/2688671292 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f40104400 0x7f3f4010bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.183+0000 7f3f45dff700 1 --2- 172.21.15.2:0/2688671292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f400ff3c0 0x7f3f400ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.183+0000 7f3f45dff700 1 --2- 172.21.15.2:0/2688671292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f400ffd60 0x7f3f40103d20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.183+0000 7f3f45dff700 1 -- 172.21.15.2:0/2688671292 >> 172.21.15.2:0/2688671292 conn(0x7f3f400763a0 msgr2=0x7f3f400767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.183+0000 7f3f45dff700 1 -- 172.21.15.2:0/2688671292 shutdown_connections 2023-11-16T16:53:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.183+0000 7f3f45dff700 1 -- 172.21.15.2:0/2688671292 wait complete. 2023-11-16T16:53:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.184+0000 7f3f45dff700 1 Processor -- start 2023-11-16T16:53:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.185+0000 7f3f45dff700 1 -- start start 2023-11-16T16:53:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.186+0000 7f3f45dff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f400ff3c0 0x7f3f4019d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.186+0000 7f3f45dff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f400ffd60 0x7f3f4019d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.186+0000 7f3f45dff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f40104400 0x7f3f401989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.186+0000 7f3f45dff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f40199100 con 0x7f3f40104400 2023-11-16T16:53:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.186+0000 7f3f45dff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f40199270 con 0x7f3f400ff3c0 2023-11-16T16:53:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.186+0000 7f3f45dff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f40199430 con 0x7f3f400ffd60 2023-11-16T16:53:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.186+0000 7f3f3f7fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f400ff3c0 0x7f3f4019d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.186+0000 7f3f3f7fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f400ff3c0 0x7f3f4019d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:34426/0 (socket says 172.21.15.2:34426) 2023-11-16T16:53:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.186+0000 7f3f3f7fe700 1 -- 172.21.15.2:0/766620783 learned_addr learned my addr 172.21.15.2:0/766620783 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.187+0000 7f3f3effd700 1 --2- 172.21.15.2:0/766620783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f400ffd60 0x7f3f4019d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.187+0000 7f3f3ffff700 1 --2- 172.21.15.2:0/766620783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f40104400 0x7f3f401989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.187+0000 7f3f3f7fe700 1 -- 172.21.15.2:0/766620783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f400ffd60 msgr2=0x7f3f4019d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.187+0000 7f3f3f7fe700 1 --2- 172.21.15.2:0/766620783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f400ffd60 0x7f3f4019d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.187+0000 7f3f3f7fe700 1 -- 172.21.15.2:0/766620783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f40104400 msgr2=0x7f3f401989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.187+0000 7f3f3f7fe700 1 --2- 172.21.15.2:0/766620783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f40104400 0x7f3f401989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.187+0000 7f3f3f7fe700 1 -- 172.21.15.2:0/766620783 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f340095f0 con 0x7f3f400ff3c0 2023-11-16T16:53:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.187+0000 7f3f3f7fe700 1 --2- 172.21.15.2:0/766620783 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f400ff3c0 0x7f3f4019d2d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3f3000d980 tx=0x7f3f3000dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.187+0000 7f3f3cff9700 1 -- 172.21.15.2:0/766620783 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3000bb60 con 0x7f3f400ff3c0 2023-11-16T16:53:12.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.188+0000 7f3f3cff9700 1 -- 172.21.15.2:0/766620783 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f30007640 con 0x7f3f400ff3c0 2023-11-16T16:53:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.188+0000 7f3f3cff9700 1 -- 172.21.15.2:0/766620783 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f30004e90 con 0x7f3f400ff3c0 2023-11-16T16:53:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.188+0000 7f3f45dff700 1 -- 172.21.15.2:0/766620783 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4010e100 con 0x7f3f400ff3c0 2023-11-16T16:53:12.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.188+0000 7f3f45dff700 1 -- 172.21.15.2:0/766620783 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4010df70 con 0x7f3f400ff3c0 2023-11-16T16:53:12.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.192+0000 7f3f3cff9700 1 -- 172.21.15.2:0/766620783 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f3f3000bcc0 con 0x7f3f400ff3c0 2023-11-16T16:53:12.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.193+0000 7f3f597f6b80 1 Processor -- start 2023-11-16T16:53:12.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.193+0000 7f3f597f6b80 1 -- start start 2023-11-16T16:53:12.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.194+0000 7f3f3cff9700 1 --2- 172.21.15.2:0/766620783 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f3f280a1340 0x7f3f280a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:12.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.194+0000 7f3f3cff9700 1 -- 172.21.15.2:0/766620783 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f3f300ccee0 con 0x7f3f400ff3c0 2023-11-16T16:53:12.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.195+0000 7f3f3ffff700 1 --2- 172.21.15.2:0/766620783 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f3f280a1340 0x7f3f280a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.195+0000 7f3f597f6b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654569296b0 0x565456ad31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.195+0000 7f3f597f6b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x565456ad3720 0x565456acd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.195+0000 7f3f597f6b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3b90 0x565456acdb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.195+0000 7f3f597f6b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654569d3000 con 0x565456ad3b90 2023-11-16T16:53:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.195+0000 7f3f597f6b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x565456a25840 con 0x5654569296b0 2023-11-16T16:53:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.195+0000 7f3f597f6b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x565456a1ecd0 con 0x565456ad3720 2023-11-16T16:53:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.195+0000 7f3f3f7fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654569296b0 0x565456ad31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.195+0000 7f3f3f7fe700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654569296b0 0x565456ad31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:34430/0 (socket says 172.21.15.2:34430) 2023-11-16T16:53:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.195+0000 7f3f3f7fe700 1 -- 172.21.15.2:0/4090559770 learned_addr learned my addr 172.21.15.2:0/4090559770 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.195+0000 7f3f3effd700 1 --2- 172.21.15.2:0/4090559770 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x565456ad3720 0x565456acd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.196+0000 7f3f3f7fe700 1 -- 172.21.15.2:0/4090559770 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x565456ad3720 msgr2=0x565456acd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.196+0000 7f3f3f7fe700 1 --2- 172.21.15.2:0/4090559770 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x565456ad3720 0x565456acd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.196+0000 7f3f3f7fe700 1 -- 172.21.15.2:0/4090559770 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3b90 msgr2=0x565456acdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.196+0000 7f3f3ffff700 1 --2- 172.21.15.2:0/4090559770 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3b90 0x565456acdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.196+0000 7f3f3f7fe700 1 --2- 172.21.15.2:0/4090559770 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3b90 0x565456acdb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.196+0000 7f3f3f7fe700 1 -- 172.21.15.2:0/4090559770 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654568a05d0 con 0x5654569296b0 2023-11-16T16:53:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.196+0000 7f3f3ffff700 1 --2- 172.21.15.2:0/4090559770 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3b90 0x565456acdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:53:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.196+0000 7f3f3ffff700 1 --2- 172.21.15.2:0/766620783 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f3f280a1340 0x7f3f280a37f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009380 tx=0x7f3f3800b040 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.196+0000 7f3f3f7fe700 1 --2- 172.21.15.2:0/4090559770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654569296b0 0x565456ad31e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3f300a1b00 tx=0x7f3f3000df30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2af50f4b51aa7c0a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.196+0000 7f3f25ffb700 1 -- 172.21.15.2:0/4090559770 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3009dc10 con 0x5654569296b0 2023-11-16T16:53:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.196+0000 7f3f25ffb700 1 -- 172.21.15.2:0/4090559770 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f300a2030 con 0x5654569296b0 2023-11-16T16:53:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.197+0000 7f3f25ffb700 1 -- 172.21.15.2:0/4090559770 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f300990d0 con 0x5654569296b0 2023-11-16T16:53:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.197+0000 7f3f597f6b80 1 -- 172.21.15.2:0/4090559770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654569296b0 msgr2=0x565456ad31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.197+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/4090559770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654569296b0 0x565456ad31e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3f300a1b00 tx=0x7f3f3000df30 comp rx=0 tx=0).stop 2023-11-16T16:53:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.197+0000 7f3f597f6b80 1 -- 172.21.15.2:0/4090559770 shutdown_connections 2023-11-16T16:53:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.197+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/4090559770 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3b90 0x565456acdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.197+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/4090559770 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654569296b0 0x565456ad31e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.197+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/4090559770 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x565456ad3720 0x565456acd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.197+0000 7f3f597f6b80 1 -- 172.21.15.2:0/4090559770 >> 172.21.15.2:0/4090559770 conn(0x565456ac2f50 msgr2=0x565456ac8270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.198+0000 7f3f597f6b80 1 -- 172.21.15.2:0/4090559770 shutdown_connections 2023-11-16T16:53:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.198+0000 7f3f597f6b80 1 -- 172.21.15.2:0/4090559770 wait complete. 2023-11-16T16:53:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.198+0000 7f3f597f6b80 1 -- start start 2023-11-16T16:53:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.198+0000 7f3f597f6b80 1 Processor -- start 2023-11-16T16:53:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.199+0000 7f3f597f6b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654569296b0 0x565456ad6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.199+0000 7f3f597f6b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3720 0x565456ad7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:12.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.199+0000 7f3f3f7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654569296b0 0x565456ad6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:12.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.199+0000 7f3f3f7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654569296b0 0x565456ad6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:60506/0 (socket says 172.21.15.2:60506) 2023-11-16T16:53:12.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.199+0000 7f3f597f6b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x565456ad3b90 0x565456ad7860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:12.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.200+0000 7f3f3effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3720 0x565456ad7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:12.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.200+0000 7f3f597f6b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654569c0880 con 0x565456ad3720 2023-11-16T16:53:12.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.200+0000 7f3f597f6b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654569c0520 con 0x565456ad3b90 2023-11-16T16:53:12.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.200+0000 7f3f597f6b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654568a28c0 con 0x5654569296b0 2023-11-16T16:53:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.200+0000 7f3f3effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3720 0x565456ad7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42602/0 (socket says 172.21.15.2:42602) 2023-11-16T16:53:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.200+0000 7f3f3effd700 1 -- 172.21.15.2:0/1363246631 learned_addr learned my addr 172.21.15.2:0/1363246631 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.200+0000 7f3f3ffff700 1 --2- 172.21.15.2:0/1363246631 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x565456ad3b90 0x565456ad7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.200+0000 7f3f597f6b80 10 client.0 _ll_register_callbacks cb 0x5654569de720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.200+0000 7f3f3f7fe700 1 -- 172.21.15.2:0/1363246631 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x565456ad3b90 msgr2=0x565456ad7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.200+0000 7f3f3f7fe700 1 --2- 172.21.15.2:0/1363246631 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x565456ad3b90 0x565456ad7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.200+0000 7f3f3f7fe700 1 -- 172.21.15.2:0/1363246631 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3720 msgr2=0x565456ad7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.200+0000 7f3f3f7fe700 1 --2- 172.21.15.2:0/1363246631 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3720 0x565456ad7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.200+0000 7f3f3f7fe700 1 -- 172.21.15.2:0/1363246631 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4010df70 con 0x5654569296b0 2023-11-16T16:53:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.200+0000 7f3f3f7fe700 1 --2- 172.21.15.2:0/1363246631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654569296b0 0x565456ad6de0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3f30096650 tx=0x7f3f30099d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3aa939a38da863d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.200+0000 7f3f1effd700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:53:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.201+0000 7f3f1effd700 1 -- 172.21.15.2:0/1363246631 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3009dc10 con 0x5654569296b0 2023-11-16T16:53:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.201+0000 7f3f1effd700 1 -- 172.21.15.2:0/1363246631 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f3f300992e0 con 0x5654569296b0 2023-11-16T16:53:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.201+0000 7f3f597f6b80 1 -- 172.21.15.2:0/1363246631 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565456ac0710 con 0x5654569296b0 2023-11-16T16:53:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.201+0000 7f3f1effd700 1 -- 172.21.15.2:0/1363246631 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f30099e40 con 0x5654569296b0 2023-11-16T16:53:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.201+0000 7f3f597f6b80 10 client.25690 fetch_fsmap learned FSMap version 34 2023-11-16T16:53:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.201+0000 7f3f597f6b80 1 -- 172.21.15.2:0/1363246631 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x5654569e5130 con 0x5654569296b0 2023-11-16T16:53:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.201+0000 7f3f1effd700 1 -- 172.21.15.2:0/1363246631 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x565456ac0710 con 0x5654569296b0 2023-11-16T16:53:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.201+0000 7f3f1effd700 1 -- 172.21.15.2:0/1363246631 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 34) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f3f30078ab0 con 0x5654569296b0 2023-11-16T16:53:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.201+0000 7f3f597f6b80 10 client.25690 fetch_fsmap finished waiting for FSMap version 34 2023-11-16T16:53:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.202+0000 7f3f597f6b80 10 client.25690 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.202+0000 7f3f597f6b80 20 client.25690 populate_metadata read hostname 'smithi002' 2023-11-16T16:53:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.202+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/1363246631 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x565456aef360 0x565456af1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.202+0000 7f3f597f6b80 4 client.25690 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:53:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.202+0000 7f3f597f6b80 1 -- 172.21.15.2:0/1363246631 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5654569e1040 con 0x565456aef360 2023-11-16T16:53:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.202+0000 7f3f3effd700 1 --2- 172.21.15.2:0/1363246631 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x565456aef360 0x565456af1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.203+0000 7f3f1effd700 10 client.25690 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:53:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.203+0000 7f3f3effd700 1 --2- 172.21.15.2:0/1363246631 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x565456aef360 0x565456af1740 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4e71fc850163455 server_cookie=e70fce6f6138383b in_seq=0 out_seq=0 2023-11-16T16:53:12.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.204+0000 7f3f1effd700 1 -- 172.21.15.2:0/1363246631 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5654569e1040 con 0x565456aef360 2023-11-16T16:53:12.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.204+0000 7f3f1effd700 10 client.25690 handle_command_reply: tid=0 2023-11-16T16:53:12.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.204+0000 7f3f597f6b80 1 client.25690 shutdown 2023-11-16T16:53:12.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.204+0000 7f3f597f6b80 1 -- 172.21.15.2:0/1363246631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654569296b0 msgr2=0x565456ad6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:12.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.204+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/1363246631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654569296b0 0x565456ad6de0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3f30096650 tx=0x7f3f30099d60 comp rx=0 tx=0).stop 2023-11-16T16:53:12.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.204+0000 7f3f597f6b80 1 -- 172.21.15.2:0/1363246631 shutdown_connections 2023-11-16T16:53:12.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.204+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/1363246631 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x565456aef360 0x565456af1740 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.204+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/1363246631 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654569296b0 0x565456ad6de0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.204+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/1363246631 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x565456ad3b90 0x565456ad7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.204+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/1363246631 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3720 0x565456ad7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.204+0000 7f3f1effd700 0 client.25690 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:53:12.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.204+0000 7f3f597f6b80 1 -- 172.21.15.2:0/1363246631 >> 172.21.15.2:0/1363246631 conn(0x565456ac2f50 msgr2=0x565456ac77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:12.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.204+0000 7f3f597f6b80 1 -- 172.21.15.2:0/1363246631 shutdown_connections 2023-11-16T16:53:12.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.204+0000 7f3f597f6b80 1 -- 172.21.15.2:0/1363246631 wait complete. 2023-11-16T16:53:12.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.205+0000 7f3f597f6b80 20 client.25690 trim_cache size 0 max 16384 2023-11-16T16:53:12.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.223+0000 7f3f597f6b80 1 Processor -- start 2023-11-16T16:53:12.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.224+0000 7f3f597f6b80 1 -- start start 2023-11-16T16:53:12.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.224+0000 7f3f597f6b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654569296b0 0x565456ae68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:12.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.224+0000 7f3f597f6b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3720 0x565456ac7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:12.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.224+0000 7f3f597f6b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x565456ad3b90 0x565456ad7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.224+0000 7f3f597f6b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x565456ac8d60 con 0x565456ad3720 2023-11-16T16:53:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.224+0000 7f3f597f6b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x565456ad1ab0 con 0x565456ad3b90 2023-11-16T16:53:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.224+0000 7f3f597f6b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x565456ad8270 con 0x5654569296b0 2023-11-16T16:53:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.224+0000 7f3f3effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3720 0x565456ac7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.224+0000 7f3f3effd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3720 0x565456ac7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42616/0 (socket says 172.21.15.2:42616) 2023-11-16T16:53:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.224+0000 7f3f3f7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654569296b0 0x565456ae68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.224+0000 7f3f3effd700 1 -- 172.21.15.2:0/3395671177 learned_addr learned my addr 172.21.15.2:0/3395671177 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.224+0000 7f3f3ffff700 1 --2- 172.21.15.2:0/3395671177 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x565456ad3b90 0x565456ad7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.225+0000 7f3f3effd700 1 -- 172.21.15.2:0/3395671177 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654569296b0 msgr2=0x565456ae68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.225+0000 7f3f3effd700 1 --2- 172.21.15.2:0/3395671177 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654569296b0 0x565456ae68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.225+0000 7f3f3effd700 1 -- 172.21.15.2:0/3395671177 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x565456ad3b90 msgr2=0x565456ad7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.225+0000 7f3f3effd700 1 --2- 172.21.15.2:0/3395671177 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x565456ad3b90 0x565456ad7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.225+0000 7f3f3effd700 1 -- 172.21.15.2:0/3395671177 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654569e4e80 con 0x565456ad3720 2023-11-16T16:53:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.225+0000 7f3f3effd700 1 --2- 172.21.15.2:0/3395671177 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3720 0x565456ac7f60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3f340058e0 tx=0x7f3f34005170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd14cd366d09981b server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.225+0000 7f3f267fc700 1 -- 172.21.15.2:0/3395671177 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f34029070 con 0x565456ad3720 2023-11-16T16:53:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.225+0000 7f3f267fc700 1 -- 172.21.15.2:0/3395671177 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f340037d0 con 0x565456ad3720 2023-11-16T16:53:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.226+0000 7f3f267fc700 1 -- 172.21.15.2:0/3395671177 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3401d040 con 0x565456ad3720 2023-11-16T16:53:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.226+0000 7f3f597f6b80 1 -- 172.21.15.2:0/3395671177 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3720 msgr2=0x565456ac7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.226+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/3395671177 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3720 0x565456ac7f60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3f340058e0 tx=0x7f3f34005170 comp rx=0 tx=0).stop 2023-11-16T16:53:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.226+0000 7f3f597f6b80 1 -- 172.21.15.2:0/3395671177 shutdown_connections 2023-11-16T16:53:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.226+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/3395671177 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654569296b0 0x565456ae68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.226+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/3395671177 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x565456ad3b90 0x565456ad7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.226+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/3395671177 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3720 0x565456ac7f60 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.226+0000 7f3f597f6b80 1 -- 172.21.15.2:0/3395671177 >> 172.21.15.2:0/3395671177 conn(0x565456ac2f50 msgr2=0x565456ad0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:12.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.227+0000 7f3f597f6b80 1 -- 172.21.15.2:0/3395671177 shutdown_connections 2023-11-16T16:53:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.228+0000 7f3f597f6b80 1 -- 172.21.15.2:0/3395671177 wait complete. 2023-11-16T16:53:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.228+0000 7f3f597f6b80 1 -- start start 2023-11-16T16:53:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.228+0000 7f3f597f6b80 1 Processor -- start 2023-11-16T16:53:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.229+0000 7f3f597f6b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654569296b0 0x565456ae4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.229+0000 7f3f597f6b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3720 0x565456ae5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.229+0000 7f3f597f6b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x565456ad3b90 0x565456b421f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.229+0000 7f3f597f6b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x565456b42730 con 0x565456ad3720 2023-11-16T16:53:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.229+0000 7f3f597f6b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x565456b42870 con 0x565456ad3b90 2023-11-16T16:53:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.229+0000 7f3f597f6b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x565456b429b0 con 0x5654569296b0 2023-11-16T16:53:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.229+0000 7f3f597f6b80 10 client.0 _ll_register_callbacks cb 0x7f3f34009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.229+0000 7f3f3f7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654569296b0 0x565456ae4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.229+0000 7f3f3ffff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x565456ad3b90 0x565456b421f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.229+0000 7f3f3f7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654569296b0 0x565456ae4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:60534/0 (socket says 172.21.15.2:60534) 2023-11-16T16:53:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.229+0000 7f3f3ffff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x565456ad3b90 0x565456b421f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:34456/0 (socket says 172.21.15.2:34456) 2023-11-16T16:53:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.229+0000 7f3f3f7fe700 1 -- 172.21.15.2:0/926409145 learned_addr learned my addr 172.21.15.2:0/926409145 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.229+0000 7f3f3effd700 1 --2- 172.21.15.2:0/926409145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3720 0x565456ae5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.229+0000 7f3f3f7fe700 1 -- 172.21.15.2:0/926409145 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x565456ad3b90 msgr2=0x565456b421f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.229+0000 7f3f3f7fe700 1 --2- 172.21.15.2:0/926409145 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x565456ad3b90 0x565456b421f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.229+0000 7f3f3f7fe700 1 -- 172.21.15.2:0/926409145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3720 msgr2=0x565456ae5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.229+0000 7f3f3f7fe700 1 --2- 172.21.15.2:0/926409145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3720 0x565456ae5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.229+0000 7f3f3f7fe700 1 -- 172.21.15.2:0/926409145 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4010c810 con 0x5654569296b0 2023-11-16T16:53:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.230+0000 7f3f3effd700 1 --2- 172.21.15.2:0/926409145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3720 0x565456ae5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:53:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.230+0000 7f3f3f7fe700 1 --2- 172.21.15.2:0/926409145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654569296b0 0x565456ae4ac0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3f30000c00 tx=0x7f3f3000b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca13f1bf1e7bfdfb server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.230+0000 7f3f1effd700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:53:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.230+0000 7f3f1effd700 1 -- 172.21.15.2:0/926409145 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f300b5f00 con 0x5654569296b0 2023-11-16T16:53:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.230+0000 7f3f1effd700 1 -- 172.21.15.2:0/926409145 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f3f3009dc10 con 0x5654569296b0 2023-11-16T16:53:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.230+0000 7f3f597f6b80 1 -- 172.21.15.2:0/926409145 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3f300a2030 con 0x5654569296b0 2023-11-16T16:53:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.230+0000 7f3f1effd700 1 -- 172.21.15.2:0/926409145 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f300ca9b0 con 0x5654569296b0 2023-11-16T16:53:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.230+0000 7f3f597f6b80 10 client.25693 fetch_fsmap learned FSMap version 34 2023-11-16T16:53:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.230+0000 7f3f597f6b80 1 -- 172.21.15.2:0/926409145 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x565456835c00 con 0x5654569296b0 2023-11-16T16:53:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.231+0000 7f3f1effd700 1 -- 172.21.15.2:0/926409145 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f300a2030 con 0x5654569296b0 2023-11-16T16:53:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.231+0000 7f3f1effd700 1 -- 172.21.15.2:0/926409145 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 34) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f3f30099e40 con 0x5654569296b0 2023-11-16T16:53:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.231+0000 7f3f597f6b80 10 client.25693 fetch_fsmap finished waiting for FSMap version 34 2023-11-16T16:53:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.231+0000 7f3f597f6b80 10 client.25693 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.231+0000 7f3f597f6b80 20 client.25693 populate_metadata read hostname 'smithi002' 2023-11-16T16:53:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.231+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/926409145 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x565456b43bc0 0x565456b45fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.232+0000 7f3f597f6b80 4 client.25693 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:53:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.232+0000 7f3f597f6b80 1 -- 172.21.15.2:0/926409145 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5654569ed600 con 0x565456b43bc0 2023-11-16T16:53:12.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.232+0000 7f3f3effd700 1 --2- 172.21.15.2:0/926409145 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x565456b43bc0 0x565456b45fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:12.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.232+0000 7f3f3effd700 1 --2- 172.21.15.2:0/926409145 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x565456b43bc0 0x565456b45fa0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1045402e2419b3a server_cookie=6efb32341f4ed91d in_seq=0 out_seq=0 2023-11-16T16:53:12.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:12.232+0000 7f3f1effd700 10 client.25693 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:53:12.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:12 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:53:12.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:12 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:53:12.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:12 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:53:13.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:13 smithi155 ceph-mon[131607]: pgmap v102: 129 pgs: 129 active+clean; 132 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 8.2 MiB/s wr, 2.12k op/s 2023-11-16T16:53:13.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:13 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:53:13.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:13 smithi002 ceph-mon[128712]: pgmap v102: 129 pgs: 129 active+clean; 132 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 8.2 MiB/s wr, 2.12k op/s 2023-11-16T16:53:13.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:13 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:53:13.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:13 smithi141 ceph-mon[131184]: pgmap v102: 129 pgs: 129 active+clean; 132 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 8.2 MiB/s wr, 2.12k op/s 2023-11-16T16:53:13.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:13 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:53:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.130+0000 7f3f1effd700 1 -- 172.21.15.2:0/926409145 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5654569ed600 con 0x565456b43bc0 2023-11-16T16:53:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.130+0000 7f3f1effd700 10 client.25693 handle_command_reply: tid=0 2023-11-16T16:53:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.130+0000 7f3f597f6b80 1 client.25693 shutdown 2023-11-16T16:53:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.130+0000 7f3f597f6b80 1 -- 172.21.15.2:0/926409145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654569296b0 msgr2=0x565456ae4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:14.133 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.130+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/926409145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654569296b0 0x565456ae4ac0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3f30000c00 tx=0x7f3f3000b670 comp rx=0 tx=0).stop 2023-11-16T16:53:14.134 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.130+0000 7f3f597f6b80 1 -- 172.21.15.2:0/926409145 shutdown_connections 2023-11-16T16:53:14.134 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.130+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/926409145 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x565456b43bc0 0x565456b45fa0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:14.134 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.130+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/926409145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654569296b0 0x565456ae4ac0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:14.134 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.130+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/926409145 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x565456ad3b90 0x565456b421f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:14.134 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.130+0000 7f3f597f6b80 1 --2- 172.21.15.2:0/926409145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565456ad3720 0x565456ae5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:14.134 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.130+0000 7f3f1effd700 0 client.25693 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:53:14.134 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.130+0000 7f3f597f6b80 1 -- 172.21.15.2:0/926409145 >> 172.21.15.2:0/926409145 conn(0x565456ac2f50 msgr2=0x565456ad3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:14.135 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.130+0000 7f3f597f6b80 1 -- 172.21.15.2:0/926409145 shutdown_connections 2023-11-16T16:53:14.135 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.130+0000 7f3f597f6b80 1 -- 172.21.15.2:0/926409145 wait complete. 2023-11-16T16:53:14.135 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.131+0000 7f3f597f6b80 20 client.25693 trim_cache size 0 max 16384 2023-11-16T16:53:14.135 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:53:14.135 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:53:14.136 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:53:14.136 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:53:14.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.131+0000 7f3f45dff700 1 -- 172.21.15.2:0/766620783 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f3f280a1340 msgr2=0x7f3f280a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:14.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.131+0000 7f3f45dff700 1 --2- 172.21.15.2:0/766620783 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f3f280a1340 0x7f3f280a37f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009380 tx=0x7f3f3800b040 comp rx=0 tx=0).stop 2023-11-16T16:53:14.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.131+0000 7f3f45dff700 1 -- 172.21.15.2:0/766620783 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f400ff3c0 msgr2=0x7f3f4019d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:14.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.131+0000 7f3f45dff700 1 --2- 172.21.15.2:0/766620783 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f400ff3c0 0x7f3f4019d2d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3f3000d980 tx=0x7f3f3000dd40 comp rx=0 tx=0).stop 2023-11-16T16:53:14.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.131+0000 7f3f45dff700 1 -- 172.21.15.2:0/766620783 shutdown_connections 2023-11-16T16:53:14.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.131+0000 7f3f45dff700 1 --2- 172.21.15.2:0/766620783 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f3f280a1340 0x7f3f280a37f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:14.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.131+0000 7f3f45dff700 1 --2- 172.21.15.2:0/766620783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f40104400 0x7f3f401989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:14.138 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.131+0000 7f3f45dff700 1 --2- 172.21.15.2:0/766620783 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3f400ff3c0 0x7f3f4019d2d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:14.138 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.131+0000 7f3f45dff700 1 --2- 172.21.15.2:0/766620783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f400ffd60 0x7f3f4019d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:14.138 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.131+0000 7f3f45dff700 1 -- 172.21.15.2:0/766620783 >> 172.21.15.2:0/766620783 conn(0x7f3f400763a0 msgr2=0x7f3f40102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:14.138 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.132+0000 7f3f45dff700 1 -- 172.21.15.2:0/766620783 shutdown_connections 2023-11-16T16:53:14.138 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:14.132+0000 7f3f45dff700 1 -- 172.21.15.2:0/766620783 wait complete. 2023-11-16T16:53:15.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:15 smithi155 ceph-mon[131607]: pgmap v103: 129 pgs: 129 active+clean; 132 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.2 MiB/s wr, 42 op/s 2023-11-16T16:53:15.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:15 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:53:15.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:15 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:53:15.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:15 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:53:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:15 smithi002 ceph-mon[128712]: pgmap v103: 129 pgs: 129 active+clean; 132 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.2 MiB/s wr, 42 op/s 2023-11-16T16:53:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:15 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:53:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:15 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:53:15.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:15 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:53:15.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:15 smithi141 ceph-mon[131184]: pgmap v103: 129 pgs: 129 active+clean; 132 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.2 MiB/s wr, 42 op/s 2023-11-16T16:53:15.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:15 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:53:15.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:15 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:53:15.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:15 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:53:16.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:16 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:53:16.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:16 smithi155 ceph-mon[131607]: Service mds.all has no related daemon already placed. 2023-11-16T16:53:16.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:16 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:53:16.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:16 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:53:16.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:16 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:53:16.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:16 smithi002 ceph-mon[128712]: Service mds.all has no related daemon already placed. 2023-11-16T16:53:16.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:16 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:53:16.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:16 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:53:16.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:16 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:53:16.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:16 smithi141 ceph-mon[131184]: Service mds.all has no related daemon already placed. 2023-11-16T16:53:16.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:16 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:53:16.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:16 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:53:17.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:17 smithi155 ceph-mon[131607]: pgmap v104: 129 pgs: 129 active+clean; 132 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.2 MiB/s wr, 42 op/s 2023-11-16T16:53:17.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:17 smithi002 ceph-mon[128712]: pgmap v104: 129 pgs: 129 active+clean; 132 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.2 MiB/s wr, 42 op/s 2023-11-16T16:53:17.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:17 smithi141 ceph-mon[131184]: pgmap v104: 129 pgs: 129 active+clean; 132 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.2 MiB/s wr, 42 op/s 2023-11-16T16:53:19.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:19 smithi155 ceph-mon[131607]: pgmap v105: 129 pgs: 129 active+clean; 123 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 529 op/s 2023-11-16T16:53:19.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:19 smithi002 ceph-mon[128712]: pgmap v105: 129 pgs: 129 active+clean; 123 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 529 op/s 2023-11-16T16:53:19.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:19 smithi141 ceph-mon[131184]: pgmap v105: 129 pgs: 129 active+clean; 123 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 529 op/s 2023-11-16T16:53:21.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:21 smithi155 ceph-mon[131607]: pgmap v106: 129 pgs: 129 active+clean; 123 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 7.5 MiB/s wr, 507 op/s 2023-11-16T16:53:21.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:21 smithi002 ceph-mon[128712]: pgmap v106: 129 pgs: 129 active+clean; 123 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 7.5 MiB/s wr, 507 op/s 2023-11-16T16:53:21.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:21 smithi141 ceph-mon[131184]: pgmap v106: 129 pgs: 129 active+clean; 123 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 7.5 MiB/s wr, 507 op/s 2023-11-16T16:53:23.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:23 smithi155 ceph-mon[131607]: pgmap v107: 129 pgs: 129 active+clean; 155 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 11 MiB/s wr, 681 op/s 2023-11-16T16:53:23.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:23 smithi002 ceph-mon[128712]: pgmap v107: 129 pgs: 129 active+clean; 155 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 11 MiB/s wr, 681 op/s 2023-11-16T16:53:23.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:23 smithi141 ceph-mon[131184]: pgmap v107: 129 pgs: 129 active+clean; 155 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 11 MiB/s wr, 681 op/s 2023-11-16T16:53:25.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:25 smithi155 ceph-mon[131607]: pgmap v108: 129 pgs: 129 active+clean; 155 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 7.2 MiB/s wr, 659 op/s 2023-11-16T16:53:25.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:25 smithi002 ceph-mon[128712]: pgmap v108: 129 pgs: 129 active+clean; 155 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 7.2 MiB/s wr, 659 op/s 2023-11-16T16:53:25.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:25 smithi141 ceph-mon[131184]: pgmap v108: 129 pgs: 129 active+clean; 155 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 7.2 MiB/s wr, 659 op/s 2023-11-16T16:53:27.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.313+0000 7ff6af7ff700 1 -- 172.21.15.2:0/1521588294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6a8101440 msgr2=0x7ff6a8101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:27.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.313+0000 7ff6af7ff700 1 --2- 172.21.15.2:0/1521588294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6a8101440 0x7ff6a8101810 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4009990 tx=0x7ff6a4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.313+0000 7ff6af7ff700 1 -- 172.21.15.2:0/1521588294 shutdown_connections 2023-11-16T16:53:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.313+0000 7ff6af7ff700 1 --2- 172.21.15.2:0/1521588294 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6a810a810 0x7ff6a810cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.313+0000 7ff6af7ff700 1 --2- 172.21.15.2:0/1521588294 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6a8101440 0x7ff6a8101810 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.313+0000 7ff6af7ff700 1 --2- 172.21.15.2:0/1521588294 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6a8101d50 0x7ff6a810a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.313+0000 7ff6af7ff700 1 -- 172.21.15.2:0/1521588294 >> 172.21.15.2:0/1521588294 conn(0x7ff6a80763a0 msgr2=0x7ff6a80767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.313+0000 7ff6af7ff700 1 -- 172.21.15.2:0/1521588294 shutdown_connections 2023-11-16T16:53:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.313+0000 7ff6af7ff700 1 -- 172.21.15.2:0/1521588294 wait complete. 2023-11-16T16:53:27.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.315+0000 7ff6af7ff700 1 Processor -- start 2023-11-16T16:53:27.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.315+0000 7ff6af7ff700 1 -- start start 2023-11-16T16:53:27.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.315+0000 7ff6af7ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6a8101440 0x7ff6a8196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:27.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.315+0000 7ff6af7ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6a8101d50 0x7ff6a81968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:27.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.315+0000 7ff6af7ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6a810a810 0x7ff6a819ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:27.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.315+0000 7ff6af7ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a819b350 con 0x7ff6a8101440 2023-11-16T16:53:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.315+0000 7ff6af7ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a819b4c0 con 0x7ff6a810a810 2023-11-16T16:53:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.315+0000 7ff6af7ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a819b680 con 0x7ff6a8101d50 2023-11-16T16:53:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.316+0000 7ff6acd9a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6a8101d50 0x7ff6a81968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.316+0000 7ff6acd9a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6a8101d50 0x7ff6a81968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:42620/0 (socket says 172.21.15.2:42620) 2023-11-16T16:53:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.316+0000 7ff6acd9a700 1 -- 172.21.15.2:0/768872885 learned_addr learned my addr 172.21.15.2:0/768872885 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.316+0000 7ff6ad59b700 1 --2- 172.21.15.2:0/768872885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6a8101440 0x7ff6a8196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.316+0000 7ff6add9c700 1 --2- 172.21.15.2:0/768872885 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6a810a810 0x7ff6a819ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.316+0000 7ff6acd9a700 1 -- 172.21.15.2:0/768872885 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6a810a810 msgr2=0x7ff6a819ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.316+0000 7ff6acd9a700 1 --2- 172.21.15.2:0/768872885 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6a810a810 0x7ff6a819ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.316+0000 7ff6acd9a700 1 -- 172.21.15.2:0/768872885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6a8101440 msgr2=0x7ff6a8196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.316+0000 7ff6acd9a700 1 --2- 172.21.15.2:0/768872885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6a8101440 0x7ff6a8196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.316+0000 7ff6acd9a700 1 -- 172.21.15.2:0/768872885 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a40095f0 con 0x7ff6a8101d50 2023-11-16T16:53:27.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.316+0000 7ff6acd9a700 1 --2- 172.21.15.2:0/768872885 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6a8101d50 0x7ff6a81968b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff69800e9a0 tx=0x7ff69800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:27.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.317+0000 7ff69e7fc700 1 -- 172.21.15.2:0/768872885 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff698009790 con 0x7ff6a8101d50 2023-11-16T16:53:27.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.317+0000 7ff69e7fc700 1 -- 172.21.15.2:0/768872885 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff698010430 con 0x7ff6a8101d50 2023-11-16T16:53:27.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.317+0000 7ff69e7fc700 1 -- 172.21.15.2:0/768872885 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff698018680 con 0x7ff6a8101d50 2023-11-16T16:53:27.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.317+0000 7ff6af7ff700 1 -- 172.21.15.2:0/768872885 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6a81102c0 con 0x7ff6a8101d50 2023-11-16T16:53:27.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.317+0000 7ff6af7ff700 1 -- 172.21.15.2:0/768872885 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6a8110130 con 0x7ff6a8101d50 2023-11-16T16:53:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.322+0000 7ff69e7fc700 1 -- 172.21.15.2:0/768872885 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7ff6980105a0 con 0x7ff6a8101d50 2023-11-16T16:53:27.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.323+0000 7ff6c307fb80 1 Processor -- start 2023-11-16T16:53:27.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.323+0000 7ff6c307fb80 1 -- start start 2023-11-16T16:53:27.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.323+0000 7ff6c307fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d4d06b0 0x55a23d67a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:27.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.323+0000 7ff6c307fb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67a720 0x55a23d674560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:27.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.323+0000 7ff6c307fb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d67ab90 0x55a23d674b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:27.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.323+0000 7ff6c307fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a23d57a000 con 0x55a23d4d06b0 2023-11-16T16:53:27.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.323+0000 7ff6c307fb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a23d5cc840 con 0x55a23d67a720 2023-11-16T16:53:27.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.323+0000 7ff6c307fb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a23d5c5cd0 con 0x55a23d67ab90 2023-11-16T16:53:27.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.323+0000 7ff69e7fc700 1 --2- 172.21.15.2:0/768872885 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ff6940a1340 0x7ff6940a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:27.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.324+0000 7ff6ad59b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67a720 0x55a23d674560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.324+0000 7ff6ad59b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67a720 0x55a23d674560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:52016/0 (socket says 172.21.15.2:52016) 2023-11-16T16:53:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.324+0000 7ff6ad59b700 1 -- 172.21.15.2:0/2578155655 learned_addr learned my addr 172.21.15.2:0/2578155655 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.324+0000 7ff69e7fc700 1 -- 172.21.15.2:0/768872885 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ff698014070 con 0x7ff6a8101d50 2023-11-16T16:53:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.324+0000 7ff6acd9a700 1 --2- 172.21.15.2:0/2578155655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d67ab90 0x55a23d674b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.324+0000 7ff6add9c700 1 --2- 172.21.15.2:0/2578155655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d4d06b0 0x55a23d67a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.324+0000 7ff6ad59b700 1 -- 172.21.15.2:0/2578155655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d67ab90 msgr2=0x55a23d674b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.324+0000 7ff6ad59b700 1 --2- 172.21.15.2:0/2578155655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d67ab90 0x55a23d674b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.324+0000 7ff6ad59b700 1 -- 172.21.15.2:0/2578155655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d4d06b0 msgr2=0x55a23d67a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.324+0000 7ff6ad59b700 1 --2- 172.21.15.2:0/2578155655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d4d06b0 0x55a23d67a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.324+0000 7ff6ad59b700 1 -- 172.21.15.2:0/2578155655 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a23d4475d0 con 0x55a23d67a720 2023-11-16T16:53:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.325+0000 7ff6ad59b700 1 --2- 172.21.15.2:0/2578155655 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67a720 0x55a23d674560 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4005ea0 tx=0x7ff6a4005ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73fc740cb65df1fa server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:27.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.325+0000 7ff6937fe700 1 -- 172.21.15.2:0/2578155655 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a401d070 con 0x55a23d67a720 2023-11-16T16:53:27.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.325+0000 7ff6937fe700 1 -- 172.21.15.2:0/2578155655 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6a401e040 con 0x55a23d67a720 2023-11-16T16:53:27.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.325+0000 7ff6937fe700 1 -- 172.21.15.2:0/2578155655 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a4004990 con 0x55a23d67a720 2023-11-16T16:53:27.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.326+0000 7ff6add9c700 1 --2- 172.21.15.2:0/768872885 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ff6940a1340 0x7ff6940a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:27.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.326+0000 7ff6c307fb80 1 -- 172.21.15.2:0/2578155655 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67a720 msgr2=0x55a23d674560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:27.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.326+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/2578155655 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67a720 0x55a23d674560 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4005ea0 tx=0x7ff6a4005ed0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.326+0000 7ff6c307fb80 1 -- 172.21.15.2:0/2578155655 shutdown_connections 2023-11-16T16:53:27.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.326+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/2578155655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d67ab90 0x55a23d674b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.326+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/2578155655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d4d06b0 0x55a23d67a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.326+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/2578155655 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67a720 0x55a23d674560 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.326+0000 7ff6c307fb80 1 -- 172.21.15.2:0/2578155655 >> 172.21.15.2:0/2578155655 conn(0x55a23d669f50 msgr2=0x55a23d66f270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.327+0000 7ff6add9c700 1 --2- 172.21.15.2:0/768872885 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ff6940a1340 0x7ff6940a37f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0009ec0 tx=0x7ff6a0004ad0 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.327+0000 7ff6c307fb80 1 -- 172.21.15.2:0/2578155655 shutdown_connections 2023-11-16T16:53:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.327+0000 7ff6c307fb80 1 -- 172.21.15.2:0/2578155655 wait complete. 2023-11-16T16:53:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.327+0000 7ff6c307fb80 1 -- start start 2023-11-16T16:53:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.327+0000 7ff6c307fb80 1 Processor -- start 2023-11-16T16:53:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.328+0000 7ff6c307fb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d4d06b0 0x55a23d6809b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.328+0000 7ff6c307fb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67a720 0x55a23d67dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.328+0000 7ff6c307fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d67ab90 0x55a23d67e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.328+0000 7ff6c307fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a23d567880 con 0x55a23d67ab90 2023-11-16T16:53:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.328+0000 7ff6c307fb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a23d567520 con 0x55a23d67a720 2023-11-16T16:53:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.328+0000 7ff6c307fb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a23d4498c0 con 0x55a23d4d06b0 2023-11-16T16:53:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.328+0000 7ff6c307fb80 10 client.0 _ll_register_callbacks cb 0x55a23d585720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.328+0000 7ff6ad59b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d4d06b0 0x55a23d6809b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.328+0000 7ff6ad59b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d4d06b0 0x55a23d6809b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:42640/0 (socket says 172.21.15.2:42640) 2023-11-16T16:53:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.329+0000 7ff6ad59b700 1 -- 172.21.15.2:0/3885160225 learned_addr learned my addr 172.21.15.2:0/3885160225 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:27.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.329+0000 7ff6acd9a700 1 --2- 172.21.15.2:0/3885160225 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67a720 0x55a23d67dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:27.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.329+0000 7ff6add9c700 1 --2- 172.21.15.2:0/3885160225 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d67ab90 0x55a23d67e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:27.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.329+0000 7ff6ad59b700 1 -- 172.21.15.2:0/3885160225 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67a720 msgr2=0x55a23d67dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:27.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.329+0000 7ff6ad59b700 1 --2- 172.21.15.2:0/3885160225 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67a720 0x55a23d67dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.329+0000 7ff6ad59b700 1 -- 172.21.15.2:0/3885160225 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d67ab90 msgr2=0x55a23d67e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:27.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.329+0000 7ff6ad59b700 1 --2- 172.21.15.2:0/3885160225 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d67ab90 0x55a23d67e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.329+0000 7ff6ad59b700 1 -- 172.21.15.2:0/3885160225 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6a810d8f0 con 0x55a23d4d06b0 2023-11-16T16:53:27.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.329+0000 7ff6add9c700 1 --2- 172.21.15.2:0/3885160225 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d67ab90 0x55a23d67e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:53:27.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.329+0000 7ff6ad59b700 1 --2- 172.21.15.2:0/3885160225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d4d06b0 0x55a23d6809b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff6a400bb30 tx=0x7ff6a4022b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbd24e8d29008f60 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:27.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.329+0000 7ff690ff9700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:53:27.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.330+0000 7ff690ff9700 1 -- 172.21.15.2:0/3885160225 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a401d070 con 0x55a23d4d06b0 2023-11-16T16:53:27.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.330+0000 7ff690ff9700 1 -- 172.21.15.2:0/3885160225 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ff6a4022e40 con 0x55a23d4d06b0 2023-11-16T16:53:27.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.330+0000 7ff6c307fb80 1 -- 172.21.15.2:0/3885160225 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a23d667710 con 0x55a23d4d06b0 2023-11-16T16:53:27.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.330+0000 7ff690ff9700 1 -- 172.21.15.2:0/3885160225 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6a40056b0 con 0x55a23d4d06b0 2023-11-16T16:53:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.330+0000 7ff6c307fb80 10 client.25705 fetch_fsmap learned FSMap version 34 2023-11-16T16:53:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.330+0000 7ff6c307fb80 1 -- 172.21.15.2:0/3885160225 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55a23d58c130 con 0x55a23d4d06b0 2023-11-16T16:53:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.330+0000 7ff690ff9700 1 -- 172.21.15.2:0/3885160225 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a23d667710 con 0x55a23d4d06b0 2023-11-16T16:53:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.331+0000 7ff690ff9700 1 -- 172.21.15.2:0/3885160225 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 34) v1 ==== 7486+0+0 (secure 0 0 0) 0x7ff6a402c400 con 0x55a23d4d06b0 2023-11-16T16:53:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.331+0000 7ff6c307fb80 10 client.25705 fetch_fsmap finished waiting for FSMap version 34 2023-11-16T16:53:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.331+0000 7ff6c307fb80 10 client.25705 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.331+0000 7ff6c307fb80 20 client.25705 populate_metadata read hostname 'smithi002' 2023-11-16T16:53:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.331+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/3885160225 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55a23d68dab0 0x55a23d68fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.331+0000 7ff6c307fb80 4 client.25705 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:53:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.331+0000 7ff6c307fb80 1 -- 172.21.15.2:0/3885160225 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a23d588040 con 0x55a23d68dab0 2023-11-16T16:53:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.332+0000 7ff6acd9a700 1 --2- 172.21.15.2:0/3885160225 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55a23d68dab0 0x55a23d68fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.332+0000 7ff6acd9a700 1 --2- 172.21.15.2:0/3885160225 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55a23d68dab0 0x55a23d68fec0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88290d115d8fd11f server_cookie=417b54d90a12db7e in_seq=0 out_seq=0 2023-11-16T16:53:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.332+0000 7ff690ff9700 10 client.25705 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:53:27.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.334+0000 7ff690ff9700 1 -- 172.21.15.2:0/3885160225 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a23d588040 con 0x55a23d68dab0 2023-11-16T16:53:27.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.334+0000 7ff690ff9700 10 client.25705 handle_command_reply: tid=0 2023-11-16T16:53:27.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.334+0000 7ff6c307fb80 1 client.25705 shutdown 2023-11-16T16:53:27.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.335+0000 7ff6c307fb80 1 -- 172.21.15.2:0/3885160225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d4d06b0 msgr2=0x55a23d6809b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:27.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.335+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/3885160225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d4d06b0 0x55a23d6809b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff6a400bb30 tx=0x7ff6a4022b30 comp rx=0 tx=0).stop 2023-11-16T16:53:27.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.336+0000 7ff6c307fb80 1 -- 172.21.15.2:0/3885160225 shutdown_connections 2023-11-16T16:53:27.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.336+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/3885160225 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55a23d68dab0 0x55a23d68fec0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.336+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/3885160225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d4d06b0 0x55a23d6809b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.336+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/3885160225 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d67ab90 0x55a23d67e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.336+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/3885160225 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67a720 0x55a23d67dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.336+0000 7ff690ff9700 0 client.25705 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:53:27.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.336+0000 7ff6c307fb80 1 -- 172.21.15.2:0/3885160225 >> 172.21.15.2:0/3885160225 conn(0x55a23d669f50 msgr2=0x55a23d66e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:27.339 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.337+0000 7ff6c307fb80 1 -- 172.21.15.2:0/3885160225 shutdown_connections 2023-11-16T16:53:27.339 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.337+0000 7ff6c307fb80 1 -- 172.21.15.2:0/3885160225 wait complete. 2023-11-16T16:53:27.339 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.337+0000 7ff6c307fb80 20 client.25705 trim_cache size 0 max 16384 2023-11-16T16:53:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.357+0000 7ff6c307fb80 1 Processor -- start 2023-11-16T16:53:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.357+0000 7ff6c307fb80 1 -- start start 2023-11-16T16:53:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.357+0000 7ff6c307fb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d4d06b0 0x55a23d66ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.357+0000 7ff6c307fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d67a720 0x55a23d680550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.357+0000 7ff6c307fb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67ab90 0x55a23d67af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.357+0000 7ff6c307fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a23d675660 con 0x55a23d67a720 2023-11-16T16:53:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.357+0000 7ff6c307fb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a23d670c50 con 0x55a23d67ab90 2023-11-16T16:53:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.357+0000 7ff6c307fb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a23d679d30 con 0x55a23d4d06b0 2023-11-16T16:53:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.357+0000 7ff6ad59b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d4d06b0 0x55a23d66ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.357+0000 7ff6ad59b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d4d06b0 0x55a23d66ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:42650/0 (socket says 172.21.15.2:42650) 2023-11-16T16:53:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.357+0000 7ff6ad59b700 1 -- 172.21.15.2:0/2116094374 learned_addr learned my addr 172.21.15.2:0/2116094374 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.358+0000 7ff6add9c700 1 --2- 172.21.15.2:0/2116094374 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67ab90 0x55a23d67af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.358+0000 7ff6ad59b700 1 -- 172.21.15.2:0/2116094374 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67ab90 msgr2=0x55a23d67af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.358+0000 7ff6ad59b700 1 --2- 172.21.15.2:0/2116094374 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67ab90 0x55a23d67af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.358+0000 7ff6ad59b700 1 -- 172.21.15.2:0/2116094374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d67a720 msgr2=0x55a23d680550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.358+0000 7ff6acd9a700 1 --2- 172.21.15.2:0/2116094374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d67a720 0x55a23d680550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.358+0000 7ff6ad59b700 1 --2- 172.21.15.2:0/2116094374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d67a720 0x55a23d680550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.358+0000 7ff6ad59b700 1 -- 172.21.15.2:0/2116094374 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a23d58be80 con 0x55a23d4d06b0 2023-11-16T16:53:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.358+0000 7ff6add9c700 1 --2- 172.21.15.2:0/2116094374 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67ab90 0x55a23d67af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:53:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.358+0000 7ff6acd9a700 1 --2- 172.21.15.2:0/2116094374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d67a720 0x55a23d680550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:53:27.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.359+0000 7ff6ad59b700 1 --2- 172.21.15.2:0/2116094374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d4d06b0 0x55a23d66ef60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff6a40098e0 tx=0x7ff6a4005f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0c6ea34c35b42c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:27.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.359+0000 7ff693fff700 1 -- 172.21.15.2:0/2116094374 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a401d040 con 0x55a23d4d06b0 2023-11-16T16:53:27.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.359+0000 7ff693fff700 1 -- 172.21.15.2:0/2116094374 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6a4005bc0 con 0x55a23d4d06b0 2023-11-16T16:53:27.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.359+0000 7ff693fff700 1 -- 172.21.15.2:0/2116094374 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a4005070 con 0x55a23d4d06b0 2023-11-16T16:53:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.360+0000 7ff6c307fb80 1 -- 172.21.15.2:0/2116094374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d4d06b0 msgr2=0x55a23d66ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.360+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/2116094374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d4d06b0 0x55a23d66ef60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff6a40098e0 tx=0x7ff6a4005f00 comp rx=0 tx=0).stop 2023-11-16T16:53:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.360+0000 7ff6c307fb80 1 -- 172.21.15.2:0/2116094374 shutdown_connections 2023-11-16T16:53:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.360+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/2116094374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d4d06b0 0x55a23d66ef60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.360+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/2116094374 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67ab90 0x55a23d67af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.360+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/2116094374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d67a720 0x55a23d680550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.360+0000 7ff6c307fb80 1 -- 172.21.15.2:0/2116094374 >> 172.21.15.2:0/2116094374 conn(0x55a23d669f50 msgr2=0x55a23d68a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.360+0000 7ff6c307fb80 1 -- 172.21.15.2:0/2116094374 shutdown_connections 2023-11-16T16:53:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.360+0000 7ff6c307fb80 1 -- 172.21.15.2:0/2116094374 wait complete. 2023-11-16T16:53:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.360+0000 7ff6c307fb80 1 -- start start 2023-11-16T16:53:27.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.361+0000 7ff6c307fb80 1 Processor -- start 2023-11-16T16:53:27.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.361+0000 7ff6c307fb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d4d06b0 0x55a23d685220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:27.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.361+0000 7ff6c307fb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d67a720 0x55a23d682fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:27.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.361+0000 7ff6c307fb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67ab90 0x55a23d683560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:27.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.361+0000 7ff6c307fb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a23d683c70 con 0x55a23d4d06b0 2023-11-16T16:53:27.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.361+0000 7ff6c307fb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a23d683de0 con 0x55a23d67ab90 2023-11-16T16:53:27.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.361+0000 7ff6c307fb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a23d6e0930 con 0x55a23d67a720 2023-11-16T16:53:27.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.361+0000 7ff6c307fb80 10 client.0 _ll_register_callbacks cb 0x7ff698010b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:27.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.362+0000 7ff6acd9a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d67a720 0x55a23d682fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:27.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.362+0000 7ff6ad59b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d4d06b0 0x55a23d685220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:27.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.362+0000 7ff6acd9a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d67a720 0x55a23d682fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:42652/0 (socket says 172.21.15.2:42652) 2023-11-16T16:53:27.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.362+0000 7ff6acd9a700 1 -- 172.21.15.2:0/4186967339 learned_addr learned my addr 172.21.15.2:0/4186967339 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:27.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.362+0000 7ff6add9c700 1 --2- 172.21.15.2:0/4186967339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67ab90 0x55a23d683560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:27.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.362+0000 7ff6acd9a700 1 -- 172.21.15.2:0/4186967339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67ab90 msgr2=0x55a23d683560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:27.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.362+0000 7ff6acd9a700 1 --2- 172.21.15.2:0/4186967339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67ab90 0x55a23d683560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.362+0000 7ff6acd9a700 1 -- 172.21.15.2:0/4186967339 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d4d06b0 msgr2=0x55a23d685220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:27.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.362+0000 7ff6acd9a700 1 --2- 172.21.15.2:0/4186967339 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d4d06b0 0x55a23d685220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:27.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.362+0000 7ff6acd9a700 1 -- 172.21.15.2:0/4186967339 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a23d58c130 con 0x55a23d67a720 2023-11-16T16:53:27.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.362+0000 7ff6ad59b700 1 --2- 172.21.15.2:0/4186967339 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d4d06b0 0x55a23d685220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:53:27.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.362+0000 7ff6acd9a700 1 --2- 172.21.15.2:0/4186967339 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d67a720 0x55a23d682fa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff6980969b0 tx=0x7ff698010c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcf66bac38583afa server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:27.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.362+0000 7ff690ff9700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:53:27.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.363+0000 7ff690ff9700 1 -- 172.21.15.2:0/4186967339 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff698099f40 con 0x55a23d67a720 2023-11-16T16:53:27.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.363+0000 7ff690ff9700 1 -- 172.21.15.2:0/4186967339 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ff69809df70 con 0x55a23d67a720 2023-11-16T16:53:27.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.363+0000 7ff6c307fb80 1 -- 172.21.15.2:0/4186967339 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff6a401e040 con 0x55a23d67a720 2023-11-16T16:53:27.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.363+0000 7ff690ff9700 1 -- 172.21.15.2:0/4186967339 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6980a2390 con 0x55a23d67a720 2023-11-16T16:53:27.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.363+0000 7ff6c307fb80 10 client.25711 fetch_fsmap learned FSMap version 34 2023-11-16T16:53:27.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.363+0000 7ff6c307fb80 1 -- 172.21.15.2:0/4186967339 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55a23d3dcc00 con 0x55a23d67a720 2023-11-16T16:53:27.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.363+0000 7ff690ff9700 1 -- 172.21.15.2:0/4186967339 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6a401e040 con 0x55a23d67a720 2023-11-16T16:53:27.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.363+0000 7ff690ff9700 1 -- 172.21.15.2:0/4186967339 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 34) v1 ==== 7486+0+0 (secure 0 0 0) 0x7ff698089070 con 0x55a23d67a720 2023-11-16T16:53:27.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.363+0000 7ff6c307fb80 10 client.25711 fetch_fsmap finished waiting for FSMap version 34 2023-11-16T16:53:27.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.363+0000 7ff6c307fb80 10 client.25711 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:27.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.363+0000 7ff6c307fb80 20 client.25711 populate_metadata read hostname 'smithi002' 2023-11-16T16:53:27.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.363+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/4186967339 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55a23d6e2140 0x55a23d6e4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:27.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.363+0000 7ff6c307fb80 4 client.25711 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:53:27.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.363+0000 7ff6c307fb80 1 -- 172.21.15.2:0/4186967339 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a23d594600 con 0x55a23d6e2140 2023-11-16T16:53:27.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.364+0000 7ff6ad59b700 1 --2- 172.21.15.2:0/4186967339 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55a23d6e2140 0x55a23d6e4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:27.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.364+0000 7ff6ad59b700 1 --2- 172.21.15.2:0/4186967339 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55a23d6e2140 0x55a23d6e4550 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d623881214ea7ab server_cookie=371798607fcc2934 in_seq=0 out_seq=0 2023-11-16T16:53:27.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:27.364+0000 7ff690ff9700 10 client.25711 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:53:27.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:27 smithi155 ceph-mon[131607]: pgmap v109: 129 pgs: 129 active+clean; 155 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 7.2 MiB/s wr, 659 op/s 2023-11-16T16:53:27.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:27 smithi002 ceph-mon[128712]: pgmap v109: 129 pgs: 129 active+clean; 155 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 7.2 MiB/s wr, 659 op/s 2023-11-16T16:53:27.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:27 smithi141 ceph-mon[131184]: pgmap v109: 129 pgs: 129 active+clean; 155 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 7.2 MiB/s wr, 659 op/s 2023-11-16T16:53:29.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:29 smithi155 ceph-mon[131607]: pgmap v110: 129 pgs: 129 active+clean; 185 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 10 MiB/s wr, 748 op/s 2023-11-16T16:53:29.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:29 smithi002 ceph-mon[128712]: pgmap v110: 129 pgs: 129 active+clean; 185 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 10 MiB/s wr, 748 op/s 2023-11-16T16:53:29.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:29 smithi141 ceph-mon[131184]: pgmap v110: 129 pgs: 129 active+clean; 185 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 10 MiB/s wr, 748 op/s 2023-11-16T16:53:31.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:31 smithi155 ceph-mon[131607]: pgmap v111: 129 pgs: 129 active+clean; 185 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 6.2 MiB/s wr, 262 op/s 2023-11-16T16:53:31.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:31 smithi002 ceph-mon[128712]: pgmap v111: 129 pgs: 129 active+clean; 185 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 6.2 MiB/s wr, 262 op/s 2023-11-16T16:53:31.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:31 smithi141 ceph-mon[131184]: pgmap v111: 129 pgs: 129 active+clean; 185 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 6.2 MiB/s wr, 262 op/s 2023-11-16T16:53:31.724 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.722+0000 7ff690ff9700 1 -- 172.21.15.2:0/4186967339 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a23d594600 con 0x55a23d6e2140 2023-11-16T16:53:31.724 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.722+0000 7ff690ff9700 10 client.25711 handle_command_reply: tid=0 2023-11-16T16:53:31.724 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.722+0000 7ff6c307fb80 1 client.25711 shutdown 2023-11-16T16:53:31.724 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.722+0000 7ff6c307fb80 1 -- 172.21.15.2:0/4186967339 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d67a720 msgr2=0x55a23d682fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:31.724 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.722+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/4186967339 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d67a720 0x55a23d682fa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff6980969b0 tx=0x7ff698010c80 comp rx=0 tx=0).stop 2023-11-16T16:53:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.723+0000 7ff6c307fb80 1 -- 172.21.15.2:0/4186967339 shutdown_connections 2023-11-16T16:53:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.723+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/4186967339 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55a23d67ab90 0x55a23d683560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.723+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/4186967339 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a23d4d06b0 0x55a23d685220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.723+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/4186967339 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55a23d6e2140 0x55a23d6e4550 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.723+0000 7ff6c307fb80 1 --2- 172.21.15.2:0/4186967339 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a23d67a720 0x55a23d682fa0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.723+0000 7ff6c307fb80 1 -- 172.21.15.2:0/4186967339 >> 172.21.15.2:0/4186967339 conn(0x55a23d669f50 msgr2=0x55a23d67af60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.723+0000 7ff690ff9700 0 client.25711 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:53:31.726 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.724+0000 7ff6c307fb80 1 -- 172.21.15.2:0/4186967339 shutdown_connections 2023-11-16T16:53:31.726 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.724+0000 7ff6c307fb80 1 -- 172.21.15.2:0/4186967339 wait complete. 2023-11-16T16:53:31.726 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.724+0000 7ff6c307fb80 20 client.25711 trim_cache size 0 max 16384 2023-11-16T16:53:31.726 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:53:31.727 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:53:31.727 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:53:31.727 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:53:31.727 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.725+0000 7ff6af7ff700 1 -- 172.21.15.2:0/768872885 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ff6940a1340 msgr2=0x7ff6940a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:31.727 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.725+0000 7ff6af7ff700 1 --2- 172.21.15.2:0/768872885 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ff6940a1340 0x7ff6940a37f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0009ec0 tx=0x7ff6a0004ad0 comp rx=0 tx=0).stop 2023-11-16T16:53:31.728 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.725+0000 7ff6af7ff700 1 -- 172.21.15.2:0/768872885 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6a8101d50 msgr2=0x7ff6a81968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:31.728 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.725+0000 7ff6af7ff700 1 --2- 172.21.15.2:0/768872885 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6a8101d50 0x7ff6a81968b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff69800e9a0 tx=0x7ff69800ecb0 comp rx=0 tx=0).stop 2023-11-16T16:53:31.728 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.726+0000 7ff6af7ff700 1 -- 172.21.15.2:0/768872885 shutdown_connections 2023-11-16T16:53:31.728 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.726+0000 7ff6af7ff700 1 --2- 172.21.15.2:0/768872885 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ff6940a1340 0x7ff6940a37f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:31.728 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.726+0000 7ff6af7ff700 1 --2- 172.21.15.2:0/768872885 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff6a810a810 0x7ff6a819ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:31.728 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.726+0000 7ff6af7ff700 1 --2- 172.21.15.2:0/768872885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff6a8101440 0x7ff6a8196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:31.728 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.726+0000 7ff6af7ff700 1 --2- 172.21.15.2:0/768872885 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6a8101d50 0x7ff6a81968b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:31.728 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.726+0000 7ff6af7ff700 1 -- 172.21.15.2:0/768872885 >> 172.21.15.2:0/768872885 conn(0x7ff6a80763a0 msgr2=0x7ff6a80fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:31.728 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.726+0000 7ff6af7ff700 1 -- 172.21.15.2:0/768872885 shutdown_connections 2023-11-16T16:53:31.728 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:31.726+0000 7ff6af7ff700 1 -- 172.21.15.2:0/768872885 wait complete. 2023-11-16T16:53:33.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:33 smithi155 ceph-mon[131607]: pgmap v112: 129 pgs: 129 active+clean; 209 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 8.9 MiB/s wr, 837 op/s 2023-11-16T16:53:33.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:33 smithi002 ceph-mon[128712]: pgmap v112: 129 pgs: 129 active+clean; 209 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 8.9 MiB/s wr, 837 op/s 2023-11-16T16:53:33.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:33 smithi141 ceph-mon[131184]: pgmap v112: 129 pgs: 129 active+clean; 209 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 8.9 MiB/s wr, 837 op/s 2023-11-16T16:53:35.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:35 smithi155 ceph-mon[131607]: pgmap v113: 129 pgs: 129 active+clean; 209 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.5 MiB/s wr, 663 op/s 2023-11-16T16:53:35.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:35 smithi002 ceph-mon[128712]: pgmap v113: 129 pgs: 129 active+clean; 209 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.5 MiB/s wr, 663 op/s 2023-11-16T16:53:35.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:35 smithi141 ceph-mon[131184]: pgmap v113: 129 pgs: 129 active+clean; 209 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.5 MiB/s wr, 663 op/s 2023-11-16T16:53:37.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:37 smithi155 ceph-mon[131607]: pgmap v114: 129 pgs: 129 active+clean; 209 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.5 MiB/s wr, 663 op/s 2023-11-16T16:53:37.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:37 smithi002 ceph-mon[128712]: pgmap v114: 129 pgs: 129 active+clean; 209 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.5 MiB/s wr, 663 op/s 2023-11-16T16:53:37.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:37 smithi141 ceph-mon[131184]: pgmap v114: 129 pgs: 129 active+clean; 209 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.5 MiB/s wr, 663 op/s 2023-11-16T16:53:39.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:39 smithi155 ceph-mon[131607]: pgmap v115: 129 pgs: 129 active+clean; 192 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 7.5 MiB/s wr, 973 op/s 2023-11-16T16:53:39.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:39 smithi002 ceph-mon[128712]: pgmap v115: 129 pgs: 129 active+clean; 192 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 7.5 MiB/s wr, 973 op/s 2023-11-16T16:53:39.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:39 smithi141 ceph-mon[131184]: pgmap v115: 129 pgs: 129 active+clean; 192 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 7.5 MiB/s wr, 973 op/s 2023-11-16T16:53:41.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:41 smithi155 ceph-mon[131607]: pgmap v116: 129 pgs: 129 active+clean; 192 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.6 MiB/s wr, 884 op/s 2023-11-16T16:53:41.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:41 smithi002 ceph-mon[128712]: pgmap v116: 129 pgs: 129 active+clean; 192 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.6 MiB/s wr, 884 op/s 2023-11-16T16:53:41.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:41 smithi141 ceph-mon[131184]: pgmap v116: 129 pgs: 129 active+clean; 192 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.6 MiB/s wr, 884 op/s 2023-11-16T16:53:43.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:43 smithi155 ceph-mon[131607]: pgmap v117: 129 pgs: 129 active+clean; 208 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.1 MiB/s wr, 893 op/s 2023-11-16T16:53:43.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:43 smithi002 ceph-mon[128712]: pgmap v117: 129 pgs: 129 active+clean; 208 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.1 MiB/s wr, 893 op/s 2023-11-16T16:53:43.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:43 smithi141 ceph-mon[131184]: pgmap v117: 129 pgs: 129 active+clean; 208 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.1 MiB/s wr, 893 op/s 2023-11-16T16:53:44.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.882+0000 7f52affff700 1 -- 172.21.15.2:0/2783467123 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52a8100560 msgr2=0x7f52a8100930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:44.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.882+0000 7f52affff700 1 --2- 172.21.15.2:0/2783467123 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52a8100560 0x7f52a8100930 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f52a40098f0 tx=0x7f52a4009c00 comp rx=0 tx=0).stop 2023-11-16T16:53:44.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.882+0000 7f52affff700 1 -- 172.21.15.2:0/2783467123 shutdown_connections 2023-11-16T16:53:44.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.882+0000 7f52affff700 1 --2- 172.21.15.2:0/2783467123 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a81055a0 0x7f52a810d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.882+0000 7f52affff700 1 --2- 172.21.15.2:0/2783467123 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52a8100560 0x7f52a8100930 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.882+0000 7f52affff700 1 --2- 172.21.15.2:0/2783467123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52a8100f00 0x7f52a8104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.882+0000 7f52affff700 1 -- 172.21.15.2:0/2783467123 >> 172.21.15.2:0/2783467123 conn(0x7f52a80781b0 msgr2=0x7f52a80785b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:44.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.882+0000 7f52affff700 1 -- 172.21.15.2:0/2783467123 shutdown_connections 2023-11-16T16:53:44.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.882+0000 7f52affff700 1 -- 172.21.15.2:0/2783467123 wait complete. 2023-11-16T16:53:44.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.884+0000 7f52affff700 1 Processor -- start 2023-11-16T16:53:44.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.884+0000 7f52affff700 1 -- start start 2023-11-16T16:53:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.884+0000 7f52affff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52a8100560 0x7f52a8072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:44.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.884+0000 7f52affff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52a8100f00 0x7f52a8073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.884+0000 7f52affff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a81055a0 0x7f52a81a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.884+0000 7f52affff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a8073b60 con 0x7f52a8100560 2023-11-16T16:53:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.884+0000 7f52affff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a81a4b30 con 0x7f52a8100f00 2023-11-16T16:53:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.884+0000 7f52affff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a81a4c70 con 0x7f52a81055a0 2023-11-16T16:53:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.885+0000 7f52ad59a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52a8100f00 0x7f52a8073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.885+0000 7f52add9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52a8100560 0x7f52a8072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.885+0000 7f52ad59a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52a8100f00 0x7f52a8073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:33858/0 (socket says 172.21.15.2:33858) 2023-11-16T16:53:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.885+0000 7f52ad59a700 1 -- 172.21.15.2:0/1584078343 learned_addr learned my addr 172.21.15.2:0/1584078343 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.885+0000 7f52ae59c700 1 --2- 172.21.15.2:0/1584078343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a81055a0 0x7f52a81a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.885+0000 7f52ad59a700 1 -- 172.21.15.2:0/1584078343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a81055a0 msgr2=0x7f52a81a45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:44.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.885+0000 7f52ad59a700 1 --2- 172.21.15.2:0/1584078343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a81055a0 0x7f52a81a45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.885+0000 7f52ad59a700 1 -- 172.21.15.2:0/1584078343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52a8100560 msgr2=0x7f52a8072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.885+0000 7f52ad59a700 1 --2- 172.21.15.2:0/1584078343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52a8100560 0x7f52a8072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.885+0000 7f52ad59a700 1 -- 172.21.15.2:0/1584078343 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a4009520 con 0x7f52a8100f00 2023-11-16T16:53:44.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.885+0000 7f52ad59a700 1 --2- 172.21.15.2:0/1584078343 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52a8100f00 0x7f52a8073330 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f529800e9c0 tx=0x7f529800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:44.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.885+0000 7f529effd700 1 -- 172.21.15.2:0/1584078343 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298009690 con 0x7f52a8100f00 2023-11-16T16:53:44.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.885+0000 7f529effd700 1 -- 172.21.15.2:0/1584078343 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f529800ce80 con 0x7f52a8100f00 2023-11-16T16:53:44.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.885+0000 7f52affff700 1 -- 172.21.15.2:0/1584078343 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a81103b0 con 0x7f52a8100f00 2023-11-16T16:53:44.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.886+0000 7f529effd700 1 -- 172.21.15.2:0/1584078343 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298018420 con 0x7f52a8100f00 2023-11-16T16:53:44.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.886+0000 7f52affff700 1 -- 172.21.15.2:0/1584078343 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a8110220 con 0x7f52a8100f00 2023-11-16T16:53:44.889 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.889+0000 7f529effd700 1 -- 172.21.15.2:0/1584078343 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f529800c9c0 con 0x7f52a8100f00 2023-11-16T16:53:44.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.890+0000 7f529effd700 1 --2- 172.21.15.2:0/1584078343 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f52940a1340 0x7f52940a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:44.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.890+0000 7f52c3a02b80 1 Processor -- start 2023-11-16T16:53:44.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.890+0000 7f529effd700 1 -- 172.21.15.2:0/1584078343 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f5298014070 con 0x7f52a8100f00 2023-11-16T16:53:44.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.891+0000 7f52add9b700 1 --2- 172.21.15.2:0/1584078343 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f52940a1340 0x7f52940a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:44.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.891+0000 7f52c3a02b80 1 -- start start 2023-11-16T16:53:44.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.891+0000 7f52c3a02b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e138b6b0 0x5585e15351e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:44.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.891+0000 7f52c3a02b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e1535720 0x5585e152f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:44.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.891+0000 7f52c3a02b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e1535b90 0x5585e152fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:44.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.891+0000 7f52c3a02b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585e1435000 con 0x5585e1535b90 2023-11-16T16:53:44.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.891+0000 7f52c3a02b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585e1487840 con 0x5585e1535720 2023-11-16T16:53:44.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.891+0000 7f52c3a02b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585e1480cd0 con 0x5585e138b6b0 2023-11-16T16:53:44.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.891+0000 7f52add9b700 1 --2- 172.21.15.2:0/1584078343 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f52940a1340 0x7f52940a37f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f52a4005e90 tx=0x7f52a4019040 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:44.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.891+0000 7f52ad59a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e1535720 0x5585e152f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:44.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.892+0000 7f52ad59a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e1535720 0x5585e152f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:33866/0 (socket says 172.21.15.2:33866) 2023-11-16T16:53:44.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.892+0000 7f52ad59a700 1 -- 172.21.15.2:0/3106372861 learned_addr learned my addr 172.21.15.2:0/3106372861 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:44.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.892+0000 7f52add9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e138b6b0 0x5585e15351e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:44.893 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.892+0000 7f52ae59c700 1 --2- 172.21.15.2:0/3106372861 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e1535b90 0x5585e152fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:44.895 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.892+0000 7f52add9b700 1 -- 172.21.15.2:0/3106372861 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e1535720 msgr2=0x5585e152f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:44.895 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.892+0000 7f52add9b700 1 --2- 172.21.15.2:0/3106372861 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e1535720 0x5585e152f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.895 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.892+0000 7f52add9b700 1 -- 172.21.15.2:0/3106372861 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e1535b90 msgr2=0x5585e152fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:44.895 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.892+0000 7f52add9b700 1 --2- 172.21.15.2:0/3106372861 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e1535b90 0x5585e152fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.892+0000 7f52add9b700 1 -- 172.21.15.2:0/3106372861 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585e13025d0 con 0x5585e138b6b0 2023-11-16T16:53:44.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.892+0000 7f52ad59a700 1 --2- 172.21.15.2:0/3106372861 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e1535720 0x5585e152f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:53:44.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.892+0000 7f52add9b700 1 --2- 172.21.15.2:0/3106372861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e138b6b0 0x5585e15351e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f52a4009810 tx=0x7f52a401e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d51072cd8183c2f1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:44.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.893+0000 7f5293fff700 1 -- 172.21.15.2:0/3106372861 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a402a070 con 0x5585e138b6b0 2023-11-16T16:53:44.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.893+0000 7f5293fff700 1 -- 172.21.15.2:0/3106372861 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f52a401d040 con 0x5585e138b6b0 2023-11-16T16:53:44.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.893+0000 7f5293fff700 1 -- 172.21.15.2:0/3106372861 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a40055f0 con 0x5585e138b6b0 2023-11-16T16:53:44.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.893+0000 7f52c3a02b80 1 -- 172.21.15.2:0/3106372861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e138b6b0 msgr2=0x5585e15351e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:44.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.893+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/3106372861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e138b6b0 0x5585e15351e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f52a4009810 tx=0x7f52a401e000 comp rx=0 tx=0).stop 2023-11-16T16:53:44.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.894+0000 7f52c3a02b80 1 -- 172.21.15.2:0/3106372861 shutdown_connections 2023-11-16T16:53:44.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.894+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/3106372861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e138b6b0 0x5585e15351e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.894+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/3106372861 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e1535b90 0x5585e152fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.894+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/3106372861 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e1535720 0x5585e152f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.894+0000 7f52c3a02b80 1 -- 172.21.15.2:0/3106372861 >> 172.21.15.2:0/3106372861 conn(0x5585e1524f50 msgr2=0x5585e152a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:44.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.894+0000 7f52c3a02b80 1 -- 172.21.15.2:0/3106372861 shutdown_connections 2023-11-16T16:53:44.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.894+0000 7f52c3a02b80 1 -- 172.21.15.2:0/3106372861 wait complete. 2023-11-16T16:53:44.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.894+0000 7f52c3a02b80 1 -- start start 2023-11-16T16:53:44.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.895+0000 7f52c3a02b80 1 Processor -- start 2023-11-16T16:53:44.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.895+0000 7f52c3a02b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e138b6b0 0x5585e153ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:44.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.895+0000 7f52c3a02b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e1535720 0x5585e153efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:44.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.895+0000 7f52c3a02b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e1535b90 0x5585e1538d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:44.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.895+0000 7f52c3a02b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585e1422880 con 0x5585e1535720 2023-11-16T16:53:44.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.895+0000 7f52c3a02b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585e1422520 con 0x5585e138b6b0 2023-11-16T16:53:44.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.895+0000 7f52c3a02b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585e13048c0 con 0x5585e1535b90 2023-11-16T16:53:44.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.895+0000 7f52c3a02b80 10 client.0 _ll_register_callbacks cb 0x5585e1440720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:44.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.895+0000 7f52add9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e138b6b0 0x5585e153ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:44.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.895+0000 7f52add9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e138b6b0 0x5585e153ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:33880/0 (socket says 172.21.15.2:33880) 2023-11-16T16:53:44.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.895+0000 7f52add9b700 1 -- 172.21.15.2:0/391171934 learned_addr learned my addr 172.21.15.2:0/391171934 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:44.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.896+0000 7f52ae59c700 1 --2- 172.21.15.2:0/391171934 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e1535b90 0x5585e1538d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:44.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.896+0000 7f52ad59a700 1 --2- 172.21.15.2:0/391171934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e1535720 0x5585e153efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:44.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.896+0000 7f52add9b700 1 -- 172.21.15.2:0/391171934 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e1535b90 msgr2=0x5585e1538d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:44.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.896+0000 7f52add9b700 1 --2- 172.21.15.2:0/391171934 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e1535b90 0x5585e1538d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.896+0000 7f52add9b700 1 -- 172.21.15.2:0/391171934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e1535720 msgr2=0x5585e153efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:44.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.896+0000 7f52add9b700 1 --2- 172.21.15.2:0/391171934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e1535720 0x5585e153efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.896+0000 7f52add9b700 1 -- 172.21.15.2:0/391171934 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a810d9b0 con 0x5585e138b6b0 2023-11-16T16:53:44.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.896+0000 7f52add9b700 1 --2- 172.21.15.2:0/391171934 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e138b6b0 0x5585e153ea70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f52a4000c00 tx=0x7f52a400b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c36f783c88c76c0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:44.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.896+0000 7f52917fa700 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2023-11-16T16:53:44.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.896+0000 7f52917fa700 1 -- 172.21.15.2:0/391171934 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a402a070 con 0x5585e138b6b0 2023-11-16T16:53:44.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.896+0000 7f52c3a02b80 1 -- 172.21.15.2:0/391171934 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5585e1522710 con 0x5585e138b6b0 2023-11-16T16:53:44.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.896+0000 7f52917fa700 1 -- 172.21.15.2:0/391171934 <== mon.2 v2:172.21.15.141:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f52a40055c0 con 0x5585e138b6b0 2023-11-16T16:53:44.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.896+0000 7f52917fa700 1 -- 172.21.15.2:0/391171934 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52a400bc10 con 0x5585e138b6b0 2023-11-16T16:53:44.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.896+0000 7f52c3a02b80 10 client.25757 fetch_fsmap learned FSMap version 34 2023-11-16T16:53:44.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.896+0000 7f52c3a02b80 1 -- 172.21.15.2:0/391171934 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x5585e1447130 con 0x5585e138b6b0 2023-11-16T16:53:44.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.897+0000 7f52917fa700 1 -- 172.21.15.2:0/391171934 <== mon.2 v2:172.21.15.141:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5585e1522710 con 0x5585e138b6b0 2023-11-16T16:53:44.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.897+0000 7f52917fa700 1 -- 172.21.15.2:0/391171934 <== mon.2 v2:172.21.15.141:3300/0 5 ==== fsmap(e 34) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f52a40377a0 con 0x5585e138b6b0 2023-11-16T16:53:44.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.897+0000 7f52c3a02b80 10 client.25757 fetch_fsmap finished waiting for FSMap version 34 2023-11-16T16:53:44.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.897+0000 7f52c3a02b80 10 client.25757 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:44.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.897+0000 7f52c3a02b80 20 client.25757 populate_metadata read hostname 'smithi002' 2023-11-16T16:53:44.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.897+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/391171934 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5585e1548ac0 0x5585e154aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:44.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.897+0000 7f52c3a02b80 4 client.25757 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:53:44.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.897+0000 7f52c3a02b80 1 -- 172.21.15.2:0/391171934 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5585e1443040 con 0x5585e1548ac0 2023-11-16T16:53:44.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.897+0000 7f52ad59a700 1 --2- 172.21.15.2:0/391171934 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5585e1548ac0 0x5585e154aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:44.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.898+0000 7f52ad59a700 1 --2- 172.21.15.2:0/391171934 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5585e1548ac0 0x5585e154aed0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dc028116e0ce9ba server_cookie=45ec82f8bd6453a7 in_seq=0 out_seq=0 2023-11-16T16:53:44.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.898+0000 7f52917fa700 10 client.25757 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:53:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.898+0000 7f52917fa700 1 -- 172.21.15.2:0/391171934 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5585e1443040 con 0x5585e1548ac0 2023-11-16T16:53:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.898+0000 7f52917fa700 10 client.25757 handle_command_reply: tid=0 2023-11-16T16:53:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.898+0000 7f52c3a02b80 1 client.25757 shutdown 2023-11-16T16:53:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.899+0000 7f52c3a02b80 1 -- 172.21.15.2:0/391171934 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e138b6b0 msgr2=0x5585e153ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.899+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/391171934 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e138b6b0 0x5585e153ea70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f52a4000c00 tx=0x7f52a400b810 comp rx=0 tx=0).stop 2023-11-16T16:53:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.899+0000 7f52c3a02b80 1 -- 172.21.15.2:0/391171934 shutdown_connections 2023-11-16T16:53:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.899+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/391171934 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5585e1548ac0 0x5585e154aed0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.899+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/391171934 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e1535b90 0x5585e1538d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.899+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/391171934 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e138b6b0 0x5585e153ea70 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.899+0000 7f52917fa700 0 client.25757 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:53:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.899+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/391171934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e1535720 0x5585e153efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.899+0000 7f52c3a02b80 1 -- 172.21.15.2:0/391171934 >> 172.21.15.2:0/391171934 conn(0x5585e1524f50 msgr2=0x5585e15297e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.899+0000 7f52c3a02b80 1 -- 172.21.15.2:0/391171934 shutdown_connections 2023-11-16T16:53:44.904 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.899+0000 7f52c3a02b80 1 -- 172.21.15.2:0/391171934 wait complete. 2023-11-16T16:53:44.904 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.899+0000 7f52c3a02b80 20 client.25757 trim_cache size 0 max 16384 2023-11-16T16:53:44.915 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.915+0000 7f52c3a02b80 1 Processor -- start 2023-11-16T16:53:44.915 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.915+0000 7f52c3a02b80 1 -- start start 2023-11-16T16:53:44.916 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.915+0000 7f52c3a02b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e1524f50 0x5585e1538e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:44.916 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.915+0000 7f52c3a02b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e138b6b0 0x5585e15344f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:44.916 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.915+0000 7f52c3a02b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e1535720 0x5585e1529f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:44.916 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.915+0000 7f52c3a02b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585e1524c00 con 0x5585e1524f50 2023-11-16T16:53:44.916 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.915+0000 7f52c3a02b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585e1533a00 con 0x5585e138b6b0 2023-11-16T16:53:44.917 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.915+0000 7f52c3a02b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585e15351e0 con 0x5585e1535720 2023-11-16T16:53:44.917 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.916+0000 7f52add9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e1524f50 0x5585e1538e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:44.917 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.916+0000 7f52ad59a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e138b6b0 0x5585e15344f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:44.917 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.916+0000 7f52add9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e1524f50 0x5585e1538e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50342/0 (socket says 172.21.15.2:50342) 2023-11-16T16:53:44.917 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.916+0000 7f52ad59a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e138b6b0 0x5585e15344f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:33886/0 (socket says 172.21.15.2:33886) 2023-11-16T16:53:44.917 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.916+0000 7f52add9b700 1 -- 172.21.15.2:0/3581296326 learned_addr learned my addr 172.21.15.2:0/3581296326 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:44.917 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.916+0000 7f52ae59c700 1 --2- 172.21.15.2:0/3581296326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e1535720 0x5585e1529f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:44.918 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.916+0000 7f52add9b700 1 -- 172.21.15.2:0/3581296326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e1535720 msgr2=0x5585e1529f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:44.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.916+0000 7f52add9b700 1 --2- 172.21.15.2:0/3581296326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e1535720 0x5585e1529f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.916+0000 7f52add9b700 1 -- 172.21.15.2:0/3581296326 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e138b6b0 msgr2=0x5585e15344f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:44.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.916+0000 7f52add9b700 1 --2- 172.21.15.2:0/3581296326 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e138b6b0 0x5585e15344f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.916+0000 7f52add9b700 1 -- 172.21.15.2:0/3581296326 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585e1446e80 con 0x5585e1524f50 2023-11-16T16:53:44.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.916+0000 7f52ad59a700 1 --2- 172.21.15.2:0/3581296326 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e138b6b0 0x5585e15344f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:53:44.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.916+0000 7f52add9b700 1 --2- 172.21.15.2:0/3581296326 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e1524f50 0x5585e1538e60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f52a4009810 tx=0x7f52a4005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b27c656b3fe37a8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:44.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.917+0000 7f529cff9700 1 -- 172.21.15.2:0/3581296326 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a402a070 con 0x5585e1524f50 2023-11-16T16:53:44.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.917+0000 7f529cff9700 1 -- 172.21.15.2:0/3581296326 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f52a400bc10 con 0x5585e1524f50 2023-11-16T16:53:44.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.917+0000 7f529cff9700 1 -- 172.21.15.2:0/3581296326 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a40038e0 con 0x5585e1524f50 2023-11-16T16:53:44.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.917+0000 7f52c3a02b80 1 -- 172.21.15.2:0/3581296326 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e1524f50 msgr2=0x5585e1538e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:44.919 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.917+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/3581296326 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e1524f50 0x5585e1538e60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f52a4009810 tx=0x7f52a4005c00 comp rx=0 tx=0).stop 2023-11-16T16:53:44.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.917+0000 7f52c3a02b80 1 -- 172.21.15.2:0/3581296326 shutdown_connections 2023-11-16T16:53:44.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.917+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/3581296326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e1535720 0x5585e1529f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.917+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/3581296326 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e1524f50 0x5585e1538e60 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.917+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/3581296326 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e138b6b0 0x5585e15344f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.917+0000 7f52c3a02b80 1 -- 172.21.15.2:0/3581296326 >> 172.21.15.2:0/3581296326 conn(0x5585e1526710 msgr2=0x5585e1533de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:44.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.917+0000 7f52c3a02b80 1 -- 172.21.15.2:0/3581296326 shutdown_connections 2023-11-16T16:53:44.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.918+0000 7f52c3a02b80 1 -- 172.21.15.2:0/3581296326 wait complete. 2023-11-16T16:53:44.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.918+0000 7f52c3a02b80 1 -- start start 2023-11-16T16:53:44.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.918+0000 7f52c3a02b80 1 Processor -- start 2023-11-16T16:53:44.922 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.918+0000 7f52c3a02b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e138b6b0 0x5585e153c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:44.922 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.918+0000 7f52c3a02b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e1524f50 0x5585e153cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:44.922 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.919+0000 7f52c3a02b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e1535720 0x5585e153d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:44.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.919+0000 7f52c3a02b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585e159b9b0 con 0x5585e138b6b0 2023-11-16T16:53:44.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.919+0000 7f52c3a02b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585e159bb20 con 0x5585e1524f50 2023-11-16T16:53:44.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.919+0000 7f52c3a02b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585e159bce0 con 0x5585e1535720 2023-11-16T16:53:44.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.919+0000 7f52c3a02b80 10 client.0 _ll_register_callbacks cb 0x7f52a0009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:44.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.919+0000 7f52ad59a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e1524f50 0x5585e153cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:44.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.919+0000 7f52add9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e138b6b0 0x5585e153c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:44.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.919+0000 7f52ad59a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e1524f50 0x5585e153cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:33900/0 (socket says 172.21.15.2:33900) 2023-11-16T16:53:44.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.919+0000 7f52ad59a700 1 -- 172.21.15.2:0/930065674 learned_addr learned my addr 172.21.15.2:0/930065674 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:44.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.919+0000 7f52ae59c700 1 --2- 172.21.15.2:0/930065674 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e1535720 0x5585e153d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:44.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.919+0000 7f52add9b700 1 -- 172.21.15.2:0/930065674 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e1535720 msgr2=0x5585e153d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:44.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.919+0000 7f52add9b700 1 --2- 172.21.15.2:0/930065674 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e1535720 0x5585e153d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.919+0000 7f52add9b700 1 -- 172.21.15.2:0/930065674 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e1524f50 msgr2=0x5585e153cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:44.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.919+0000 7f52add9b700 1 --2- 172.21.15.2:0/930065674 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e1524f50 0x5585e153cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:44.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.919+0000 7f52add9b700 1 -- 172.21.15.2:0/930065674 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a81103b0 con 0x5585e138b6b0 2023-11-16T16:53:44.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.919+0000 7f52ad59a700 1 --2- 172.21.15.2:0/930065674 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e1524f50 0x5585e153cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:53:44.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.920+0000 7f52add9b700 1 --2- 172.21.15.2:0/930065674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e138b6b0 0x5585e153c5e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f52a4037b00 tx=0x7f52a400b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb24aed3de14d907 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:44.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.920+0000 7f52917fa700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:53:44.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.920+0000 7f52917fa700 1 -- 172.21.15.2:0/930065674 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a402a040 con 0x5585e138b6b0 2023-11-16T16:53:44.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.920+0000 7f52c3a02b80 1 -- 172.21.15.2:0/930065674 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f52a401d040 con 0x5585e138b6b0 2023-11-16T16:53:44.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.920+0000 7f52917fa700 1 -- 172.21.15.2:0/930065674 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f52a400b810 con 0x5585e138b6b0 2023-11-16T16:53:44.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.920+0000 7f52917fa700 1 -- 172.21.15.2:0/930065674 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52a4003880 con 0x5585e138b6b0 2023-11-16T16:53:44.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.920+0000 7f52c3a02b80 10 client.15717 fetch_fsmap learned FSMap version 34 2023-11-16T16:53:44.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.920+0000 7f52c3a02b80 1 -- 172.21.15.2:0/930065674 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x5585e1447130 con 0x5585e138b6b0 2023-11-16T16:53:44.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.920+0000 7f52917fa700 1 -- 172.21.15.2:0/930065674 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f52a401d040 con 0x5585e138b6b0 2023-11-16T16:53:44.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.921+0000 7f52917fa700 1 -- 172.21.15.2:0/930065674 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 34) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f52a404c070 con 0x5585e138b6b0 2023-11-16T16:53:44.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.921+0000 7f52c3a02b80 10 client.15717 fetch_fsmap finished waiting for FSMap version 34 2023-11-16T16:53:44.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.921+0000 7f52c3a02b80 10 client.15717 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:44.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.921+0000 7f52c3a02b80 20 client.15717 populate_metadata read hostname 'smithi002' 2023-11-16T16:53:44.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.921+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/930065674 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5585e1535b90 0x5585e159efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:44.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.921+0000 7f52c3a02b80 4 client.15717 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:53:44.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.921+0000 7f52c3a02b80 1 -- 172.21.15.2:0/930065674 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5585e144f600 con 0x5585e1535b90 2023-11-16T16:53:44.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.921+0000 7f52ad59a700 1 --2- 172.21.15.2:0/930065674 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5585e1535b90 0x5585e159efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:44.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.921+0000 7f52ad59a700 1 --2- 172.21.15.2:0/930065674 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5585e1535b90 0x5585e159efb0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70d74a81c95dcb92 server_cookie=73628d1ec8b56690 in_seq=0 out_seq=0 2023-11-16T16:53:44.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:44.921+0000 7f52917fa700 10 client.15717 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:53:45.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:45 smithi155 ceph-mon[131607]: pgmap v118: 129 pgs: 129 active+clean; 208 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.4 MiB/s wr, 319 op/s 2023-11-16T16:53:45.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:45 smithi002 ceph-mon[128712]: pgmap v118: 129 pgs: 129 active+clean; 208 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.4 MiB/s wr, 319 op/s 2023-11-16T16:53:45.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:45 smithi141 ceph-mon[131184]: pgmap v118: 129 pgs: 129 active+clean; 208 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.4 MiB/s wr, 319 op/s 2023-11-16T16:53:45.807 INFO:tasks.workunit.client.0.smithi002.stdout:created hierarchy, now cleaning up 2023-11-16T16:53:46.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.290+0000 7f52917fa700 1 -- 172.21.15.2:0/930065674 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5585e144f600 con 0x5585e1535b90 2023-11-16T16:53:46.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.290+0000 7f52917fa700 10 client.15717 handle_command_reply: tid=0 2023-11-16T16:53:46.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.290+0000 7f52c3a02b80 1 client.15717 shutdown 2023-11-16T16:53:46.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.290+0000 7f52c3a02b80 1 -- 172.21.15.2:0/930065674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e138b6b0 msgr2=0x5585e153c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:46.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.290+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/930065674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e138b6b0 0x5585e153c5e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f52a4037b00 tx=0x7f52a400b770 comp rx=0 tx=0).stop 2023-11-16T16:53:46.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.290+0000 7f52c3a02b80 1 -- 172.21.15.2:0/930065674 shutdown_connections 2023-11-16T16:53:46.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.290+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/930065674 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5585e1535b90 0x5585e159efb0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:46.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.290+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/930065674 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5585e1535720 0x5585e153d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:46.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.290+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/930065674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5585e138b6b0 0x5585e153c5e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:46.291 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.290+0000 7f52c3a02b80 1 --2- 172.21.15.2:0/930065674 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5585e1524f50 0x5585e153cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.290+0000 7f52c3a02b80 1 -- 172.21.15.2:0/930065674 >> 172.21.15.2:0/930065674 conn(0x5585e1526710 msgr2=0x5585e1535f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.290+0000 7f52917fa700 0 client.15717 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:53:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.290+0000 7f52c3a02b80 1 -- 172.21.15.2:0/930065674 shutdown_connections 2023-11-16T16:53:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.290+0000 7f52c3a02b80 1 -- 172.21.15.2:0/930065674 wait complete. 2023-11-16T16:53:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.290+0000 7f52c3a02b80 20 client.15717 trim_cache size 0 max 16384 2023-11-16T16:53:46.292 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:53:46.293 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:53:46.293 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:53:46.293 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:53:46.293 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.291+0000 7f52affff700 1 -- 172.21.15.2:0/1584078343 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f52940a1340 msgr2=0x7f52940a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:46.293 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.291+0000 7f52affff700 1 --2- 172.21.15.2:0/1584078343 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f52940a1340 0x7f52940a37f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f52a4005e90 tx=0x7f52a4019040 comp rx=0 tx=0).stop 2023-11-16T16:53:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.291+0000 7f52affff700 1 -- 172.21.15.2:0/1584078343 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52a8100f00 msgr2=0x7f52a8073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.291+0000 7f52affff700 1 --2- 172.21.15.2:0/1584078343 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52a8100f00 0x7f52a8073330 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f529800e9c0 tx=0x7f529800ecd0 comp rx=0 tx=0).stop 2023-11-16T16:53:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.291+0000 7f52affff700 1 -- 172.21.15.2:0/1584078343 shutdown_connections 2023-11-16T16:53:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.291+0000 7f52affff700 1 --2- 172.21.15.2:0/1584078343 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f52940a1340 0x7f52940a37f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.291+0000 7f52affff700 1 --2- 172.21.15.2:0/1584078343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a81055a0 0x7f52a81a45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.291+0000 7f52affff700 1 --2- 172.21.15.2:0/1584078343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52a8100560 0x7f52a8072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.291+0000 7f52affff700 1 --2- 172.21.15.2:0/1584078343 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f52a8100f00 0x7f52a8073330 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.291+0000 7f52affff700 1 -- 172.21.15.2:0/1584078343 >> 172.21.15.2:0/1584078343 conn(0x7f52a80781b0 msgr2=0x7f52a8103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.291+0000 7f52affff700 1 -- 172.21.15.2:0/1584078343 shutdown_connections 2023-11-16T16:53:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:46.291+0000 7f52affff700 1 -- 172.21.15.2:0/1584078343 wait complete. 2023-11-16T16:53:47.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:47 smithi155 ceph-mon[131607]: pgmap v119: 129 pgs: 129 active+clean; 208 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.4 MiB/s wr, 319 op/s 2023-11-16T16:53:47.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:47 smithi002 ceph-mon[128712]: pgmap v119: 129 pgs: 129 active+clean; 208 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.4 MiB/s wr, 319 op/s 2023-11-16T16:53:47.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:47 smithi141 ceph-mon[131184]: pgmap v119: 129 pgs: 129 active+clean; 208 MiB data, 673 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.4 MiB/s wr, 319 op/s 2023-11-16T16:53:49.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:49 smithi155 ceph-mon[131607]: pgmap v120: 129 pgs: 129 active+clean; 217 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.4 MiB/s wr, 1.15k op/s 2023-11-16T16:53:49.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:49 smithi002 ceph-mon[128712]: pgmap v120: 129 pgs: 129 active+clean; 217 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.4 MiB/s wr, 1.15k op/s 2023-11-16T16:53:49.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:49 smithi141 ceph-mon[131184]: pgmap v120: 129 pgs: 129 active+clean; 217 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.4 MiB/s wr, 1.15k op/s 2023-11-16T16:53:51.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:51 smithi155 ceph-mon[131607]: pgmap v121: 129 pgs: 129 active+clean; 217 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 837 op/s 2023-11-16T16:53:51.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:51 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:53:51.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:51 smithi002 ceph-mon[128712]: pgmap v121: 129 pgs: 129 active+clean; 217 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 837 op/s 2023-11-16T16:53:51.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:51 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:53:51.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:51 smithi141 ceph-mon[131184]: pgmap v121: 129 pgs: 129 active+clean; 217 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.5 MiB/s wr, 837 op/s 2023-11-16T16:53:51.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:51 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:53:53.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:53 smithi155 ceph-mon[131607]: pgmap v122: 129 pgs: 129 active+clean; 223 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 1.00k op/s 2023-11-16T16:53:53.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:53 smithi002 ceph-mon[128712]: pgmap v122: 129 pgs: 129 active+clean; 223 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 1.00k op/s 2023-11-16T16:53:53.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:53 smithi141 ceph-mon[131184]: pgmap v122: 129 pgs: 129 active+clean; 223 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 1.00k op/s 2023-11-16T16:53:55.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:55 smithi155 ceph-mon[131607]: pgmap v123: 129 pgs: 129 active+clean; 223 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 992 op/s 2023-11-16T16:53:55.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:55 smithi002 ceph-mon[128712]: pgmap v123: 129 pgs: 129 active+clean; 223 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 992 op/s 2023-11-16T16:53:55.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:55 smithi141 ceph-mon[131184]: pgmap v123: 129 pgs: 129 active+clean; 223 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 992 op/s 2023-11-16T16:53:57.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:57 smithi002 ceph-mon[128712]: pgmap v124: 129 pgs: 129 active+clean; 223 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 992 op/s 2023-11-16T16:53:57.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:57 smithi141 ceph-mon[131184]: pgmap v124: 129 pgs: 129 active+clean; 223 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 992 op/s 2023-11-16T16:53:57.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:57 smithi155 ceph-mon[131607]: pgmap v124: 129 pgs: 129 active+clean; 223 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 992 op/s 2023-11-16T16:53:59.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:53:59 smithi002 ceph-mon[128712]: pgmap v125: 129 pgs: 129 active+clean; 246 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.4 MiB/s wr, 1.08k op/s 2023-11-16T16:53:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.498+0000 7fe428bff700 1 -- 172.21.15.2:0/686356855 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4240736d0 msgr2=0x7fe424113630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.498+0000 7fe428bff700 1 --2- 172.21.15.2:0/686356855 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4240736d0 0x7fe424113630 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00b0d0 tx=0x7fe41c00b3e0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.499+0000 7fe428bff700 1 -- 172.21.15.2:0/686356855 shutdown_connections 2023-11-16T16:53:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.499+0000 7fe428bff700 1 --2- 172.21.15.2:0/686356855 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4240736d0 0x7fe424113630 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.499+0000 7fe428bff700 1 --2- 172.21.15.2:0/686356855 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe424074400 0x7fe4240747d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.499+0000 7fe428bff700 1 --2- 172.21.15.2:0/686356855 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe424072d20 0x7fe424073190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.499+0000 7fe428bff700 1 -- 172.21.15.2:0/686356855 >> 172.21.15.2:0/686356855 conn(0x7fe424077550 msgr2=0x7fe424077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.499+0000 7fe428bff700 1 -- 172.21.15.2:0/686356855 shutdown_connections 2023-11-16T16:53:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.499+0000 7fe428bff700 1 -- 172.21.15.2:0/686356855 wait complete. 2023-11-16T16:53:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.500+0000 7fe428bff700 1 Processor -- start 2023-11-16T16:53:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.500+0000 7fe428bff700 1 -- start start 2023-11-16T16:53:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.500+0000 7fe428bff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe424072d20 0x7fe4241a2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.501+0000 7fe428bff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4240736d0 0x7fe4241a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.501+0000 7fe428bff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe424074400 0x7fe42419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.501+0000 7fe428bff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42419d0f0 con 0x7fe4240736d0 2023-11-16T16:53:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.501+0000 7fe428bff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42419d260 con 0x7fe424074400 2023-11-16T16:53:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.501+0000 7fe428bff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42419d420 con 0x7fe424072d20 2023-11-16T16:53:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.501+0000 7fe42259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe424072d20 0x7fe4241a2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.501+0000 7fe42259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe424072d20 0x7fe4241a2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:45156/0 (socket says 172.21.15.2:45156) 2023-11-16T16:53:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.501+0000 7fe42259c700 1 -- 172.21.15.2:0/4144335759 learned_addr learned my addr 172.21.15.2:0/4144335759 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.501+0000 7fe421d9b700 1 --2- 172.21.15.2:0/4144335759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4240736d0 0x7fe4241a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.501+0000 7fe422d9d700 1 --2- 172.21.15.2:0/4144335759 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe424074400 0x7fe42419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.501+0000 7fe421d9b700 1 -- 172.21.15.2:0/4144335759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe424072d20 msgr2=0x7fe4241a2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.501+0000 7fe421d9b700 1 --2- 172.21.15.2:0/4144335759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe424072d20 0x7fe4241a2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.501+0000 7fe421d9b700 1 -- 172.21.15.2:0/4144335759 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe424074400 msgr2=0x7fe42419c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.501+0000 7fe421d9b700 1 --2- 172.21.15.2:0/4144335759 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe424074400 0x7fe42419c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.501+0000 7fe421d9b700 1 -- 172.21.15.2:0/4144335759 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41c009d00 con 0x7fe4240736d0 2023-11-16T16:53:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.502+0000 7fe42259c700 1 --2- 172.21.15.2:0/4144335759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe424072d20 0x7fe4241a2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:53:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.502+0000 7fe421d9b700 1 --2- 172.21.15.2:0/4144335759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4240736d0 0x7fe4241a2e80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe41800e9f0 tx=0x7fe41800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.502+0000 7fe4137fe700 1 -- 172.21.15.2:0/4144335759 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41800c9c0 con 0x7fe4240736d0 2023-11-16T16:53:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.502+0000 7fe4137fe700 1 -- 172.21.15.2:0/4144335759 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe418010430 con 0x7fe4240736d0 2023-11-16T16:53:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.502+0000 7fe4137fe700 1 -- 172.21.15.2:0/4144335759 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe418018680 con 0x7fe4240736d0 2023-11-16T16:53:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.502+0000 7fe428bff700 1 -- 172.21.15.2:0/4144335759 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe424116970 con 0x7fe4240736d0 2023-11-16T16:53:59.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.503+0000 7fe428bff700 1 -- 172.21.15.2:0/4144335759 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4241167e0 con 0x7fe4240736d0 2023-11-16T16:53:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.507+0000 7fe4137fe700 1 -- 172.21.15.2:0/4144335759 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fe418010860 con 0x7fe4240736d0 2023-11-16T16:53:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.508+0000 7fe43c54db80 1 Processor -- start 2023-11-16T16:53:59.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.509+0000 7fe43c54db80 1 -- start start 2023-11-16T16:53:59.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.509+0000 7fe43c54db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014cf66b0 0x55e014ea01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:59.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.509+0000 7fe43c54db80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014ea0720 0x55e014e9a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:59.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.509+0000 7fe43c54db80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014ea0b90 0x55e014e9ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:59.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.509+0000 7fe43c54db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e014da0000 con 0x55e014cf66b0 2023-11-16T16:53:59.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.509+0000 7fe43c54db80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e014df2840 con 0x55e014ea0b90 2023-11-16T16:53:59.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.509+0000 7fe43c54db80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e014debcd0 con 0x55e014ea0720 2023-11-16T16:53:59.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.509+0000 7fe4137fe700 1 --2- 172.21.15.2:0/4144335759 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fe40c0a1340 0x7fe40c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:59.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.509+0000 7fe42259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014ea0720 0x55e014e9a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:59.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.510+0000 7fe4137fe700 1 -- 172.21.15.2:0/4144335759 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe418014070 con 0x7fe4240736d0 2023-11-16T16:53:59.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.510+0000 7fe422d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014cf66b0 0x55e014ea01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:59.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.510+0000 7fe42259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014ea0720 0x55e014e9a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:45164/0 (socket says 172.21.15.2:45164) 2023-11-16T16:53:59.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.510+0000 7fe42259c700 1 -- 172.21.15.2:0/1099013947 learned_addr learned my addr 172.21.15.2:0/1099013947 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.510+0000 7fe422d9d700 1 --2- 172.21.15.2:0/4144335759 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fe40c0a1340 0x7fe40c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.510+0000 7fe42259c700 1 -- 172.21.15.2:0/1099013947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014ea0b90 msgr2=0x55e014e9ab00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-16T16:53:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.510+0000 7fe42259c700 1 --2- 172.21.15.2:0/1099013947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014ea0b90 0x55e014e9ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.510+0000 7fe42259c700 1 -- 172.21.15.2:0/1099013947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014cf66b0 msgr2=0x55e014ea01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.510+0000 7fe42259c700 1 --2- 172.21.15.2:0/1099013947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014cf66b0 0x55e014ea01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.510+0000 7fe42259c700 1 -- 172.21.15.2:0/1099013947 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e014c6d5d0 con 0x55e014ea0720 2023-11-16T16:53:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.511+0000 7fe42259c700 1 --2- 172.21.15.2:0/1099013947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014ea0720 0x55e014e9a560 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe414011650 tx=0x7fe414011960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=624702c03ef0b88d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.511+0000 7fe422d9d700 1 --2- 172.21.15.2:0/4144335759 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fe40c0a1340 0x7fe40c0a37f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe41c0060b0 tx=0x7fe41c00b3e0 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.511+0000 7fe410ff9700 1 -- 172.21.15.2:0/1099013947 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe414019630 con 0x55e014ea0720 2023-11-16T16:53:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.511+0000 7fe410ff9700 1 -- 172.21.15.2:0/1099013947 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe414019c50 con 0x55e014ea0720 2023-11-16T16:53:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.511+0000 7fe410ff9700 1 -- 172.21.15.2:0/1099013947 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41400ed80 con 0x55e014ea0720 2023-11-16T16:53:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.512+0000 7fe43c54db80 1 -- 172.21.15.2:0/1099013947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014ea0720 msgr2=0x55e014e9a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.512+0000 7fe43c54db80 1 --2- 172.21.15.2:0/1099013947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014ea0720 0x55e014e9a560 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe414011650 tx=0x7fe414011960 comp rx=0 tx=0).stop 2023-11-16T16:53:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.512+0000 7fe43c54db80 1 -- 172.21.15.2:0/1099013947 shutdown_connections 2023-11-16T16:53:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.512+0000 7fe43c54db80 1 --2- 172.21.15.2:0/1099013947 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014ea0b90 0x55e014e9ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.512+0000 7fe43c54db80 1 --2- 172.21.15.2:0/1099013947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014cf66b0 0x55e014ea01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.512+0000 7fe43c54db80 1 --2- 172.21.15.2:0/1099013947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014ea0720 0x55e014e9a560 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.512+0000 7fe43c54db80 1 -- 172.21.15.2:0/1099013947 >> 172.21.15.2:0/1099013947 conn(0x55e014e8ff50 msgr2=0x55e014e95270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.512+0000 7fe43c54db80 1 -- 172.21.15.2:0/1099013947 shutdown_connections 2023-11-16T16:53:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.512+0000 7fe43c54db80 1 -- 172.21.15.2:0/1099013947 wait complete. 2023-11-16T16:53:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.513+0000 7fe43c54db80 1 -- start start 2023-11-16T16:53:59.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:53:59 smithi141 ceph-mon[131184]: pgmap v125: 129 pgs: 129 active+clean; 246 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.4 MiB/s wr, 1.08k op/s 2023-11-16T16:53:59.519 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.513+0000 7fe43c54db80 1 Processor -- start 2023-11-16T16:53:59.519 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.514+0000 7fe43c54db80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014cf66b0 0x55e014ea3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:59.519 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.514+0000 7fe43c54db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014ea0720 0x55e014ea42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:59.519 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.514+0000 7fe43c54db80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014ea0b90 0x55e014ea4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:59.519 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.514+0000 7fe43c54db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e014d8d880 con 0x55e014ea0720 2023-11-16T16:53:59.519 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.514+0000 7fe43c54db80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e014d8d520 con 0x55e014ea0b90 2023-11-16T16:53:59.519 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.514+0000 7fe43c54db80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e014c6f8c0 con 0x55e014cf66b0 2023-11-16T16:53:59.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.514+0000 7fe43c54db80 10 client.0 _ll_register_callbacks cb 0x55e014dab720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:59.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.515+0000 7fe42259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014cf66b0 0x55e014ea3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:59.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.515+0000 7fe42259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014cf66b0 0x55e014ea3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:45166/0 (socket says 172.21.15.2:45166) 2023-11-16T16:53:59.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.515+0000 7fe42259c700 1 -- 172.21.15.2:0/1062274434 learned_addr learned my addr 172.21.15.2:0/1062274434 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:59.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.515+0000 7fe422d9d700 1 --2- 172.21.15.2:0/1062274434 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014ea0b90 0x55e014ea4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:59.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.515+0000 7fe42259c700 1 -- 172.21.15.2:0/1062274434 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014ea0b90 msgr2=0x55e014ea4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:59.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.515+0000 7fe42259c700 1 --2- 172.21.15.2:0/1062274434 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014ea0b90 0x55e014ea4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.515+0000 7fe42259c700 1 -- 172.21.15.2:0/1062274434 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014ea0720 msgr2=0x55e014ea42c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T16:53:59.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.515+0000 7fe421d9b700 1 --2- 172.21.15.2:0/1062274434 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014ea0720 0x55e014ea42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:59.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.515+0000 7fe42259c700 1 --2- 172.21.15.2:0/1062274434 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014ea0720 0x55e014ea42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.516+0000 7fe42259c700 1 -- 172.21.15.2:0/1062274434 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe414009e10 con 0x55e014cf66b0 2023-11-16T16:53:59.520 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.516+0000 7fe422d9d700 1 --2- 172.21.15.2:0/1062274434 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014ea0b90 0x55e014ea4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:53:59.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.516+0000 7fe421d9b700 1 --2- 172.21.15.2:0/1062274434 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014ea0720 0x55e014ea42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:53:59.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.516+0000 7fe42259c700 1 --2- 172.21.15.2:0/1062274434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014cf66b0 0x55e014ea3d80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe414011cb0 tx=0x7fe41400e7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6be794da321e4bf1 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:59.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.516+0000 7fe409ffb700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:53:59.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.516+0000 7fe409ffb700 1 -- 172.21.15.2:0/1062274434 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe414019630 con 0x55e014cf66b0 2023-11-16T16:53:59.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.516+0000 7fe409ffb700 1 -- 172.21.15.2:0/1062274434 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe414019790 con 0x55e014cf66b0 2023-11-16T16:53:59.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.516+0000 7fe43c54db80 1 -- 172.21.15.2:0/1062274434 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e014e8d710 con 0x55e014cf66b0 2023-11-16T16:53:59.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.517+0000 7fe409ffb700 1 -- 172.21.15.2:0/1062274434 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe414019950 con 0x55e014cf66b0 2023-11-16T16:53:59.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.517+0000 7fe43c54db80 10 client.25741 fetch_fsmap learned FSMap version 35 2023-11-16T16:53:59.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.517+0000 7fe43c54db80 1 -- 172.21.15.2:0/1062274434 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x55e014db2130 con 0x55e014cf66b0 2023-11-16T16:53:59.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.517+0000 7fe409ffb700 1 -- 172.21.15.2:0/1062274434 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e014e8d710 con 0x55e014cf66b0 2023-11-16T16:53:59.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.517+0000 7fe409ffb700 1 -- 172.21.15.2:0/1062274434 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fe414004b20 con 0x55e014cf66b0 2023-11-16T16:53:59.521 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.517+0000 7fe43c54db80 10 client.25741 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:53:59.522 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.517+0000 7fe43c54db80 10 client.25741 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:59.522 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.517+0000 7fe43c54db80 20 client.25741 populate_metadata read hostname 'smithi002' 2023-11-16T16:53:59.522 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.517+0000 7fe43c54db80 1 --2- 172.21.15.2:0/1062274434 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55e014ec4ec0 0x55e014ec72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:59.522 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.517+0000 7fe43c54db80 4 client.25741 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:53:59.522 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.517+0000 7fe43c54db80 1 -- 172.21.15.2:0/1062274434 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e014dae040 con 0x55e014ec4ec0 2023-11-16T16:53:59.522 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.518+0000 7fe421d9b700 1 --2- 172.21.15.2:0/1062274434 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55e014ec4ec0 0x55e014ec72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:59.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.518+0000 7fe421d9b700 1 --2- 172.21.15.2:0/1062274434 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55e014ec4ec0 0x55e014ec72d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc4b19481ba8daea server_cookie=4f7e1ff284cfabbc in_seq=0 out_seq=0 2023-11-16T16:53:59.523 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.518+0000 7fe409ffb700 10 client.25741 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:53:59.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.521+0000 7fe409ffb700 1 -- 172.21.15.2:0/1062274434 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55e014dae040 con 0x55e014ec4ec0 2023-11-16T16:53:59.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.521+0000 7fe409ffb700 10 client.25741 handle_command_reply: tid=0 2023-11-16T16:53:59.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.521+0000 7fe43c54db80 1 client.25741 shutdown 2023-11-16T16:53:59.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.521+0000 7fe43c54db80 1 -- 172.21.15.2:0/1062274434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014cf66b0 msgr2=0x55e014ea3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:59.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.521+0000 7fe43c54db80 1 --2- 172.21.15.2:0/1062274434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014cf66b0 0x55e014ea3d80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe414011cb0 tx=0x7fe41400e7f0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.521+0000 7fe43c54db80 1 -- 172.21.15.2:0/1062274434 shutdown_connections 2023-11-16T16:53:59.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.521+0000 7fe43c54db80 1 --2- 172.21.15.2:0/1062274434 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55e014ec4ec0 0x55e014ec72d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.521+0000 7fe43c54db80 1 --2- 172.21.15.2:0/1062274434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014cf66b0 0x55e014ea3d80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.521+0000 7fe43c54db80 1 --2- 172.21.15.2:0/1062274434 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014ea0b90 0x55e014ea4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.521+0000 7fe43c54db80 1 --2- 172.21.15.2:0/1062274434 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014ea0720 0x55e014ea42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.521+0000 7fe409ffb700 0 client.25741 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:53:59.525 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.521+0000 7fe43c54db80 1 -- 172.21.15.2:0/1062274434 >> 172.21.15.2:0/1062274434 conn(0x55e014e8ff50 msgr2=0x55e014e947e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:59.525 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.521+0000 7fe43c54db80 1 -- 172.21.15.2:0/1062274434 shutdown_connections 2023-11-16T16:53:59.525 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.521+0000 7fe43c54db80 1 -- 172.21.15.2:0/1062274434 wait complete. 2023-11-16T16:53:59.525 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.522+0000 7fe43c54db80 20 client.25741 trim_cache size 0 max 16384 2023-11-16T16:53:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.541+0000 7fe43c54db80 1 Processor -- start 2023-11-16T16:53:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.541+0000 7fe43c54db80 1 -- start start 2023-11-16T16:53:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.541+0000 7fe43c54db80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014cf66b0 0x55e014eab120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.541+0000 7fe43c54db80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014ea0720 0x55e014ea4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.541+0000 7fe43c54db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014ea0b90 0x55e014e94f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.541+0000 7fe43c54db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e014e96470 con 0x55e014ea0b90 2023-11-16T16:53:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.541+0000 7fe43c54db80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e014ea22e0 con 0x55e014cf66b0 2023-11-16T16:53:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.541+0000 7fe43c54db80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e014e9ea20 con 0x55e014ea0720 2023-11-16T16:53:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.542+0000 7fe42259c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014cf66b0 0x55e014eab120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.542+0000 7fe421d9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014ea0720 0x55e014ea4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.542+0000 7fe42259c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014cf66b0 0x55e014eab120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:56808/0 (socket says 172.21.15.2:56808) 2023-11-16T16:53:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.542+0000 7fe422d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014ea0b90 0x55e014e94f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.542+0000 7fe42259c700 1 -- 172.21.15.2:0/1395395357 learned_addr learned my addr 172.21.15.2:0/1395395357 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.543+0000 7fe421d9b700 1 -- 172.21.15.2:0/1395395357 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014cf66b0 msgr2=0x55e014eab120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.543+0000 7fe421d9b700 1 --2- 172.21.15.2:0/1395395357 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014cf66b0 0x55e014eab120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.543+0000 7fe421d9b700 1 -- 172.21.15.2:0/1395395357 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014ea0b90 msgr2=0x55e014e94f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.543+0000 7fe421d9b700 1 --2- 172.21.15.2:0/1395395357 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014ea0b90 0x55e014e94f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.543+0000 7fe421d9b700 1 -- 172.21.15.2:0/1395395357 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e014db1e80 con 0x55e014ea0720 2023-11-16T16:53:59.546 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.543+0000 7fe421d9b700 1 --2- 172.21.15.2:0/1395395357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014ea0720 0x55e014ea4630 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe4180ca530 tx=0x7fe418010c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d044fd1c0179e437 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.543+0000 7fe42259c700 1 --2- 172.21.15.2:0/1395395357 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014cf66b0 0x55e014eab120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:53:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.543+0000 7fe422d9d700 1 --2- 172.21.15.2:0/1395395357 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014ea0b90 0x55e014e94f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:53:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.544+0000 7fe4117fa700 1 -- 172.21.15.2:0/1395395357 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41809dee0 con 0x55e014ea0720 2023-11-16T16:53:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.544+0000 7fe4117fa700 1 -- 172.21.15.2:0/1395395357 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe4180a2300 con 0x55e014ea0720 2023-11-16T16:53:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.544+0000 7fe4117fa700 1 -- 172.21.15.2:0/1395395357 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4180993a0 con 0x55e014ea0720 2023-11-16T16:53:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.545+0000 7fe43c54db80 1 -- 172.21.15.2:0/1395395357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014ea0720 msgr2=0x55e014ea4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:53:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.545+0000 7fe43c54db80 1 --2- 172.21.15.2:0/1395395357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014ea0720 0x55e014ea4630 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe4180ca530 tx=0x7fe418010c60 comp rx=0 tx=0).stop 2023-11-16T16:53:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.545+0000 7fe43c54db80 1 -- 172.21.15.2:0/1395395357 shutdown_connections 2023-11-16T16:53:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.545+0000 7fe43c54db80 1 --2- 172.21.15.2:0/1395395357 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014ea0b90 0x55e014e94f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.545+0000 7fe43c54db80 1 --2- 172.21.15.2:0/1395395357 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014cf66b0 0x55e014eab120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.545+0000 7fe43c54db80 1 --2- 172.21.15.2:0/1395395357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014ea0720 0x55e014ea4630 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.545+0000 7fe43c54db80 1 -- 172.21.15.2:0/1395395357 >> 172.21.15.2:0/1395395357 conn(0x55e014e8ff50 msgr2=0x55e014ec1350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:53:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.545+0000 7fe43c54db80 1 -- 172.21.15.2:0/1395395357 shutdown_connections 2023-11-16T16:53:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.545+0000 7fe43c54db80 1 -- 172.21.15.2:0/1395395357 wait complete. 2023-11-16T16:53:59.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.545+0000 7fe43c54db80 1 -- start start 2023-11-16T16:53:59.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.546+0000 7fe43c54db80 1 Processor -- start 2023-11-16T16:53:59.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.546+0000 7fe43c54db80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014cf66b0 0x55e014ea7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:59.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.546+0000 7fe43c54db80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014ea0720 0x55e014ea7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:59.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.546+0000 7fe43c54db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014ea0b90 0x55e014ea7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:59.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.546+0000 7fe43c54db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e014f17ea0 con 0x55e014ea0b90 2023-11-16T16:53:59.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.546+0000 7fe43c54db80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e014f18010 con 0x55e014ea0720 2023-11-16T16:53:59.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.546+0000 7fe43c54db80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e014f181d0 con 0x55e014cf66b0 2023-11-16T16:53:59.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.546+0000 7fe43c54db80 10 client.0 _ll_register_callbacks cb 0x7fe414005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:53:59.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.546+0000 7fe42259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014cf66b0 0x55e014ea7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:59.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.546+0000 7fe422d9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014ea0b90 0x55e014ea7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:59.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.546+0000 7fe42259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014cf66b0 0x55e014ea7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:45188/0 (socket says 172.21.15.2:45188) 2023-11-16T16:53:59.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.546+0000 7fe421d9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014ea0720 0x55e014ea7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:59.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.546+0000 7fe42259c700 1 -- 172.21.15.2:0/4072503053 learned_addr learned my addr 172.21.15.2:0/4072503053 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:53:59.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.547+0000 7fe42259c700 1 -- 172.21.15.2:0/4072503053 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014ea0720 msgr2=0x55e014ea7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:59.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.547+0000 7fe42259c700 1 --2- 172.21.15.2:0/4072503053 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014ea0720 0x55e014ea7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.547+0000 7fe42259c700 1 -- 172.21.15.2:0/4072503053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014ea0b90 msgr2=0x55e014ea7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:53:59.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.547+0000 7fe42259c700 1 --2- 172.21.15.2:0/4072503053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014ea0b90 0x55e014ea7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:53:59.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.547+0000 7fe42259c700 1 -- 172.21.15.2:0/4072503053 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe424113fb0 con 0x55e014cf66b0 2023-11-16T16:53:59.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.547+0000 7fe42259c700 1 --2- 172.21.15.2:0/4072503053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014cf66b0 0x55e014ea7100 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe414000c00 tx=0x7fe414019660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b47b8118dec5e28c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:53:59.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.547+0000 7fe409ffb700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:53:59.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.547+0000 7fe409ffb700 1 -- 172.21.15.2:0/4072503053 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe414030070 con 0x55e014cf66b0 2023-11-16T16:53:59.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.547+0000 7fe43c54db80 1 -- 172.21.15.2:0/4072503053 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe414019c50 con 0x55e014cf66b0 2023-11-16T16:53:59.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.547+0000 7fe409ffb700 1 -- 172.21.15.2:0/4072503053 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fe414000e40 con 0x55e014cf66b0 2023-11-16T16:53:59.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.548+0000 7fe409ffb700 1 -- 172.21.15.2:0/4072503053 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4140224a0 con 0x55e014cf66b0 2023-11-16T16:53:59.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.548+0000 7fe43c54db80 10 client.25747 fetch_fsmap learned FSMap version 35 2023-11-16T16:53:59.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.548+0000 7fe43c54db80 1 -- 172.21.15.2:0/4072503053 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x7fe4241167e0 con 0x55e014cf66b0 2023-11-16T16:53:59.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.548+0000 7fe409ffb700 1 -- 172.21.15.2:0/4072503053 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe414019c50 con 0x55e014cf66b0 2023-11-16T16:53:59.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.548+0000 7fe409ffb700 1 -- 172.21.15.2:0/4072503053 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fe414003680 con 0x55e014cf66b0 2023-11-16T16:53:59.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.548+0000 7fe43c54db80 10 client.25747 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:53:59.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.548+0000 7fe43c54db80 10 client.25747 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:53:59.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.548+0000 7fe43c54db80 20 client.25747 populate_metadata read hostname 'smithi002' 2023-11-16T16:53:59.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.548+0000 7fe43c54db80 1 --2- 172.21.15.2:0/4072503053 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55e014f193b0 0x55e014f1b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:53:59.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.548+0000 7fe43c54db80 4 client.25747 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:53:59.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.548+0000 7fe43c54db80 1 -- 172.21.15.2:0/4072503053 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e014dba600 con 0x55e014f193b0 2023-11-16T16:53:59.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.549+0000 7fe421d9b700 1 --2- 172.21.15.2:0/4072503053 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55e014f193b0 0x55e014f1b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:53:59.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.549+0000 7fe421d9b700 1 --2- 172.21.15.2:0/4072503053 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55e014f193b0 0x55e014f1b790 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b83a9a313df7ab4 server_cookie=e33985db1e237d84 in_seq=0 out_seq=0 2023-11-16T16:53:59.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:53:59.549+0000 7fe409ffb700 10 client.25747 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:53:59.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:53:59 smithi155 ceph-mon[131607]: pgmap v125: 129 pgs: 129 active+clean; 246 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.4 MiB/s wr, 1.08k op/s 2023-11-16T16:54:01.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.365+0000 7fe409ffb700 1 -- 172.21.15.2:0/4072503053 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e014dba600 con 0x55e014f193b0 2023-11-16T16:54:01.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.365+0000 7fe409ffb700 10 client.25747 handle_command_reply: tid=0 2023-11-16T16:54:01.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.365+0000 7fe43c54db80 1 client.25747 shutdown 2023-11-16T16:54:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.367+0000 7fe43c54db80 1 -- 172.21.15.2:0/4072503053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014cf66b0 msgr2=0x55e014ea7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.367+0000 7fe43c54db80 1 --2- 172.21.15.2:0/4072503053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014cf66b0 0x55e014ea7100 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe414000c00 tx=0x7fe414019660 comp rx=0 tx=0).stop 2023-11-16T16:54:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.368+0000 7fe43c54db80 1 -- 172.21.15.2:0/4072503053 shutdown_connections 2023-11-16T16:54:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.368+0000 7fe43c54db80 1 --2- 172.21.15.2:0/4072503053 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55e014f193b0 0x55e014f1b790 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.368+0000 7fe43c54db80 1 --2- 172.21.15.2:0/4072503053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e014cf66b0 0x55e014ea7100 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.368+0000 7fe43c54db80 1 --2- 172.21.15.2:0/4072503053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55e014ea0b90 0x55e014ea7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.368+0000 7fe43c54db80 1 --2- 172.21.15.2:0/4072503053 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55e014ea0720 0x55e014ea7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.368+0000 7fe43c54db80 1 -- 172.21.15.2:0/4072503053 >> 172.21.15.2:0/4072503053 conn(0x55e014e8ff50 msgr2=0x55e014ea4670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:01.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.368+0000 7fe409ffb700 0 client.25747 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:54:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.369+0000 7fe43c54db80 1 -- 172.21.15.2:0/4072503053 shutdown_connections 2023-11-16T16:54:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.369+0000 7fe43c54db80 1 -- 172.21.15.2:0/4072503053 wait complete. 2023-11-16T16:54:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.369+0000 7fe43c54db80 20 client.25747 trim_cache size 0 max 16384 2023-11-16T16:54:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:54:01.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:54:01.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:54:01.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:54:01.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.371+0000 7fe428bff700 1 -- 172.21.15.2:0/4144335759 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fe40c0a1340 msgr2=0x7fe40c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:01.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.371+0000 7fe428bff700 1 --2- 172.21.15.2:0/4144335759 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fe40c0a1340 0x7fe40c0a37f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe41c0060b0 tx=0x7fe41c00b3e0 comp rx=0 tx=0).stop 2023-11-16T16:54:01.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.371+0000 7fe428bff700 1 -- 172.21.15.2:0/4144335759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4240736d0 msgr2=0x7fe4241a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:01.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.371+0000 7fe428bff700 1 --2- 172.21.15.2:0/4144335759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4240736d0 0x7fe4241a2e80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe41800e9f0 tx=0x7fe41800ed00 comp rx=0 tx=0).stop 2023-11-16T16:54:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.371+0000 7fe428bff700 1 -- 172.21.15.2:0/4144335759 shutdown_connections 2023-11-16T16:54:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.371+0000 7fe428bff700 1 --2- 172.21.15.2:0/4144335759 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fe40c0a1340 0x7fe40c0a37f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.371+0000 7fe428bff700 1 --2- 172.21.15.2:0/4144335759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe424072d20 0x7fe4241a2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.371+0000 7fe428bff700 1 --2- 172.21.15.2:0/4144335759 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fe424074400 0x7fe42419c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.371+0000 7fe428bff700 1 --2- 172.21.15.2:0/4144335759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4240736d0 0x7fe4241a2e80 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.371+0000 7fe428bff700 1 -- 172.21.15.2:0/4144335759 >> 172.21.15.2:0/4144335759 conn(0x7fe424077550 msgr2=0x7fe424112690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.371+0000 7fe428bff700 1 -- 172.21.15.2:0/4144335759 shutdown_connections 2023-11-16T16:54:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:01.371+0000 7fe428bff700 1 -- 172.21.15.2:0/4144335759 wait complete. 2023-11-16T16:54:01.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:01 smithi002 ceph-mon[128712]: pgmap v126: 129 pgs: 129 active+clean; 246 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.4 MiB/s wr, 251 op/s 2023-11-16T16:54:01.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:01 smithi141 ceph-mon[131184]: pgmap v126: 129 pgs: 129 active+clean; 246 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.4 MiB/s wr, 251 op/s 2023-11-16T16:54:01.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:01 smithi155 ceph-mon[131607]: pgmap v126: 129 pgs: 129 active+clean; 246 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.4 MiB/s wr, 251 op/s 2023-11-16T16:54:03.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:03 smithi002 ceph-mon[128712]: pgmap v127: 129 pgs: 129 active+clean; 279 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.2 MiB/s wr, 381 op/s 2023-11-16T16:54:03.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:03 smithi141 ceph-mon[131184]: pgmap v127: 129 pgs: 129 active+clean; 279 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.2 MiB/s wr, 381 op/s 2023-11-16T16:54:03.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:03 smithi155 ceph-mon[131607]: pgmap v127: 129 pgs: 129 active+clean; 279 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 6.2 MiB/s wr, 381 op/s 2023-11-16T16:54:05.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:05 smithi002 ceph-mon[128712]: pgmap v128: 129 pgs: 129 active+clean; 279 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.7 MiB/s wr, 217 op/s 2023-11-16T16:54:05.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:05 smithi141 ceph-mon[131184]: pgmap v128: 129 pgs: 129 active+clean; 279 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.7 MiB/s wr, 217 op/s 2023-11-16T16:54:05.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:05 smithi155 ceph-mon[131607]: pgmap v128: 129 pgs: 129 active+clean; 279 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.7 MiB/s wr, 217 op/s 2023-11-16T16:54:07.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:07 smithi002 ceph-mon[128712]: pgmap v129: 129 pgs: 129 active+clean; 279 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.7 MiB/s wr, 217 op/s 2023-11-16T16:54:07.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:07 smithi141 ceph-mon[131184]: pgmap v129: 129 pgs: 129 active+clean; 279 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.7 MiB/s wr, 217 op/s 2023-11-16T16:54:07.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:07 smithi155 ceph-mon[131607]: pgmap v129: 129 pgs: 129 active+clean; 279 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.7 MiB/s wr, 217 op/s 2023-11-16T16:54:09.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:09 smithi002 ceph-mon[128712]: pgmap v130: 129 pgs: 129 active+clean; 311 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 8.8 MiB/s wr, 585 op/s 2023-11-16T16:54:09.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:09 smithi141 ceph-mon[131184]: pgmap v130: 129 pgs: 129 active+clean; 311 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 8.8 MiB/s wr, 585 op/s 2023-11-16T16:54:09.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:09 smithi155 ceph-mon[131607]: pgmap v130: 129 pgs: 129 active+clean; 311 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 8.8 MiB/s wr, 585 op/s 2023-11-16T16:54:11.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:11 smithi002 ceph-mon[128712]: pgmap v131: 129 pgs: 129 active+clean; 311 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.9 MiB/s wr, 497 op/s 2023-11-16T16:54:11.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:11 smithi141 ceph-mon[131184]: pgmap v131: 129 pgs: 129 active+clean; 311 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.9 MiB/s wr, 497 op/s 2023-11-16T16:54:11.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:11 smithi155 ceph-mon[131607]: pgmap v131: 129 pgs: 129 active+clean; 311 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.9 MiB/s wr, 497 op/s 2023-11-16T16:54:13.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:13 smithi002 ceph-mon[128712]: pgmap v132: 129 pgs: 129 active+clean; 345 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.8 MiB/s wr, 675 op/s 2023-11-16T16:54:13.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:13 smithi141 ceph-mon[131184]: pgmap v132: 129 pgs: 129 active+clean; 345 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.8 MiB/s wr, 675 op/s 2023-11-16T16:54:13.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:13 smithi155 ceph-mon[131607]: pgmap v132: 129 pgs: 129 active+clean; 345 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.8 MiB/s wr, 675 op/s 2023-11-16T16:54:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.561+0000 7fcc149ff700 1 -- 172.21.15.2:0/140361914 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc10100f00 msgr2=0x7fcc10104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.561+0000 7fcc149ff700 1 --2- 172.21.15.2:0/140361914 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc10100f00 0x7fcc10104ec0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009990 tx=0x7fcbf8009ca0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.561+0000 7fcc149ff700 1 -- 172.21.15.2:0/140361914 shutdown_connections 2023-11-16T16:54:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.561+0000 7fcc149ff700 1 --2- 172.21.15.2:0/140361914 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc101055a0 0x7fcc1010d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.561+0000 7fcc149ff700 1 --2- 172.21.15.2:0/140361914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc10100560 0x7fcc10100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.561+0000 7fcc149ff700 1 --2- 172.21.15.2:0/140361914 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc10100f00 0x7fcc10104ec0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.561+0000 7fcc149ff700 1 -- 172.21.15.2:0/140361914 >> 172.21.15.2:0/140361914 conn(0x7fcc100781b0 msgr2=0x7fcc100785b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.561+0000 7fcc149ff700 1 -- 172.21.15.2:0/140361914 shutdown_connections 2023-11-16T16:54:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.562+0000 7fcc149ff700 1 -- 172.21.15.2:0/140361914 wait complete. 2023-11-16T16:54:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.563+0000 7fcc149ff700 1 Processor -- start 2023-11-16T16:54:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.563+0000 7fcc149ff700 1 -- start start 2023-11-16T16:54:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.563+0000 7fcc149ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc10100560 0x7fcc10196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.563+0000 7fcc149ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc10100f00 0x7fcc10196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.563+0000 7fcc149ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc101055a0 0x7fcc1019ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.564+0000 7fcc149ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1019b3c0 con 0x7fcc101055a0 2023-11-16T16:54:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.564+0000 7fcc149ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1019b530 con 0x7fcc10100560 2023-11-16T16:54:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.564+0000 7fcc149ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1019b6f0 con 0x7fcc10100f00 2023-11-16T16:54:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.564+0000 7fcc0ed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc101055a0 0x7fcc1019ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.564+0000 7fcc0e59c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc10100560 0x7fcc10196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.564+0000 7fcc0ed9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc101055a0 0x7fcc1019ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59814/0 (socket says 172.21.15.2:59814) 2023-11-16T16:54:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.564+0000 7fcc0ed9d700 1 -- 172.21.15.2:0/3593614253 learned_addr learned my addr 172.21.15.2:0/3593614253 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:54:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.564+0000 7fcc07fff700 1 --2- 172.21.15.2:0/3593614253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc10100f00 0x7fcc10196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.564+0000 7fcc0ed9d700 1 -- 172.21.15.2:0/3593614253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc10100f00 msgr2=0x7fcc10196970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.564+0000 7fcc0ed9d700 1 --2- 172.21.15.2:0/3593614253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc10100f00 0x7fcc10196970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.564+0000 7fcc0ed9d700 1 -- 172.21.15.2:0/3593614253 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc10100560 msgr2=0x7fcc10196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.564+0000 7fcc0ed9d700 1 --2- 172.21.15.2:0/3593614253 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc10100560 0x7fcc10196430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.564+0000 7fcc0ed9d700 1 -- 172.21.15.2:0/3593614253 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf80095f0 con 0x7fcc101055a0 2023-11-16T16:54:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.564+0000 7fcc0e59c700 1 --2- 172.21.15.2:0/3593614253 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc10100560 0x7fcc10196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:54:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.564+0000 7fcc0ed9d700 1 --2- 172.21.15.2:0/3593614253 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc101055a0 0x7fcc1019ad80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800ef30 tx=0x7fcc0800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.565+0000 7fcc077fe700 1 -- 172.21.15.2:0/3593614253 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08018070 con 0x7fcc101055a0 2023-11-16T16:54:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.565+0000 7fcc077fe700 1 -- 172.21.15.2:0/3593614253 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc0800f040 con 0x7fcc101055a0 2023-11-16T16:54:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.565+0000 7fcc149ff700 1 -- 172.21.15.2:0/3593614253 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc10110370 con 0x7fcc101055a0 2023-11-16T16:54:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.565+0000 7fcc077fe700 1 -- 172.21.15.2:0/3593614253 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0800cd00 con 0x7fcc101055a0 2023-11-16T16:54:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.565+0000 7fcc149ff700 1 -- 172.21.15.2:0/3593614253 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc101101e0 con 0x7fcc101055a0 2023-11-16T16:54:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.568+0000 7fcc077fe700 1 -- 172.21.15.2:0/3593614253 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fcc08003a00 con 0x7fcc101055a0 2023-11-16T16:54:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.569+0000 7fcc2824db80 1 Processor -- start 2023-11-16T16:54:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.569+0000 7fcc2824db80 1 -- start start 2023-11-16T16:54:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.570+0000 7fcc077fe700 1 --2- 172.21.15.2:0/3593614253 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fcbfc0a12c0 0x7fcbfc0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.570+0000 7fcc2824db80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cc3a6b0 0x557f3cde41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.570+0000 7fcc2824db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cde4720 0x557f3cdde560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.570+0000 7fcc2824db80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cde4b90 0x557f3cddeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.570+0000 7fcc2824db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f3cce4000 con 0x557f3cde4720 2023-11-16T16:54:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.570+0000 7fcc077fe700 1 -- 172.21.15.2:0/3593614253 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fcc080ccb60 con 0x7fcc101055a0 2023-11-16T16:54:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.570+0000 7fcc2824db80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f3cd36840 con 0x557f3cc3a6b0 2023-11-16T16:54:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.570+0000 7fcc2824db80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f3cd2fcd0 con 0x557f3cde4b90 2023-11-16T16:54:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.570+0000 7fcc0e59c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cc3a6b0 0x557f3cde41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.570+0000 7fcc07fff700 1 --2- 172.21.15.2:0/3593614253 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fcbfc0a12c0 0x7fcbfc0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.570+0000 7fcc0e59c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cc3a6b0 0x557f3cde41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:44302/0 (socket says 172.21.15.2:44302) 2023-11-16T16:54:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.570+0000 7fcc0e59c700 1 -- 172.21.15.2:0/3797044148 learned_addr learned my addr 172.21.15.2:0/3797044148 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:54:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.570+0000 7fcc0ed9d700 1 --2- 172.21.15.2:0/3797044148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cde4b90 0x557f3cddeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.570+0000 7fcc07fff700 1 --2- 172.21.15.2:0/3797044148 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cde4720 0x557f3cdde560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.571+0000 7fcc0e59c700 1 -- 172.21.15.2:0/3797044148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cde4b90 msgr2=0x557f3cddeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.571+0000 7fcc0e59c700 1 --2- 172.21.15.2:0/3797044148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cde4b90 0x557f3cddeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.571+0000 7fcc0e59c700 1 -- 172.21.15.2:0/3797044148 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cde4720 msgr2=0x557f3cdde560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.571+0000 7fcc0e59c700 1 --2- 172.21.15.2:0/3797044148 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cde4720 0x557f3cdde560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.571+0000 7fcc0e59c700 1 -- 172.21.15.2:0/3797044148 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f3cbb15d0 con 0x557f3cc3a6b0 2023-11-16T16:54:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.571+0000 7fcc0e59c700 1 --2- 172.21.15.2:0/3797044148 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cc3a6b0 0x557f3cde41e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcc00009910 tx=0x7fcc00009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c6a740e5029a88d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.571+0000 7fcc07fff700 1 --2- 172.21.15.2:0/3593614253 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fcbfc0a12c0 0x7fcbfc0a3770 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8000c00 tx=0x7fcbf801a040 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.571+0000 7fcc04ff9700 1 -- 172.21.15.2:0/3797044148 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc00015050 con 0x557f3cc3a6b0 2023-11-16T16:54:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.571+0000 7fcc04ff9700 1 -- 172.21.15.2:0/3797044148 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc00016040 con 0x557f3cc3a6b0 2023-11-16T16:54:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.571+0000 7fcc04ff9700 1 -- 172.21.15.2:0/3797044148 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0001a960 con 0x557f3cc3a6b0 2023-11-16T16:54:14.578 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.572+0000 7fcc2824db80 1 -- 172.21.15.2:0/3797044148 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cc3a6b0 msgr2=0x557f3cde41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:14.578 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.572+0000 7fcc2824db80 1 --2- 172.21.15.2:0/3797044148 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cc3a6b0 0x557f3cde41e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcc00009910 tx=0x7fcc00009260 comp rx=0 tx=0).stop 2023-11-16T16:54:14.578 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.572+0000 7fcc2824db80 1 -- 172.21.15.2:0/3797044148 shutdown_connections 2023-11-16T16:54:14.578 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.572+0000 7fcc2824db80 1 --2- 172.21.15.2:0/3797044148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cde4b90 0x557f3cddeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.578 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.572+0000 7fcc2824db80 1 --2- 172.21.15.2:0/3797044148 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cc3a6b0 0x557f3cde41e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.578 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.572+0000 7fcc2824db80 1 --2- 172.21.15.2:0/3797044148 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cde4720 0x557f3cdde560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.578 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.572+0000 7fcc2824db80 1 -- 172.21.15.2:0/3797044148 >> 172.21.15.2:0/3797044148 conn(0x557f3cdd3f50 msgr2=0x557f3cdd9270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:14.578 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.572+0000 7fcc2824db80 1 -- 172.21.15.2:0/3797044148 shutdown_connections 2023-11-16T16:54:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.572+0000 7fcc2824db80 1 -- 172.21.15.2:0/3797044148 wait complete. 2023-11-16T16:54:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.573+0000 7fcc2824db80 1 -- start start 2023-11-16T16:54:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.573+0000 7fcc2824db80 1 Processor -- start 2023-11-16T16:54:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.573+0000 7fcc2824db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cc3a6b0 0x557f3cdea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.573+0000 7fcc2824db80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cde4720 0x557f3cde7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.573+0000 7fcc2824db80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cde4b90 0x557f3cde8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.573+0000 7fcc2824db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f3ccd1880 con 0x557f3cc3a6b0 2023-11-16T16:54:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.573+0000 7fcc2824db80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f3ccd1520 con 0x557f3cde4720 2023-11-16T16:54:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.573+0000 7fcc2824db80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f3cbb38c0 con 0x557f3cde4b90 2023-11-16T16:54:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.573+0000 7fcc2824db80 10 client.0 _ll_register_callbacks cb 0x557f3ccef720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.574+0000 7fcc0ed9d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cde4b90 0x557f3cde8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.574+0000 7fcc0ed9d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cde4b90 0x557f3cde8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:57730/0 (socket says 172.21.15.2:57730) 2023-11-16T16:54:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.574+0000 7fcc0ed9d700 1 -- 172.21.15.2:0/1227264287 learned_addr learned my addr 172.21.15.2:0/1227264287 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:54:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.574+0000 7fcc0e59c700 1 --2- 172.21.15.2:0/1227264287 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cc3a6b0 0x557f3cdea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.574+0000 7fcc07fff700 1 --2- 172.21.15.2:0/1227264287 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cde4720 0x557f3cde7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.574+0000 7fcc0ed9d700 1 -- 172.21.15.2:0/1227264287 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cde4720 msgr2=0x557f3cde7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.574+0000 7fcc0ed9d700 1 --2- 172.21.15.2:0/1227264287 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cde4720 0x557f3cde7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.574+0000 7fcc0ed9d700 1 -- 172.21.15.2:0/1227264287 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cc3a6b0 msgr2=0x557f3cdea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.574+0000 7fcc0ed9d700 1 --2- 172.21.15.2:0/1227264287 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cc3a6b0 0x557f3cdea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.574+0000 7fcc0ed9d700 1 -- 172.21.15.2:0/1227264287 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc000095c0 con 0x557f3cde4b90 2023-11-16T16:54:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.574+0000 7fcc0ed9d700 1 --2- 172.21.15.2:0/1227264287 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cde4b90 0x557f3cde8290 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcc080995f0 tx=0x7fcc08099380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b375b7799cecfae4 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.574+0000 7fcbf5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:54:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.574+0000 7fcbf5ffb700 1 -- 172.21.15.2:0/1227264287 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0809d920 con 0x557f3cde4b90 2023-11-16T16:54:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.574+0000 7fcc2824db80 1 -- 172.21.15.2:0/1227264287 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557f3cdd1710 con 0x557f3cde4b90 2023-11-16T16:54:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.574+0000 7fcbf5ffb700 1 -- 172.21.15.2:0/1227264287 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fcc080a1cb0 con 0x557f3cde4b90 2023-11-16T16:54:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.575+0000 7fcbf5ffb700 1 -- 172.21.15.2:0/1227264287 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc08098de0 con 0x557f3cde4b90 2023-11-16T16:54:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.575+0000 7fcc2824db80 10 client.25759 fetch_fsmap learned FSMap version 35 2023-11-16T16:54:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.575+0000 7fcc2824db80 1 -- 172.21.15.2:0/1227264287 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x557f3ccf6130 con 0x557f3cde4b90 2023-11-16T16:54:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.575+0000 7fcbf5ffb700 1 -- 172.21.15.2:0/1227264287 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557f3cdd1710 con 0x557f3cde4b90 2023-11-16T16:54:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.575+0000 7fcbf5ffb700 1 -- 172.21.15.2:0/1227264287 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fcc080a1820 con 0x557f3cde4b90 2023-11-16T16:54:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.575+0000 7fcc2824db80 10 client.25759 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:54:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.575+0000 7fcc2824db80 10 client.25759 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.575+0000 7fcc2824db80 20 client.25759 populate_metadata read hostname 'smithi002' 2023-11-16T16:54:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.575+0000 7fcc2824db80 1 --2- 172.21.15.2:0/1227264287 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x557f3cdf7e30 0x557f3cdfa240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.575+0000 7fcc2824db80 4 client.25759 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:54:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.575+0000 7fcc2824db80 1 -- 172.21.15.2:0/1227264287 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557f3ccf2040 con 0x557f3cdf7e30 2023-11-16T16:54:14.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.575+0000 7fcc0e59c700 1 --2- 172.21.15.2:0/1227264287 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x557f3cdf7e30 0x557f3cdfa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:14.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.576+0000 7fcc0e59c700 1 --2- 172.21.15.2:0/1227264287 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x557f3cdf7e30 0x557f3cdfa240 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4dade55070d2e65 server_cookie=10e204c989c8d1b7 in_seq=0 out_seq=0 2023-11-16T16:54:14.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.576+0000 7fcbf5ffb700 10 client.25759 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:54:14.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.580+0000 7fcbf5ffb700 1 -- 172.21.15.2:0/1227264287 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x557f3ccf2040 con 0x557f3cdf7e30 2023-11-16T16:54:14.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.580+0000 7fcbf5ffb700 10 client.25759 handle_command_reply: tid=0 2023-11-16T16:54:14.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.580+0000 7fcc2824db80 1 client.25759 shutdown 2023-11-16T16:54:14.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.580+0000 7fcc2824db80 1 -- 172.21.15.2:0/1227264287 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cde4b90 msgr2=0x557f3cde8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:14.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.580+0000 7fcc2824db80 1 --2- 172.21.15.2:0/1227264287 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cde4b90 0x557f3cde8290 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcc080995f0 tx=0x7fcc08099380 comp rx=0 tx=0).stop 2023-11-16T16:54:14.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.580+0000 7fcc2824db80 1 -- 172.21.15.2:0/1227264287 shutdown_connections 2023-11-16T16:54:14.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.580+0000 7fcc2824db80 1 --2- 172.21.15.2:0/1227264287 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x557f3cdf7e30 0x557f3cdfa240 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.580+0000 7fcc2824db80 1 --2- 172.21.15.2:0/1227264287 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cde4b90 0x557f3cde8290 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.580+0000 7fcc2824db80 1 --2- 172.21.15.2:0/1227264287 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cc3a6b0 0x557f3cdea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.580+0000 7fcc2824db80 1 --2- 172.21.15.2:0/1227264287 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cde4720 0x557f3cde7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.580+0000 7fcc2824db80 1 -- 172.21.15.2:0/1227264287 >> 172.21.15.2:0/1227264287 conn(0x557f3cdd3f50 msgr2=0x557f3cdd87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:14.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.580+0000 7fcbf5ffb700 0 client.25759 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:54:14.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.581+0000 7fcc2824db80 1 -- 172.21.15.2:0/1227264287 shutdown_connections 2023-11-16T16:54:14.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.581+0000 7fcc2824db80 1 -- 172.21.15.2:0/1227264287 wait complete. 2023-11-16T16:54:14.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.581+0000 7fcc2824db80 20 client.25759 trim_cache size 0 max 16384 2023-11-16T16:54:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.600+0000 7fcc2824db80 1 Processor -- start 2023-11-16T16:54:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.600+0000 7fcc2824db80 1 -- start start 2023-11-16T16:54:14.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.600+0000 7fcc2824db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cc3a6b0 0x557f3cdef000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.600+0000 7fcc2824db80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cde4720 0x557f3cdea580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.600+0000 7fcc2824db80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cde4b90 0x557f3cdde560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.600+0000 7fcc2824db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f3cdd3b70 con 0x557f3cc3a6b0 2023-11-16T16:54:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.600+0000 7fcc2824db80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f3cdda400 con 0x557f3cde4b90 2023-11-16T16:54:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.600+0000 7fcc2824db80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f3cde61e0 con 0x557f3cde4720 2023-11-16T16:54:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.600+0000 7fcc0e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cc3a6b0 0x557f3cdef000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.600+0000 7fcc0ed9d700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cde4b90 0x557f3cdde560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.600+0000 7fcc0e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cc3a6b0 0x557f3cdef000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59844/0 (socket says 172.21.15.2:59844) 2023-11-16T16:54:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.600+0000 7fcc0e59c700 1 -- 172.21.15.2:0/3075053165 learned_addr learned my addr 172.21.15.2:0/3075053165 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:54:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.600+0000 7fcc07fff700 1 --2- 172.21.15.2:0/3075053165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cde4720 0x557f3cdea580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.601+0000 7fcc0e59c700 1 -- 172.21.15.2:0/3075053165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cde4720 msgr2=0x557f3cdea580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.601+0000 7fcc0e59c700 1 --2- 172.21.15.2:0/3075053165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cde4720 0x557f3cdea580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.601+0000 7fcc0e59c700 1 -- 172.21.15.2:0/3075053165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cde4b90 msgr2=0x557f3cdde560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.601+0000 7fcc0e59c700 1 --2- 172.21.15.2:0/3075053165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cde4b90 0x557f3cdde560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.601+0000 7fcc0e59c700 1 -- 172.21.15.2:0/3075053165 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f3ccf5e80 con 0x557f3cc3a6b0 2023-11-16T16:54:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.601+0000 7fcc0ed9d700 1 --2- 172.21.15.2:0/3075053165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cde4b90 0x557f3cdde560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:54:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.601+0000 7fcc0e59c700 1 --2- 172.21.15.2:0/3075053165 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cc3a6b0 0x557f3cdef000 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fcc00009910 tx=0x7fcc0001a4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fea835d7b039663 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:14.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.601+0000 7fcc057fa700 1 -- 172.21.15.2:0/3075053165 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0002a070 con 0x557f3cc3a6b0 2023-11-16T16:54:14.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.601+0000 7fcc057fa700 1 -- 172.21.15.2:0/3075053165 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc000051e0 con 0x557f3cc3a6b0 2023-11-16T16:54:14.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.601+0000 7fcc057fa700 1 -- 172.21.15.2:0/3075053165 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc00015070 con 0x557f3cc3a6b0 2023-11-16T16:54:14.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.602+0000 7fcc2824db80 1 -- 172.21.15.2:0/3075053165 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cc3a6b0 msgr2=0x557f3cdef000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:14.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.602+0000 7fcc2824db80 1 --2- 172.21.15.2:0/3075053165 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cc3a6b0 0x557f3cdef000 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fcc00009910 tx=0x7fcc0001a4a0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.602+0000 7fcc2824db80 1 -- 172.21.15.2:0/3075053165 shutdown_connections 2023-11-16T16:54:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.602+0000 7fcc2824db80 1 --2- 172.21.15.2:0/3075053165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cde4b90 0x557f3cdde560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.602+0000 7fcc2824db80 1 --2- 172.21.15.2:0/3075053165 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cc3a6b0 0x557f3cdef000 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.602+0000 7fcc2824db80 1 --2- 172.21.15.2:0/3075053165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cde4720 0x557f3cdea580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.602+0000 7fcc2824db80 1 -- 172.21.15.2:0/3075053165 >> 172.21.15.2:0/3075053165 conn(0x557f3cdd3f50 msgr2=0x557f3cde1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.602+0000 7fcc2824db80 1 -- 172.21.15.2:0/3075053165 shutdown_connections 2023-11-16T16:54:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.602+0000 7fcc2824db80 1 -- 172.21.15.2:0/3075053165 wait complete. 2023-11-16T16:54:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.603+0000 7fcc2824db80 1 -- start start 2023-11-16T16:54:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.603+0000 7fcc2824db80 1 Processor -- start 2023-11-16T16:54:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.603+0000 7fcc2824db80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cc3a6b0 0x557f3cdef1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.603+0000 7fcc2824db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cde4720 0x557f3cddfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.603+0000 7fcc2824db80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cde4b90 0x557f3cde0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.603+0000 7fcc2824db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f3cde07e0 con 0x557f3cde4720 2023-11-16T16:54:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.603+0000 7fcc2824db80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f3cde0950 con 0x557f3cde4b90 2023-11-16T16:54:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.603+0000 7fcc2824db80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f3cde0b10 con 0x557f3cc3a6b0 2023-11-16T16:54:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.603+0000 7fcc2824db80 10 client.0 _ll_register_callbacks cb 0x7fcc08099740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.604+0000 7fcc0e59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cc3a6b0 0x557f3cdef1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.604+0000 7fcc0e59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cc3a6b0 0x557f3cdef1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:57758/0 (socket says 172.21.15.2:57758) 2023-11-16T16:54:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.604+0000 7fcc0e59c700 1 -- 172.21.15.2:0/2477887003 learned_addr learned my addr 172.21.15.2:0/2477887003 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:54:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.604+0000 7fcc0ed9d700 1 --2- 172.21.15.2:0/2477887003 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cde4b90 0x557f3cde0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.604+0000 7fcc07fff700 1 --2- 172.21.15.2:0/2477887003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cde4720 0x557f3cddfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.604+0000 7fcc0e59c700 1 -- 172.21.15.2:0/2477887003 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cde4b90 msgr2=0x557f3cde0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.604+0000 7fcc0e59c700 1 --2- 172.21.15.2:0/2477887003 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cde4b90 0x557f3cde0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.604+0000 7fcc0e59c700 1 -- 172.21.15.2:0/2477887003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cde4720 msgr2=0x557f3cddfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.604+0000 7fcc0e59c700 1 --2- 172.21.15.2:0/2477887003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cde4720 0x557f3cddfc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.604+0000 7fcc0e59c700 1 -- 172.21.15.2:0/2477887003 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557f3ccf6130 con 0x557f3cc3a6b0 2023-11-16T16:54:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.604+0000 7fcc0e59c700 1 --2- 172.21.15.2:0/2477887003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cc3a6b0 0x557f3cdef1e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcc000098e0 tx=0x7fcc00009f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=573919106e561056 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.604+0000 7fcbf5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:54:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.604+0000 7fcbf5ffb700 1 -- 172.21.15.2:0/2477887003 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc00015050 con 0x557f3cc3a6b0 2023-11-16T16:54:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.605+0000 7fcc2824db80 1 -- 172.21.15.2:0/2477887003 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc00016040 con 0x557f3cc3a6b0 2023-11-16T16:54:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.605+0000 7fcbf5ffb700 1 -- 172.21.15.2:0/2477887003 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fcc000056b0 con 0x557f3cc3a6b0 2023-11-16T16:54:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.605+0000 7fcbf5ffb700 1 -- 172.21.15.2:0/2477887003 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc0002a040 con 0x557f3cc3a6b0 2023-11-16T16:54:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.605+0000 7fcc2824db80 10 client.25765 fetch_fsmap learned FSMap version 35 2023-11-16T16:54:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.605+0000 7fcc2824db80 1 -- 172.21.15.2:0/2477887003 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x557f3cbb15d0 con 0x557f3cc3a6b0 2023-11-16T16:54:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.605+0000 7fcbf5ffb700 1 -- 172.21.15.2:0/2477887003 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc00016040 con 0x557f3cc3a6b0 2023-11-16T16:54:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.605+0000 7fcbf5ffb700 1 -- 172.21.15.2:0/2477887003 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fcc0003e070 con 0x557f3cc3a6b0 2023-11-16T16:54:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.605+0000 7fcc2824db80 10 client.25765 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:54:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.605+0000 7fcc2824db80 10 client.25765 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.605+0000 7fcc2824db80 20 client.25765 populate_metadata read hostname 'smithi002' 2023-11-16T16:54:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.605+0000 7fcc2824db80 1 --2- 172.21.15.2:0/2477887003 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x557f3ce4c160 0x557f3ce4e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.605+0000 7fcc2824db80 4 client.25765 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:54:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.605+0000 7fcc2824db80 1 -- 172.21.15.2:0/2477887003 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557f3ccfe600 con 0x557f3ce4c160 2023-11-16T16:54:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.606+0000 7fcc07fff700 1 --2- 172.21.15.2:0/2477887003 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x557f3ce4c160 0x557f3ce4e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.606+0000 7fcc07fff700 1 --2- 172.21.15.2:0/2477887003 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x557f3ce4c160 0x557f3ce4e570 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66720ac5f87c2090 server_cookie=3fdb75d7c2be5d3d in_seq=0 out_seq=0 2023-11-16T16:54:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:14.606+0000 7fcbf5ffb700 10 client.25765 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:54:15.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:15 smithi141 ceph-mon[131184]: pgmap v133: 129 pgs: 129 active+clean; 345 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.0 MiB/s wr, 546 op/s 2023-11-16T16:54:15.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:15 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:54:15.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:15 smithi155 ceph-mon[131607]: pgmap v133: 129 pgs: 129 active+clean; 345 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.0 MiB/s wr, 546 op/s 2023-11-16T16:54:15.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:15 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:54:15.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:15 smithi002 ceph-mon[128712]: pgmap v133: 129 pgs: 129 active+clean; 345 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.0 MiB/s wr, 546 op/s 2023-11-16T16:54:15.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:15 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:54:16.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.498+0000 7fcbf5ffb700 1 -- 172.21.15.2:0/2477887003 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557f3ccfe600 con 0x557f3ce4c160 2023-11-16T16:54:16.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.498+0000 7fcbf5ffb700 10 client.25765 handle_command_reply: tid=0 2023-11-16T16:54:16.501 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.499+0000 7fcc2824db80 1 client.25765 shutdown 2023-11-16T16:54:16.502 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.500+0000 7fcc2824db80 1 -- 172.21.15.2:0/2477887003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cc3a6b0 msgr2=0x557f3cdef1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:16.503 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.500+0000 7fcc2824db80 1 --2- 172.21.15.2:0/2477887003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cc3a6b0 0x557f3cdef1e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcc000098e0 tx=0x7fcc00009f00 comp rx=0 tx=0).stop 2023-11-16T16:54:16.504 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.502+0000 7fcc2824db80 1 -- 172.21.15.2:0/2477887003 shutdown_connections 2023-11-16T16:54:16.504 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.502+0000 7fcc2824db80 1 --2- 172.21.15.2:0/2477887003 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x557f3ce4c160 0x557f3ce4e570 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:16.504 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.502+0000 7fcc2824db80 1 --2- 172.21.15.2:0/2477887003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f3cc3a6b0 0x557f3cdef1e0 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcc000098e0 tx=0x7fcc00009f00 comp rx=0 tx=0).stop 2023-11-16T16:54:16.504 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.502+0000 7fcc2824db80 1 --2- 172.21.15.2:0/2477887003 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x557f3cde4b90 0x557f3cde0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:16.504 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.502+0000 7fcc2824db80 1 --2- 172.21.15.2:0/2477887003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557f3cde4720 0x557f3cddfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:16.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.502+0000 7fcc2824db80 1 -- 172.21.15.2:0/2477887003 >> 172.21.15.2:0/2477887003 conn(0x557f3cdd3f50 msgr2=0x557f3cde4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:16.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.503+0000 7fcbf5ffb700 0 client.25765 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:54:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.505+0000 7fcc2824db80 1 -- 172.21.15.2:0/2477887003 shutdown_connections 2023-11-16T16:54:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.505+0000 7fcc2824db80 1 -- 172.21.15.2:0/2477887003 wait complete. 2023-11-16T16:54:16.507 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.506+0000 7fcc2824db80 20 client.25765 trim_cache size 0 max 16384 2023-11-16T16:54:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:54:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:54:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:54:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:54:16.509 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.508+0000 7fcc149ff700 1 -- 172.21.15.2:0/3593614253 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fcbfc0a12c0 msgr2=0x7fcbfc0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.508+0000 7fcc149ff700 1 --2- 172.21.15.2:0/3593614253 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fcbfc0a12c0 0x7fcbfc0a3770 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8000c00 tx=0x7fcbf801a040 comp rx=0 tx=0).stop 2023-11-16T16:54:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.508+0000 7fcc149ff700 1 -- 172.21.15.2:0/3593614253 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc101055a0 msgr2=0x7fcc1019ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.508+0000 7fcc149ff700 1 --2- 172.21.15.2:0/3593614253 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc101055a0 0x7fcc1019ad80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800ef30 tx=0x7fcc0800c3a0 comp rx=0 tx=0).stop 2023-11-16T16:54:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.509+0000 7fcc149ff700 1 -- 172.21.15.2:0/3593614253 shutdown_connections 2023-11-16T16:54:16.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.509+0000 7fcc149ff700 1 --2- 172.21.15.2:0/3593614253 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fcbfc0a12c0 0x7fcbfc0a3770 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:16.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.509+0000 7fcc149ff700 1 --2- 172.21.15.2:0/3593614253 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc101055a0 0x7fcc1019ad80 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:16.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.509+0000 7fcc149ff700 1 --2- 172.21.15.2:0/3593614253 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fcc10100560 0x7fcc10196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:16.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.509+0000 7fcc149ff700 1 --2- 172.21.15.2:0/3593614253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc10100f00 0x7fcc10196970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:16.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.509+0000 7fcc149ff700 1 -- 172.21.15.2:0/3593614253 >> 172.21.15.2:0/3593614253 conn(0x7fcc100781b0 msgr2=0x7fcc10103ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:16.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.509+0000 7fcc149ff700 1 -- 172.21.15.2:0/3593614253 shutdown_connections 2023-11-16T16:54:16.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:16.509+0000 7fcc149ff700 1 -- 172.21.15.2:0/3593614253 wait complete. 2023-11-16T16:54:17.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:17 smithi141 ceph-mon[131184]: pgmap v134: 129 pgs: 129 active+clean; 345 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.0 MiB/s wr, 546 op/s 2023-11-16T16:54:17.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:17 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:54:17.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:17 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:54:17.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:17 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:54:17.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:17 smithi155 ceph-mon[131607]: pgmap v134: 129 pgs: 129 active+clean; 345 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.0 MiB/s wr, 546 op/s 2023-11-16T16:54:17.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:17 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:54:17.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:17 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:54:17.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:17 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:54:17.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:17 smithi002 ceph-mon[128712]: pgmap v134: 129 pgs: 129 active+clean; 345 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.0 MiB/s wr, 546 op/s 2023-11-16T16:54:17.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:17 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:54:17.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:17 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:54:17.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:17 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:54:18.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:18 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:54:18.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:18 smithi141 ceph-mon[131184]: Service mds.all has no related daemon already placed. 2023-11-16T16:54:18.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:18 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:54:18.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:18 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:54:18.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:18 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:54:18.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:18 smithi155 ceph-mon[131607]: Service mds.all has no related daemon already placed. 2023-11-16T16:54:18.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:18 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:54:18.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:18 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:54:18.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:18 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:54:18.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:18 smithi002 ceph-mon[128712]: Service mds.all has no related daemon already placed. 2023-11-16T16:54:18.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:18 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:54:18.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:18 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:54:19.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:19 smithi141 ceph-mon[131184]: pgmap v135: 129 pgs: 129 active+clean; 380 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 10 MiB/s wr, 882 op/s 2023-11-16T16:54:19.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:19 smithi155 ceph-mon[131607]: pgmap v135: 129 pgs: 129 active+clean; 380 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 10 MiB/s wr, 882 op/s 2023-11-16T16:54:19.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:19 smithi002 ceph-mon[128712]: pgmap v135: 129 pgs: 129 active+clean; 380 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 10 MiB/s wr, 882 op/s 2023-11-16T16:54:21.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:21 smithi141 ceph-mon[131184]: pgmap v136: 129 pgs: 129 active+clean; 380 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.0 MiB/s wr, 514 op/s 2023-11-16T16:54:21.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:21 smithi155 ceph-mon[131607]: pgmap v136: 129 pgs: 129 active+clean; 380 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.0 MiB/s wr, 514 op/s 2023-11-16T16:54:21.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:21 smithi002 ceph-mon[128712]: pgmap v136: 129 pgs: 129 active+clean; 380 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.0 MiB/s wr, 514 op/s 2023-11-16T16:54:23.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:23 smithi141 ceph-mon[131184]: pgmap v137: 129 pgs: 129 active+clean; 371 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 8.5 MiB/s wr, 807 op/s 2023-11-16T16:54:23.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:23 smithi155 ceph-mon[131607]: pgmap v137: 129 pgs: 129 active+clean; 371 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 8.5 MiB/s wr, 807 op/s 2023-11-16T16:54:23.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:23 smithi002 ceph-mon[128712]: pgmap v137: 129 pgs: 129 active+clean; 371 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 8.5 MiB/s wr, 807 op/s 2023-11-16T16:54:25.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:25 smithi141 ceph-mon[131184]: pgmap v138: 129 pgs: 129 active+clean; 371 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.6 MiB/s wr, 629 op/s 2023-11-16T16:54:25.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:25 smithi155 ceph-mon[131607]: pgmap v138: 129 pgs: 129 active+clean; 371 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.6 MiB/s wr, 629 op/s 2023-11-16T16:54:25.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:25 smithi002 ceph-mon[128712]: pgmap v138: 129 pgs: 129 active+clean; 371 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.6 MiB/s wr, 629 op/s 2023-11-16T16:54:27.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:27 smithi141 ceph-mon[131184]: pgmap v139: 129 pgs: 129 active+clean; 371 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.6 MiB/s wr, 629 op/s 2023-11-16T16:54:27.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:27 smithi155 ceph-mon[131607]: pgmap v139: 129 pgs: 129 active+clean; 371 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.6 MiB/s wr, 629 op/s 2023-11-16T16:54:27.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:27 smithi002 ceph-mon[128712]: pgmap v139: 129 pgs: 129 active+clean; 371 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.6 MiB/s wr, 629 op/s 2023-11-16T16:54:29.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:29 smithi141 ceph-mon[131184]: pgmap v140: 129 pgs: 129 active+clean; 391 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.3 MiB/s wr, 851 op/s 2023-11-16T16:54:29.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:29 smithi155 ceph-mon[131607]: pgmap v140: 129 pgs: 129 active+clean; 391 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.3 MiB/s wr, 851 op/s 2023-11-16T16:54:29.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:29 smithi002 ceph-mon[128712]: pgmap v140: 129 pgs: 129 active+clean; 391 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.3 MiB/s wr, 851 op/s 2023-11-16T16:54:29.700 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.698+0000 7f53b8fff700 1 -- 172.21.15.2:0/3795899302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53b4100560 msgr2=0x7f53b4100930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:29.700 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.698+0000 7f53b8fff700 1 --2- 172.21.15.2:0/3795899302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53b4100560 0x7f53b4100930 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009990 tx=0x7f53a4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.700 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.698+0000 7f53b8fff700 1 -- 172.21.15.2:0/3795899302 shutdown_connections 2023-11-16T16:54:29.700 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.698+0000 7f53b8fff700 1 --2- 172.21.15.2:0/3795899302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53b41055a0 0x7f53b410d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.700 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.698+0000 7f53b8fff700 1 --2- 172.21.15.2:0/3795899302 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53b4100560 0x7f53b4100930 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.700 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.698+0000 7f53b8fff700 1 --2- 172.21.15.2:0/3795899302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53b4100f00 0x7f53b4104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.700 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.698+0000 7f53b8fff700 1 -- 172.21.15.2:0/3795899302 >> 172.21.15.2:0/3795899302 conn(0x7f53b40781b0 msgr2=0x7f53b40785b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:29.701 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.699+0000 7f53b8fff700 1 -- 172.21.15.2:0/3795899302 shutdown_connections 2023-11-16T16:54:29.701 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.699+0000 7f53b8fff700 1 -- 172.21.15.2:0/3795899302 wait complete. 2023-11-16T16:54:29.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.700+0000 7f53b8fff700 1 Processor -- start 2023-11-16T16:54:29.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.700+0000 7f53b8fff700 1 -- start start 2023-11-16T16:54:29.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.700+0000 7f53b8fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53b4100560 0x7f53b4198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:29.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.700+0000 7f53b8fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53b4100f00 0x7f53b4198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:29.704 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.701+0000 7f53b8fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53b41055a0 0x7f53b419cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:29.704 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.701+0000 7f53b8fff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b419d560 con 0x7f53b4100560 2023-11-16T16:54:29.704 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.701+0000 7f53b8fff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b419d6d0 con 0x7f53b41055a0 2023-11-16T16:54:29.704 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.701+0000 7f53b8fff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b419d890 con 0x7f53b4100f00 2023-11-16T16:54:29.704 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.701+0000 7f53b1d9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53b4100f00 0x7f53b4198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:29.704 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.701+0000 7f53b259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53b4100560 0x7f53b4198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:29.704 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.701+0000 7f53b1d9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53b4100f00 0x7f53b4198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:52764/0 (socket says 172.21.15.2:52764) 2023-11-16T16:54:29.705 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.701+0000 7f53b1d9b700 1 -- 172.21.15.2:0/2637355996 learned_addr learned my addr 172.21.15.2:0/2637355996 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:54:29.705 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.701+0000 7f53b2d9d700 1 --2- 172.21.15.2:0/2637355996 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53b41055a0 0x7f53b419cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:29.705 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.701+0000 7f53b1d9b700 1 -- 172.21.15.2:0/2637355996 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53b41055a0 msgr2=0x7f53b419cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:29.705 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.701+0000 7f53b1d9b700 1 --2- 172.21.15.2:0/2637355996 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53b41055a0 0x7f53b419cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.705 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.701+0000 7f53b1d9b700 1 -- 172.21.15.2:0/2637355996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53b4100560 msgr2=0x7f53b4198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:29.705 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.701+0000 7f53b1d9b700 1 --2- 172.21.15.2:0/2637355996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53b4100560 0x7f53b4198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.705 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.701+0000 7f53b1d9b700 1 -- 172.21.15.2:0/2637355996 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a40095f0 con 0x7f53b4100f00 2023-11-16T16:54:29.705 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.702+0000 7f53b1d9b700 1 --2- 172.21.15.2:0/2637355996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53b4100f00 0x7f53b4198ac0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f53a800d950 tx=0x7f53a800dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.702+0000 7f53a37fe700 1 -- 172.21.15.2:0/2637355996 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a800f890 con 0x7f53b4100f00 2023-11-16T16:54:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.702+0000 7f53a37fe700 1 -- 172.21.15.2:0/2637355996 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53a8010460 con 0x7f53b4100f00 2023-11-16T16:54:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.702+0000 7f53b8fff700 1 -- 172.21.15.2:0/2637355996 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b4110400 con 0x7f53b4100f00 2023-11-16T16:54:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.702+0000 7f53a37fe700 1 -- 172.21.15.2:0/2637355996 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a800e610 con 0x7f53b4100f00 2023-11-16T16:54:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.702+0000 7f53b8fff700 1 -- 172.21.15.2:0/2637355996 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b4110270 con 0x7f53b4100f00 2023-11-16T16:54:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.706+0000 7f53a37fe700 1 -- 172.21.15.2:0/2637355996 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f53a800f9f0 con 0x7f53b4100f00 2023-11-16T16:54:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.709+0000 7f53a37fe700 1 --2- 172.21.15.2:0/2637355996 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f539c0a1300 0x7f539c0a37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.709+0000 7f53a37fe700 1 -- 172.21.15.2:0/2637355996 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f53a80ccb50 con 0x7f53b4100f00 2023-11-16T16:54:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.709+0000 7f53b259c700 1 --2- 172.21.15.2:0/2637355996 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f539c0a1300 0x7f539c0a37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:29.712 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.710+0000 7f53b259c700 1 --2- 172.21.15.2:0/2637355996 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f539c0a1300 0x7f539c0a37b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f53a400b5c0 tx=0x7f53a4005de0 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.710+0000 7f53cca1ab80 1 Processor -- start 2023-11-16T16:54:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.710+0000 7f53cca1ab80 1 -- start start 2023-11-16T16:54:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.711+0000 7f53cca1ab80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5646168526b0 0x564616a049b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.711+0000 7f53cca1ab80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x564616a04ef0 0x5646169f6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.711+0000 7f53cca1ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564616a05360 0x5646169f6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.711+0000 7f53cca1ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646168fc000 con 0x564616a05360 2023-11-16T16:54:29.713 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.711+0000 7f53cca1ab80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x56461694e840 con 0x564616a04ef0 2023-11-16T16:54:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.711+0000 7f53cca1ab80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x564616947cd0 con 0x5646168526b0 2023-11-16T16:54:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.711+0000 7f53b1d9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x564616a04ef0 0x5646169f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.711+0000 7f53b1d9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x564616a04ef0 0x5646169f6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:36948/0 (socket says 172.21.15.2:36948) 2023-11-16T16:54:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.711+0000 7f53b259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5646168526b0 0x564616a049b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.711+0000 7f53b1d9b700 1 -- 172.21.15.2:0/3846927828 learned_addr learned my addr 172.21.15.2:0/3846927828 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:54:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.711+0000 7f53b2d9d700 1 --2- 172.21.15.2:0/3846927828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564616a05360 0x5646169f6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.712+0000 7f53b1d9b700 1 -- 172.21.15.2:0/3846927828 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5646168526b0 msgr2=0x564616a049b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:29.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.712+0000 7f53b1d9b700 1 --2- 172.21.15.2:0/3846927828 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5646168526b0 0x564616a049b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.715 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.712+0000 7f53b1d9b700 1 -- 172.21.15.2:0/3846927828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564616a05360 msgr2=0x5646169f6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:29.715 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.712+0000 7f53b1d9b700 1 --2- 172.21.15.2:0/3846927828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564616a05360 0x5646169f6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.715 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.712+0000 7f53b1d9b700 1 -- 172.21.15.2:0/3846927828 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5646167c95d0 con 0x564616a04ef0 2023-11-16T16:54:29.715 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.712+0000 7f53b2d9d700 1 --2- 172.21.15.2:0/3846927828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564616a05360 0x5646169f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:54:29.715 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.712+0000 7f53b1d9b700 1 --2- 172.21.15.2:0/3846927828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x564616a04ef0 0x5646169f6560 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f53a8095d50 tx=0x7f53a80994c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afb4a7c53c150f5d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.712+0000 7f53a0ff9700 1 -- 172.21.15.2:0/3846927828 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a80a1ca0 con 0x564616a04ef0 2023-11-16T16:54:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.712+0000 7f53a0ff9700 1 -- 172.21.15.2:0/3846927828 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53a800fca0 con 0x564616a04ef0 2023-11-16T16:54:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.712+0000 7f53a0ff9700 1 -- 172.21.15.2:0/3846927828 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a8098d40 con 0x564616a04ef0 2023-11-16T16:54:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.713+0000 7f53cca1ab80 1 -- 172.21.15.2:0/3846927828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x564616a04ef0 msgr2=0x5646169f6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:29.720 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.713+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/3846927828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x564616a04ef0 0x5646169f6560 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f53a8095d50 tx=0x7f53a80994c0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.713+0000 7f53cca1ab80 1 -- 172.21.15.2:0/3846927828 shutdown_connections 2023-11-16T16:54:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.713+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/3846927828 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5646168526b0 0x564616a049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.713+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/3846927828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564616a05360 0x5646169f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.713+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/3846927828 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x564616a04ef0 0x5646169f6560 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.713+0000 7f53cca1ab80 1 -- 172.21.15.2:0/3846927828 >> 172.21.15.2:0/3846927828 conn(0x5646169ebf50 msgr2=0x5646169f1410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.713+0000 7f53cca1ab80 1 -- 172.21.15.2:0/3846927828 shutdown_connections 2023-11-16T16:54:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.714+0000 7f53cca1ab80 1 -- 172.21.15.2:0/3846927828 wait complete. 2023-11-16T16:54:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.714+0000 7f53cca1ab80 1 -- start start 2023-11-16T16:54:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.714+0000 7f53cca1ab80 1 Processor -- start 2023-11-16T16:54:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.715+0000 7f53cca1ab80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5646168526b0 0x564616a08550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.715+0000 7f53cca1ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564616a04ef0 0x564616a08a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.715+0000 7f53cca1ab80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x564616a05360 0x564616a08fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.716+0000 7f53cca1ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646168e9880 con 0x564616a04ef0 2023-11-16T16:54:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.716+0000 7f53cca1ab80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646168e9520 con 0x564616a05360 2023-11-16T16:54:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.716+0000 7f53cca1ab80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646167cb8c0 con 0x5646168526b0 2023-11-16T16:54:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.716+0000 7f53cca1ab80 10 client.0 _ll_register_callbacks cb 0x564616907720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.716+0000 7f53b259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5646168526b0 0x564616a08550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.716+0000 7f53b259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5646168526b0 0x564616a08550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:52778/0 (socket says 172.21.15.2:52778) 2023-11-16T16:54:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.716+0000 7f53b259c700 1 -- 172.21.15.2:0/1391015598 learned_addr learned my addr 172.21.15.2:0/1391015598 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:54:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.716+0000 7f53b1d9b700 1 --2- 172.21.15.2:0/1391015598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564616a04ef0 0x564616a08a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.716+0000 7f53b2d9d700 1 --2- 172.21.15.2:0/1391015598 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x564616a05360 0x564616a08fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.716+0000 7f53b259c700 1 -- 172.21.15.2:0/1391015598 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x564616a05360 msgr2=0x564616a08fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:29.724 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.716+0000 7f53b259c700 1 --2- 172.21.15.2:0/1391015598 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x564616a05360 0x564616a08fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.724 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.716+0000 7f53b259c700 1 -- 172.21.15.2:0/1391015598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564616a04ef0 msgr2=0x564616a08a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:29.724 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.716+0000 7f53b259c700 1 --2- 172.21.15.2:0/1391015598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564616a04ef0 0x564616a08a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.724 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.716+0000 7f53b259c700 1 -- 172.21.15.2:0/1391015598 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b4110270 con 0x5646168526b0 2023-11-16T16:54:29.724 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.716+0000 7f53b1d9b700 1 --2- 172.21.15.2:0/1391015598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564616a04ef0 0x564616a08a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:54:29.724 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.717+0000 7f53b259c700 1 --2- 172.21.15.2:0/1391015598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5646168526b0 0x564616a08550 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009990 tx=0x7f53a4009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d9858098317b5d9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:29.724 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.717+0000 7f5399ffb700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:54:29.724 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.717+0000 7f5399ffb700 1 -- 172.21.15.2:0/1391015598 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a4023670 con 0x5646168526b0 2023-11-16T16:54:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.717+0000 7f5399ffb700 1 -- 172.21.15.2:0/1391015598 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f53a40237d0 con 0x5646168526b0 2023-11-16T16:54:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.717+0000 7f53cca1ab80 1 -- 172.21.15.2:0/1391015598 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5646169e9710 con 0x5646168526b0 2023-11-16T16:54:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.718+0000 7f5399ffb700 1 -- 172.21.15.2:0/1391015598 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53a4023990 con 0x5646168526b0 2023-11-16T16:54:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.718+0000 7f53cca1ab80 10 client.25777 fetch_fsmap learned FSMap version 35 2023-11-16T16:54:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.718+0000 7f53cca1ab80 1 -- 172.21.15.2:0/1391015598 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x56461690e130 con 0x5646168526b0 2023-11-16T16:54:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.718+0000 7f5399ffb700 1 -- 172.21.15.2:0/1391015598 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5646169e9710 con 0x5646168526b0 2023-11-16T16:54:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.718+0000 7f5399ffb700 1 -- 172.21.15.2:0/1391015598 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f53a402e070 con 0x5646168526b0 2023-11-16T16:54:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.718+0000 7f53cca1ab80 10 client.25777 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:54:29.725 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.718+0000 7f53cca1ab80 10 client.25777 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:29.726 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.718+0000 7f53cca1ab80 20 client.25777 populate_metadata read hostname 'smithi002' 2023-11-16T16:54:29.726 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.718+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/1391015598 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x564616a188b0 0x564616a1acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:29.726 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.719+0000 7f53cca1ab80 4 client.25777 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:54:29.726 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.719+0000 7f53cca1ab80 1 -- 172.21.15.2:0/1391015598 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56461690a040 con 0x564616a188b0 2023-11-16T16:54:29.726 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.719+0000 7f53b1d9b700 1 --2- 172.21.15.2:0/1391015598 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x564616a188b0 0x564616a1acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:29.726 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.719+0000 7f53b1d9b700 1 --2- 172.21.15.2:0/1391015598 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x564616a188b0 0x564616a1acc0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=264fe4c3f5291741 server_cookie=e293eb4ab02cdd95 in_seq=0 out_seq=0 2023-11-16T16:54:29.726 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.719+0000 7f5399ffb700 10 client.25777 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:54:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.721+0000 7f5399ffb700 1 -- 172.21.15.2:0/1391015598 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56461690a040 con 0x564616a188b0 2023-11-16T16:54:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.721+0000 7f5399ffb700 10 client.25777 handle_command_reply: tid=0 2023-11-16T16:54:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.721+0000 7f53cca1ab80 1 client.25777 shutdown 2023-11-16T16:54:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.722+0000 7f53cca1ab80 1 -- 172.21.15.2:0/1391015598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5646168526b0 msgr2=0x564616a08550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.722+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/1391015598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5646168526b0 0x564616a08550 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009990 tx=0x7f53a4009ca0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.722+0000 7f53cca1ab80 1 -- 172.21.15.2:0/1391015598 shutdown_connections 2023-11-16T16:54:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.722+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/1391015598 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x564616a188b0 0x564616a1acc0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.722+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/1391015598 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5646168526b0 0x564616a08550 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.722+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/1391015598 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x564616a05360 0x564616a08fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.722+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/1391015598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564616a04ef0 0x564616a08a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.722+0000 7f53cca1ab80 1 -- 172.21.15.2:0/1391015598 >> 172.21.15.2:0/1391015598 conn(0x5646169ebf50 msgr2=0x5646169f07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.722+0000 7f5399ffb700 0 client.25777 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:54:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.722+0000 7f53cca1ab80 1 -- 172.21.15.2:0/1391015598 shutdown_connections 2023-11-16T16:54:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.722+0000 7f53cca1ab80 1 -- 172.21.15.2:0/1391015598 wait complete. 2023-11-16T16:54:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.722+0000 7f53cca1ab80 20 client.25777 trim_cache size 0 max 16384 2023-11-16T16:54:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.744+0000 7f53cca1ab80 1 Processor -- start 2023-11-16T16:54:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.744+0000 7f53cca1ab80 1 -- start start 2023-11-16T16:54:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.745+0000 7f53cca1ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5646168526b0 0x564616a0f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:29.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.745+0000 7f53cca1ab80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x564616a04ef0 0x564616a13db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.745+0000 7f53cca1ab80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564616a05360 0x564616a142f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.745+0000 7f53cca1ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646169f7330 con 0x5646168526b0 2023-11-16T16:54:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.745+0000 7f53cca1ab80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646169f2340 con 0x564616a04ef0 2023-11-16T16:54:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.745+0000 7f53cca1ab80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x564616a033a0 con 0x564616a05360 2023-11-16T16:54:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.745+0000 7f53b2d9d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564616a05360 0x564616a142f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.745+0000 7f53b2d9d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564616a05360 0x564616a142f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:52784/0 (socket says 172.21.15.2:52784) 2023-11-16T16:54:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.745+0000 7f53b2d9d700 1 -- 172.21.15.2:0/1179083627 learned_addr learned my addr 172.21.15.2:0/1179083627 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:54:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.745+0000 7f53b259c700 1 --2- 172.21.15.2:0/1179083627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5646168526b0 0x564616a0f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.746+0000 7f53b2d9d700 1 -- 172.21.15.2:0/1179083627 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x564616a04ef0 msgr2=0x564616a13db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T16:54:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.746+0000 7f53b2d9d700 1 --2- 172.21.15.2:0/1179083627 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x564616a04ef0 0x564616a13db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.746+0000 7f53b2d9d700 1 -- 172.21.15.2:0/1179083627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5646168526b0 msgr2=0x564616a0f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.746+0000 7f53b2d9d700 1 --2- 172.21.15.2:0/1179083627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5646168526b0 0x564616a0f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.746+0000 7f53b2d9d700 1 -- 172.21.15.2:0/1179083627 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56461690de80 con 0x564616a05360 2023-11-16T16:54:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.746+0000 7f53b2d9d700 1 --2- 172.21.15.2:0/1179083627 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564616a05360 0x564616a142f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f53ac0135f0 tx=0x7f53ac007950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9516d987ca226fc server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.746+0000 7f53a17fa700 1 -- 172.21.15.2:0/1179083627 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac003a00 con 0x564616a05360 2023-11-16T16:54:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.746+0000 7f53a17fa700 1 -- 172.21.15.2:0/1179083627 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53ac003b60 con 0x564616a05360 2023-11-16T16:54:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.746+0000 7f53a17fa700 1 -- 172.21.15.2:0/1179083627 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac004130 con 0x564616a05360 2023-11-16T16:54:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.747+0000 7f53cca1ab80 1 -- 172.21.15.2:0/1179083627 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564616a05360 msgr2=0x564616a142f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.747+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/1179083627 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564616a05360 0x564616a142f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f53ac0135f0 tx=0x7f53ac007950 comp rx=0 tx=0).stop 2023-11-16T16:54:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.747+0000 7f53cca1ab80 1 -- 172.21.15.2:0/1179083627 shutdown_connections 2023-11-16T16:54:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.747+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/1179083627 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564616a05360 0x564616a142f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.747+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/1179083627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5646168526b0 0x564616a0f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.747+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/1179083627 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x564616a04ef0 0x564616a13db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.747+0000 7f53cca1ab80 1 -- 172.21.15.2:0/1179083627 >> 172.21.15.2:0/1179083627 conn(0x5646169ebf50 msgr2=0x564616a02830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.748+0000 7f53cca1ab80 1 -- 172.21.15.2:0/1179083627 shutdown_connections 2023-11-16T16:54:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.748+0000 7f53cca1ab80 1 -- 172.21.15.2:0/1179083627 wait complete. 2023-11-16T16:54:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.748+0000 7f53cca1ab80 1 -- start start 2023-11-16T16:54:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.748+0000 7f53cca1ab80 1 Processor -- start 2023-11-16T16:54:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.749+0000 7f53cca1ab80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646168526b0 0x564616a65110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.749+0000 7f53cca1ab80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564616a04ef0 0x564616a65650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.749+0000 7f53cca1ab80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564616a05360 0x564616a0ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.749+0000 7f53cca1ab80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x564616a0f170 con 0x564616a05360 2023-11-16T16:54:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.749+0000 7f53cca1ab80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x564616a0f2e0 con 0x5646168526b0 2023-11-16T16:54:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.749+0000 7f53cca1ab80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x564616a0f4a0 con 0x564616a04ef0 2023-11-16T16:54:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.749+0000 7f53cca1ab80 10 client.0 _ll_register_callbacks cb 0x7f53a4003bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.749+0000 7f53b1d9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564616a04ef0 0x564616a65650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.749+0000 7f53b1d9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564616a04ef0 0x564616a65650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:52798/0 (socket says 172.21.15.2:52798) 2023-11-16T16:54:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.749+0000 7f53b1d9b700 1 -- 172.21.15.2:0/659359525 learned_addr learned my addr 172.21.15.2:0/659359525 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:54:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.750+0000 7f53b2d9d700 1 --2- 172.21.15.2:0/659359525 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564616a05360 0x564616a0ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.750+0000 7f53b259c700 1 --2- 172.21.15.2:0/659359525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646168526b0 0x564616a65110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:29.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.750+0000 7f53b1d9b700 1 -- 172.21.15.2:0/659359525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646168526b0 msgr2=0x564616a65110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:29.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.750+0000 7f53b1d9b700 1 --2- 172.21.15.2:0/659359525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646168526b0 0x564616a65110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.750+0000 7f53b1d9b700 1 -- 172.21.15.2:0/659359525 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564616a05360 msgr2=0x564616a0ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:29.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.750+0000 7f53b1d9b700 1 --2- 172.21.15.2:0/659359525 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564616a05360 0x564616a0ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:29.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.750+0000 7f53b1d9b700 1 -- 172.21.15.2:0/659359525 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53ac0132d0 con 0x564616a04ef0 2023-11-16T16:54:29.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.750+0000 7f53b1d9b700 1 --2- 172.21.15.2:0/659359525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564616a04ef0 0x564616a65650 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f53a8012040 tx=0x7f53a800b510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a9e41928d4f4594 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:29.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.750+0000 7f5399ffb700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:54:29.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.750+0000 7f5399ffb700 1 -- 172.21.15.2:0/659359525 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a80b1070 con 0x564616a04ef0 2023-11-16T16:54:29.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.750+0000 7f5399ffb700 1 -- 172.21.15.2:0/659359525 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f53a80b79e0 con 0x564616a04ef0 2023-11-16T16:54:29.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.750+0000 7f53cca1ab80 1 -- 172.21.15.2:0/659359525 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f53a800fca0 con 0x564616a04ef0 2023-11-16T16:54:29.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.751+0000 7f5399ffb700 1 -- 172.21.15.2:0/659359525 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53a80b7ba0 con 0x564616a04ef0 2023-11-16T16:54:29.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.751+0000 7f53cca1ab80 10 client.25783 fetch_fsmap learned FSMap version 35 2023-11-16T16:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.751+0000 7f53cca1ab80 1 -- 172.21.15.2:0/659359525 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x56461675ec00 con 0x564616a04ef0 2023-11-16T16:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.751+0000 7f5399ffb700 1 -- 172.21.15.2:0/659359525 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53a800fca0 con 0x564616a04ef0 2023-11-16T16:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.752+0000 7f5399ffb700 1 -- 172.21.15.2:0/659359525 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f53a8099ab0 con 0x564616a04ef0 2023-11-16T16:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.752+0000 7f53cca1ab80 10 client.25783 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.752+0000 7f53cca1ab80 10 client.25783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.752+0000 7f53cca1ab80 20 client.25783 populate_metadata read hostname 'smithi002' 2023-11-16T16:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.752+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/659359525 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x564616a6cbc0 0x564616a6efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.752+0000 7f53cca1ab80 4 client.25783 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:54:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.752+0000 7f53cca1ab80 1 -- 172.21.15.2:0/659359525 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564616916600 con 0x564616a6cbc0 2023-11-16T16:54:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.752+0000 7f53b259c700 1 --2- 172.21.15.2:0/659359525 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x564616a6cbc0 0x564616a6efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.752+0000 7f53b259c700 1 --2- 172.21.15.2:0/659359525 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x564616a6cbc0 0x564616a6efd0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85db79d600cf643 server_cookie=92e048b9ba0e95f8 in_seq=0 out_seq=0 2023-11-16T16:54:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:29.752+0000 7f5399ffb700 10 client.25783 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:54:31.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.483+0000 7f5399ffb700 1 -- 172.21.15.2:0/659359525 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564616916600 con 0x564616a6cbc0 2023-11-16T16:54:31.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.483+0000 7f5399ffb700 10 client.25783 handle_command_reply: tid=0 2023-11-16T16:54:31.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.483+0000 7f53cca1ab80 1 client.25783 shutdown 2023-11-16T16:54:31.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.484+0000 7f53cca1ab80 1 -- 172.21.15.2:0/659359525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564616a04ef0 msgr2=0x564616a65650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:31.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.485+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/659359525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564616a04ef0 0x564616a65650 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f53a8012040 tx=0x7f53a800b510 comp rx=0 tx=0).stop 2023-11-16T16:54:31.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.485+0000 7f53cca1ab80 1 -- 172.21.15.2:0/659359525 shutdown_connections 2023-11-16T16:54:31.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.485+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/659359525 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564616a05360 0x564616a0ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:31.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.485+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/659359525 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5646168526b0 0x564616a65110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:31.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.485+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/659359525 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x564616a6cbc0 0x564616a6efd0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:31.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.485+0000 7f53cca1ab80 1 --2- 172.21.15.2:0/659359525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564616a04ef0 0x564616a65650 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:31.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.485+0000 7f53cca1ab80 1 -- 172.21.15.2:0/659359525 >> 172.21.15.2:0/659359525 conn(0x5646169ebf50 msgr2=0x564616a02830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:31.486 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.485+0000 7f5399ffb700 0 client.25783 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:54:31.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.485+0000 7f53cca1ab80 1 -- 172.21.15.2:0/659359525 shutdown_connections 2023-11-16T16:54:31.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.486+0000 7f53cca1ab80 1 -- 172.21.15.2:0/659359525 wait complete. 2023-11-16T16:54:31.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.488+0000 7f53cca1ab80 20 client.25783 trim_cache size 0 max 16384 2023-11-16T16:54:31.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:54:31.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:54:31.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:54:31.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:54:31.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.489+0000 7f53b8fff700 1 -- 172.21.15.2:0/2637355996 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f539c0a1300 msgr2=0x7f539c0a37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:31.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.489+0000 7f53b8fff700 1 --2- 172.21.15.2:0/2637355996 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f539c0a1300 0x7f539c0a37b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f53a400b5c0 tx=0x7f53a4005de0 comp rx=0 tx=0).stop 2023-11-16T16:54:31.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.489+0000 7f53b8fff700 1 -- 172.21.15.2:0/2637355996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53b4100f00 msgr2=0x7f53b4198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:31.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.489+0000 7f53b8fff700 1 --2- 172.21.15.2:0/2637355996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53b4100f00 0x7f53b4198ac0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f53a800d950 tx=0x7f53a800dc60 comp rx=0 tx=0).stop 2023-11-16T16:54:31.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.491+0000 7f53b8fff700 1 -- 172.21.15.2:0/2637355996 shutdown_connections 2023-11-16T16:54:31.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.491+0000 7f53b8fff700 1 --2- 172.21.15.2:0/2637355996 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f539c0a1300 0x7f539c0a37b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:31.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.491+0000 7f53b8fff700 1 --2- 172.21.15.2:0/2637355996 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f53b41055a0 0x7f53b419cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:31.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.491+0000 7f53b8fff700 1 --2- 172.21.15.2:0/2637355996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53b4100560 0x7f53b4198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:31.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.491+0000 7f53b8fff700 1 --2- 172.21.15.2:0/2637355996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53b4100f00 0x7f53b4198ac0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:31.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.491+0000 7f53b8fff700 1 -- 172.21.15.2:0/2637355996 >> 172.21.15.2:0/2637355996 conn(0x7f53b40781b0 msgr2=0x7f53b40fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:31.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.492+0000 7f53b8fff700 1 -- 172.21.15.2:0/2637355996 shutdown_connections 2023-11-16T16:54:31.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:31.492+0000 7f53b8fff700 1 -- 172.21.15.2:0/2637355996 wait complete. 2023-11-16T16:54:31.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:31 smithi141 ceph-mon[131184]: pgmap v141: 129 pgs: 129 active+clean; 391 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 515 op/s 2023-11-16T16:54:31.619 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:31 smithi155 ceph-mon[131607]: pgmap v141: 129 pgs: 129 active+clean; 391 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 515 op/s 2023-11-16T16:54:31.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:31 smithi002 ceph-mon[128712]: pgmap v141: 129 pgs: 129 active+clean; 391 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 515 op/s 2023-11-16T16:54:33.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:33 smithi141 ceph-mon[131184]: pgmap v142: 129 pgs: 129 active+clean; 416 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.5 MiB/s wr, 948 op/s 2023-11-16T16:54:33.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:33 smithi155 ceph-mon[131607]: pgmap v142: 129 pgs: 129 active+clean; 416 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.5 MiB/s wr, 948 op/s 2023-11-16T16:54:33.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:33 smithi002 ceph-mon[128712]: pgmap v142: 129 pgs: 129 active+clean; 416 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.5 MiB/s wr, 948 op/s 2023-11-16T16:54:35.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:35 smithi141 ceph-mon[131184]: pgmap v143: 129 pgs: 129 active+clean; 416 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.0 MiB/s wr, 655 op/s 2023-11-16T16:54:35.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:35 smithi155 ceph-mon[131607]: pgmap v143: 129 pgs: 129 active+clean; 416 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.0 MiB/s wr, 655 op/s 2023-11-16T16:54:35.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:35 smithi002 ceph-mon[128712]: pgmap v143: 129 pgs: 129 active+clean; 416 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.0 MiB/s wr, 655 op/s 2023-11-16T16:54:37.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:37 smithi141 ceph-mon[131184]: pgmap v144: 129 pgs: 129 active+clean; 416 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.0 MiB/s wr, 655 op/s 2023-11-16T16:54:37.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:37 smithi155 ceph-mon[131607]: pgmap v144: 129 pgs: 129 active+clean; 416 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.0 MiB/s wr, 655 op/s 2023-11-16T16:54:37.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:37 smithi002 ceph-mon[128712]: pgmap v144: 129 pgs: 129 active+clean; 416 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.0 MiB/s wr, 655 op/s 2023-11-16T16:54:39.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:39 smithi141 ceph-mon[131184]: pgmap v145: 129 pgs: 129 active+clean; 416 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.0 MiB/s wr, 801 op/s 2023-11-16T16:54:39.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:39 smithi155 ceph-mon[131607]: pgmap v145: 129 pgs: 129 active+clean; 416 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.0 MiB/s wr, 801 op/s 2023-11-16T16:54:39.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:39 smithi002 ceph-mon[128712]: pgmap v145: 129 pgs: 129 active+clean; 416 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.0 MiB/s wr, 801 op/s 2023-11-16T16:54:40.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:40 smithi155 ceph-mon[131607]: pgmap v146: 129 pgs: 129 active+clean; 416 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.3 MiB/s wr, 578 op/s 2023-11-16T16:54:40.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:40 smithi002 ceph-mon[128712]: pgmap v146: 129 pgs: 129 active+clean; 416 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.3 MiB/s wr, 578 op/s 2023-11-16T16:54:41.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:40 smithi141 ceph-mon[131184]: pgmap v146: 129 pgs: 129 active+clean; 416 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.3 MiB/s wr, 578 op/s 2023-11-16T16:54:42.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:42 smithi155 ceph-mon[131607]: pgmap v147: 129 pgs: 129 active+clean; 439 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.2 MiB/s wr, 802 op/s 2023-11-16T16:54:42.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:42 smithi002 ceph-mon[128712]: pgmap v147: 129 pgs: 129 active+clean; 439 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.2 MiB/s wr, 802 op/s 2023-11-16T16:54:43.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:42 smithi141 ceph-mon[131184]: pgmap v147: 129 pgs: 129 active+clean; 439 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.2 MiB/s wr, 802 op/s 2023-11-16T16:54:44.744 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.742+0000 7ff1283ff700 1 -- 172.21.15.2:0/3807750444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff120101d50 msgr2=0x7ff12010a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:44.744 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.742+0000 7ff1283ff700 1 --2- 172.21.15.2:0/3807750444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff120101d50 0x7ff12010a240 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff1100098f0 tx=0x7ff110009c00 comp rx=0 tx=0).stop 2023-11-16T16:54:44.744 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.742+0000 7ff1283ff700 1 -- 172.21.15.2:0/3807750444 shutdown_connections 2023-11-16T16:54:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.742+0000 7ff1283ff700 1 --2- 172.21.15.2:0/3807750444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff12010a810 0x7ff12010cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.742+0000 7ff1283ff700 1 --2- 172.21.15.2:0/3807750444 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff120101440 0x7ff120101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.742+0000 7ff1283ff700 1 --2- 172.21.15.2:0/3807750444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff120101d50 0x7ff12010a240 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.742+0000 7ff1283ff700 1 -- 172.21.15.2:0/3807750444 >> 172.21.15.2:0/3807750444 conn(0x7ff1200763a0 msgr2=0x7ff1200767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.743+0000 7ff1283ff700 1 -- 172.21.15.2:0/3807750444 shutdown_connections 2023-11-16T16:54:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.743+0000 7ff1283ff700 1 -- 172.21.15.2:0/3807750444 wait complete. 2023-11-16T16:54:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.744+0000 7ff1283ff700 1 Processor -- start 2023-11-16T16:54:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.744+0000 7ff1283ff700 1 -- start start 2023-11-16T16:54:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.744+0000 7ff1283ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff120101440 0x7ff1201985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.744+0000 7ff1283ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff120101d50 0x7ff120198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.744+0000 7ff1283ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff12010a810 0x7ff12019cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.744+0000 7ff1283ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12019d5a0 con 0x7ff120101d50 2023-11-16T16:54:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.744+0000 7ff1283ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12019d710 con 0x7ff120101440 2023-11-16T16:54:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.744+0000 7ff1283ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12019d8d0 con 0x7ff12010a810 2023-11-16T16:54:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.744+0000 7ff12619b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff120101440 0x7ff1201985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.744+0000 7ff12699c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff12010a810 0x7ff12019cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.744+0000 7ff12619b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff120101440 0x7ff1201985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:53050/0 (socket says 172.21.15.2:53050) 2023-11-16T16:54:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.744+0000 7ff12619b700 1 -- 172.21.15.2:0/1746617612 learned_addr learned my addr 172.21.15.2:0/1746617612 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:54:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.744+0000 7ff12599a700 1 --2- 172.21.15.2:0/1746617612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff120101d50 0x7ff120198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.745+0000 7ff12699c700 1 -- 172.21.15.2:0/1746617612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff120101440 msgr2=0x7ff1201985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.745+0000 7ff12699c700 1 --2- 172.21.15.2:0/1746617612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff120101440 0x7ff1201985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.745+0000 7ff12699c700 1 -- 172.21.15.2:0/1746617612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff120101d50 msgr2=0x7ff120198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.745+0000 7ff12699c700 1 --2- 172.21.15.2:0/1746617612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff120101d50 0x7ff120198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.745+0000 7ff12699c700 1 -- 172.21.15.2:0/1746617612 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff110009520 con 0x7ff12010a810 2023-11-16T16:54:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.745+0000 7ff12619b700 1 --2- 172.21.15.2:0/1746617612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff120101440 0x7ff1201985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:54:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.745+0000 7ff12699c700 1 --2- 172.21.15.2:0/1746617612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff12010a810 0x7ff12019cf60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff11800efc0 tx=0x7ff11800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.745+0000 7ff1177fe700 1 -- 172.21.15.2:0/1746617612 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff118018070 con 0x7ff12010a810 2023-11-16T16:54:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.746+0000 7ff1177fe700 1 -- 172.21.15.2:0/1746617612 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff11800f040 con 0x7ff12010a810 2023-11-16T16:54:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.746+0000 7ff1283ff700 1 -- 172.21.15.2:0/1746617612 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1201102d0 con 0x7ff12010a810 2023-11-16T16:54:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.746+0000 7ff1177fe700 1 -- 172.21.15.2:0/1746617612 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1180077f0 con 0x7ff12010a810 2023-11-16T16:54:44.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.746+0000 7ff1283ff700 1 -- 172.21.15.2:0/1746617612 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff120110140 con 0x7ff12010a810 2023-11-16T16:54:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.750+0000 7ff1177fe700 1 -- 172.21.15.2:0/1746617612 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7ff118007990 con 0x7ff12010a810 2023-11-16T16:54:44.753 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.752+0000 7ff13bca5b80 1 Processor -- start 2023-11-16T16:54:44.753 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.752+0000 7ff13bca5b80 1 -- start start 2023-11-16T16:54:44.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.752+0000 7ff13bca5b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbc7c6b0 0x5616cbe261e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.752+0000 7ff13bca5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbe26720 0x5616cbe20560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.752+0000 7ff13bca5b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe26b90 0x5616cbe20b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.752+0000 7ff13bca5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616cbd26000 con 0x5616cbe26720 2023-11-16T16:54:44.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.752+0000 7ff13bca5b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616cbd78840 con 0x5616cbe26b90 2023-11-16T16:54:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.752+0000 7ff13bca5b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616cbd71cd0 con 0x5616cbc7c6b0 2023-11-16T16:54:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.753+0000 7ff12599a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbc7c6b0 0x5616cbe261e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.753+0000 7ff12599a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbc7c6b0 0x5616cbe261e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:37736/0 (socket says 172.21.15.2:37736) 2023-11-16T16:54:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.753+0000 7ff12599a700 1 -- 172.21.15.2:0/1530816796 learned_addr learned my addr 172.21.15.2:0/1530816796 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:54:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.753+0000 7ff12599a700 1 --2- 172.21.15.2:0/1530816796 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe26b90 0x5616cbe20b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.753+0000 7ff1177fe700 1 --2- 172.21.15.2:0/1746617612 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ff10c0a1340 0x7ff10c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.753+0000 7ff1177fe700 1 -- 172.21.15.2:0/1746617612 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ff1180cc910 con 0x7ff12010a810 2023-11-16T16:54:44.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.753+0000 7ff12619b700 1 --2- 172.21.15.2:0/1530816796 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbe26720 0x5616cbe20560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.753+0000 7ff12599a700 1 -- 172.21.15.2:0/1530816796 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe26b90 msgr2=0x5616cbe20b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:44.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.753+0000 7ff12599a700 1 --2- 172.21.15.2:0/1530816796 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe26b90 0x5616cbe20b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.753+0000 7ff12699c700 1 --2- 172.21.15.2:0/1746617612 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ff10c0a1340 0x7ff10c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.753+0000 7ff12599a700 1 -- 172.21.15.2:0/1530816796 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbe26720 msgr2=0x5616cbe20560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:44.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.753+0000 7ff12599a700 1 --2- 172.21.15.2:0/1530816796 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbe26720 0x5616cbe20560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.753+0000 7ff12599a700 1 -- 172.21.15.2:0/1530816796 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616cbbf35d0 con 0x5616cbc7c6b0 2023-11-16T16:54:44.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.754+0000 7ff12619b700 1 --2- 172.21.15.2:0/1530816796 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbe26720 0x5616cbe20560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:54:44.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.754+0000 7ff12599a700 1 --2- 172.21.15.2:0/1530816796 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbc7c6b0 0x5616cbe261e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff1100098f0 tx=0x7ff110019040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaf129c55219a512 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:44.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.754+0000 7ff12699c700 1 --2- 172.21.15.2:0/1746617612 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ff10c0a1340 0x7ff10c0a37f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff118096080 tx=0x7ff118004c50 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:44.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.754+0000 7ff114bfe700 1 -- 172.21.15.2:0/1530816796 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11002bb30 con 0x5616cbc7c6b0 2023-11-16T16:54:44.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.754+0000 7ff114bfe700 1 -- 172.21.15.2:0/1530816796 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff11000f920 con 0x5616cbc7c6b0 2023-11-16T16:54:44.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.754+0000 7ff114bfe700 1 -- 172.21.15.2:0/1530816796 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1100056a0 con 0x5616cbc7c6b0 2023-11-16T16:54:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.755+0000 7ff13bca5b80 1 -- 172.21.15.2:0/1530816796 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbc7c6b0 msgr2=0x5616cbe261e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.755+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/1530816796 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbc7c6b0 0x5616cbe261e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff1100098f0 tx=0x7ff110019040 comp rx=0 tx=0).stop 2023-11-16T16:54:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.755+0000 7ff13bca5b80 1 -- 172.21.15.2:0/1530816796 shutdown_connections 2023-11-16T16:54:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.755+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/1530816796 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbc7c6b0 0x5616cbe261e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.755+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/1530816796 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe26b90 0x5616cbe20b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.755+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/1530816796 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbe26720 0x5616cbe20560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.755+0000 7ff13bca5b80 1 -- 172.21.15.2:0/1530816796 >> 172.21.15.2:0/1530816796 conn(0x5616cbe15f50 msgr2=0x5616cbe1b270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.755+0000 7ff13bca5b80 1 -- 172.21.15.2:0/1530816796 shutdown_connections 2023-11-16T16:54:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.756+0000 7ff13bca5b80 1 -- 172.21.15.2:0/1530816796 wait complete. 2023-11-16T16:54:44.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.756+0000 7ff13bca5b80 1 -- start start 2023-11-16T16:54:44.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.756+0000 7ff13bca5b80 1 Processor -- start 2023-11-16T16:54:44.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.757+0000 7ff13bca5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbc7c6b0 0x5616cbe2cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.757+0000 7ff13bca5b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe26720 0x5616cbe29da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.757+0000 7ff13bca5b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe26b90 0x5616cbe2a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.757+0000 7ff13bca5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616cbd13880 con 0x5616cbc7c6b0 2023-11-16T16:54:44.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.757+0000 7ff13bca5b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616cbd13520 con 0x5616cbe26720 2023-11-16T16:54:44.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.757+0000 7ff13bca5b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616cbbf58c0 con 0x5616cbe26b90 2023-11-16T16:54:44.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.757+0000 7ff13bca5b80 10 client.0 _ll_register_callbacks cb 0x5616cbd31720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:44.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.757+0000 7ff12619b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe26720 0x5616cbe29da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.758+0000 7ff12619b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe26720 0x5616cbe29da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:53076/0 (socket says 172.21.15.2:53076) 2023-11-16T16:54:44.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.758+0000 7ff12619b700 1 -- 172.21.15.2:0/2354356099 learned_addr learned my addr 172.21.15.2:0/2354356099 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:54:44.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.758+0000 7ff12599a700 1 --2- 172.21.15.2:0/2354356099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe26b90 0x5616cbe2a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.760 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.758+0000 7ff12599a700 1 --2- 172.21.15.2:0/2354356099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbc7c6b0 0x5616cbe2cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.758+0000 7ff12619b700 1 -- 172.21.15.2:0/2354356099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe26b90 msgr2=0x5616cbe2a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:44.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.758+0000 7ff12619b700 1 --2- 172.21.15.2:0/2354356099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe26b90 0x5616cbe2a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.758+0000 7ff12619b700 1 -- 172.21.15.2:0/2354356099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbc7c6b0 msgr2=0x5616cbe2cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:44.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.758+0000 7ff12619b700 1 --2- 172.21.15.2:0/2354356099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbc7c6b0 0x5616cbe2cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.758+0000 7ff12619b700 1 -- 172.21.15.2:0/2354356099 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff11001f040 con 0x5616cbe26720 2023-11-16T16:54:44.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.758+0000 7ff12599a700 1 --2- 172.21.15.2:0/2354356099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe26b90 0x5616cbe2a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:54:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.758+0000 7ff12619b700 1 --2- 172.21.15.2:0/2354356099 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe26720 0x5616cbe29da0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff11c013d30 tx=0x7ff11c0116c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=719172d2c1cee6bb server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.758+0000 7ff109ffb700 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2023-11-16T16:54:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.759+0000 7ff109ffb700 1 -- 172.21.15.2:0/2354356099 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11c011ee0 con 0x5616cbe26720 2023-11-16T16:54:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.759+0000 7ff109ffb700 1 -- 172.21.15.2:0/2354356099 <== mon.2 v2:172.21.15.141:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ff11c004220 con 0x5616cbe26720 2023-11-16T16:54:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.759+0000 7ff13bca5b80 1 -- 172.21.15.2:0/2354356099 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5616cbe13710 con 0x5616cbe26720 2023-11-16T16:54:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.759+0000 7ff109ffb700 1 -- 172.21.15.2:0/2354356099 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff11c0043e0 con 0x5616cbe26720 2023-11-16T16:54:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.759+0000 7ff13bca5b80 10 client.25817 fetch_fsmap learned FSMap version 35 2023-11-16T16:54:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.759+0000 7ff13bca5b80 1 -- 172.21.15.2:0/2354356099 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x5616cbd38130 con 0x5616cbe26720 2023-11-16T16:54:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.760+0000 7ff109ffb700 1 -- 172.21.15.2:0/2354356099 <== mon.2 v2:172.21.15.141:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5616cbe13710 con 0x5616cbe26720 2023-11-16T16:54:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.760+0000 7ff109ffb700 1 -- 172.21.15.2:0/2354356099 <== mon.2 v2:172.21.15.141:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7ff11c01b690 con 0x5616cbe26720 2023-11-16T16:54:44.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.760+0000 7ff13bca5b80 10 client.25817 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:54:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.760+0000 7ff13bca5b80 10 client.25817 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.760+0000 7ff13bca5b80 20 client.25817 populate_metadata read hostname 'smithi002' 2023-11-16T16:54:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.760+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/2354356099 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5616cbe4b0d0 0x5616cbe4d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.760+0000 7ff13bca5b80 4 client.25817 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:54:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.760+0000 7ff13bca5b80 1 -- 172.21.15.2:0/2354356099 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5616cbd34040 con 0x5616cbe4b0d0 2023-11-16T16:54:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.760+0000 7ff12599a700 1 --2- 172.21.15.2:0/2354356099 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5616cbe4b0d0 0x5616cbe4d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.761+0000 7ff12599a700 1 --2- 172.21.15.2:0/2354356099 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5616cbe4b0d0 0x5616cbe4d4e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fc7fc0aa5c8eed8 server_cookie=6f6e742bbb61919e in_seq=0 out_seq=0 2023-11-16T16:54:44.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.761+0000 7ff109ffb700 10 client.25817 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:54:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.762+0000 7ff109ffb700 1 -- 172.21.15.2:0/2354356099 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5616cbd34040 con 0x5616cbe4b0d0 2023-11-16T16:54:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.762+0000 7ff109ffb700 10 client.25817 handle_command_reply: tid=0 2023-11-16T16:54:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.762+0000 7ff13bca5b80 1 client.25817 shutdown 2023-11-16T16:54:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.762+0000 7ff13bca5b80 1 -- 172.21.15.2:0/2354356099 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe26720 msgr2=0x5616cbe29da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.762+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/2354356099 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe26720 0x5616cbe29da0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff11c013d30 tx=0x7ff11c0116c0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.762+0000 7ff13bca5b80 1 -- 172.21.15.2:0/2354356099 shutdown_connections 2023-11-16T16:54:44.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.762+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/2354356099 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5616cbe4b0d0 0x5616cbe4d4e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.762+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/2354356099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe26b90 0x5616cbe2a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.762+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/2354356099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbc7c6b0 0x5616cbe2cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.762+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/2354356099 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe26720 0x5616cbe29da0 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff11c013d30 tx=0x7ff11c0116c0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.762+0000 7ff109ffb700 0 client.25817 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:54:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.762+0000 7ff13bca5b80 1 -- 172.21.15.2:0/2354356099 >> 172.21.15.2:0/2354356099 conn(0x5616cbe15f50 msgr2=0x5616cbe1a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.762+0000 7ff13bca5b80 1 -- 172.21.15.2:0/2354356099 shutdown_connections 2023-11-16T16:54:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.762+0000 7ff13bca5b80 1 -- 172.21.15.2:0/2354356099 wait complete. 2023-11-16T16:54:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.763+0000 7ff13bca5b80 20 client.25817 trim_cache size 0 max 16384 2023-11-16T16:54:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.781+0000 7ff13bca5b80 1 Processor -- start 2023-11-16T16:54:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.781+0000 7ff13bca5b80 1 -- start start 2023-11-16T16:54:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.782+0000 7ff13bca5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbc7c6b0 0x5616cbe25720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.782+0000 7ff13bca5b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe26b90 0x5616cbe2f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.782+0000 7ff13bca5b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe245c0 0x5616cbe27670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.782+0000 7ff13bca5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616cbe4db40 con 0x5616cbc7c6b0 2023-11-16T16:54:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.782+0000 7ff13bca5b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616cbe24a80 con 0x5616cbe245c0 2023-11-16T16:54:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.782+0000 7ff13bca5b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616cbe25dc0 con 0x5616cbe26b90 2023-11-16T16:54:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.782+0000 7ff12619b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe26b90 0x5616cbe2f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.782+0000 7ff12619b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe26b90 0x5616cbe2f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:37762/0 (socket says 172.21.15.2:37762) 2023-11-16T16:54:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.782+0000 7ff12619b700 1 -- 172.21.15.2:0/886173549 learned_addr learned my addr 172.21.15.2:0/886173549 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:54:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.782+0000 7ff12599a700 1 --2- 172.21.15.2:0/886173549 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbc7c6b0 0x5616cbe25720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.782+0000 7ff12599a700 1 --2- 172.21.15.2:0/886173549 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe245c0 0x5616cbe27670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.782+0000 7ff12619b700 1 -- 172.21.15.2:0/886173549 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe245c0 msgr2=0x5616cbe27670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.782+0000 7ff12619b700 1 --2- 172.21.15.2:0/886173549 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe245c0 0x5616cbe27670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.782+0000 7ff12619b700 1 -- 172.21.15.2:0/886173549 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbc7c6b0 msgr2=0x5616cbe25720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.782+0000 7ff12619b700 1 --2- 172.21.15.2:0/886173549 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbc7c6b0 0x5616cbe25720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.782+0000 7ff12619b700 1 -- 172.21.15.2:0/886173549 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616cbe176f0 con 0x5616cbe26b90 2023-11-16T16:54:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.782+0000 7ff12599a700 1 --2- 172.21.15.2:0/886173549 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbc7c6b0 0x5616cbe25720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:54:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.783+0000 7ff12619b700 1 --2- 172.21.15.2:0/886173549 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe26b90 0x5616cbe2f670 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff11c005950 tx=0x7ff11c0094f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=794a6bd5a4b37285 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.783+0000 7ff1153ff700 1 -- 172.21.15.2:0/886173549 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11c011ac0 con 0x5616cbe26b90 2023-11-16T16:54:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.783+0000 7ff1153ff700 1 -- 172.21.15.2:0/886173549 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff11c0050b0 con 0x5616cbe26b90 2023-11-16T16:54:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.783+0000 7ff1153ff700 1 -- 172.21.15.2:0/886173549 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11c0038e0 con 0x5616cbe26b90 2023-11-16T16:54:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.784+0000 7ff13bca5b80 1 -- 172.21.15.2:0/886173549 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe26b90 msgr2=0x5616cbe2f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.784+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/886173549 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe26b90 0x5616cbe2f670 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff11c005950 tx=0x7ff11c0094f0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.784+0000 7ff13bca5b80 1 -- 172.21.15.2:0/886173549 shutdown_connections 2023-11-16T16:54:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.784+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/886173549 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe245c0 0x5616cbe27670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.784+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/886173549 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbc7c6b0 0x5616cbe25720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.784+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/886173549 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe26b90 0x5616cbe2f670 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.784+0000 7ff13bca5b80 1 -- 172.21.15.2:0/886173549 >> 172.21.15.2:0/886173549 conn(0x5616cbe15f50 msgr2=0x5616cbe26f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.784+0000 7ff13bca5b80 1 -- 172.21.15.2:0/886173549 shutdown_connections 2023-11-16T16:54:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.784+0000 7ff13bca5b80 1 -- 172.21.15.2:0/886173549 wait complete. 2023-11-16T16:54:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.785+0000 7ff13bca5b80 1 -- start start 2023-11-16T16:54:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.785+0000 7ff13bca5b80 1 Processor -- start 2023-11-16T16:54:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.786+0000 7ff13bca5b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbc7c6b0 0x5616cbe97210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.786+0000 7ff13bca5b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe245c0 0x5616cbe97750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.786+0000 7ff13bca5b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe26b90 0x5616cbe97fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.786+0000 7ff13bca5b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616cbe98580 con 0x5616cbc7c6b0 2023-11-16T16:54:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.786+0000 7ff13bca5b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616cbe986c0 con 0x5616cbe26b90 2023-11-16T16:54:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.786+0000 7ff13bca5b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616cbe98880 con 0x5616cbe245c0 2023-11-16T16:54:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.786+0000 7ff13bca5b80 10 client.0 _ll_register_callbacks cb 0x7ff11c013800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:54:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.786+0000 7ff12619b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe245c0 0x5616cbe97750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.786+0000 7ff12599a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbc7c6b0 0x5616cbe97210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.786+0000 7ff12619b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe245c0 0x5616cbe97750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:37772/0 (socket says 172.21.15.2:37772) 2023-11-16T16:54:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.786+0000 7ff12619b700 1 -- 172.21.15.2:0/3176620385 learned_addr learned my addr 172.21.15.2:0/3176620385 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:54:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.786+0000 7ff12599a700 1 --2- 172.21.15.2:0/3176620385 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe26b90 0x5616cbe97fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.786+0000 7ff12619b700 1 -- 172.21.15.2:0/3176620385 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe26b90 msgr2=0x5616cbe97fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.786+0000 7ff12619b700 1 --2- 172.21.15.2:0/3176620385 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe26b90 0x5616cbe97fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.786+0000 7ff12619b700 1 -- 172.21.15.2:0/3176620385 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbc7c6b0 msgr2=0x5616cbe97210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:54:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.786+0000 7ff12619b700 1 --2- 172.21.15.2:0/3176620385 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbc7c6b0 0x5616cbe97210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.786+0000 7ff12619b700 1 -- 172.21.15.2:0/3176620385 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5616cbd38130 con 0x5616cbe245c0 2023-11-16T16:54:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.786+0000 7ff12599a700 1 --2- 172.21.15.2:0/3176620385 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe26b90 0x5616cbe97fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:54:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.787+0000 7ff12619b700 1 --2- 172.21.15.2:0/3176620385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe245c0 0x5616cbe97750 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff11c030040 tx=0x7ff11c0054d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f19c64815ea71be7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:54:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.787+0000 7ff109ffb700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:54:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.787+0000 7ff109ffb700 1 -- 172.21.15.2:0/3176620385 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11c011ac0 con 0x5616cbe245c0 2023-11-16T16:54:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.787+0000 7ff109ffb700 1 -- 172.21.15.2:0/3176620385 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ff11c009310 con 0x5616cbe245c0 2023-11-16T16:54:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.787+0000 7ff13bca5b80 1 -- 172.21.15.2:0/3176620385 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff11000f920 con 0x5616cbe245c0 2023-11-16T16:54:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.788+0000 7ff109ffb700 1 -- 172.21.15.2:0/3176620385 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff11c003890 con 0x5616cbe245c0 2023-11-16T16:54:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.788+0000 7ff13bca5b80 10 client.25801 fetch_fsmap learned FSMap version 35 2023-11-16T16:54:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.788+0000 7ff13bca5b80 1 -- 172.21.15.2:0/3176620385 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x5616cbb9fd80 con 0x5616cbe245c0 2023-11-16T16:54:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.788+0000 7ff109ffb700 1 -- 172.21.15.2:0/3176620385 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff11000f920 con 0x5616cbe245c0 2023-11-16T16:54:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.788+0000 7ff109ffb700 1 -- 172.21.15.2:0/3176620385 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7ff11c034070 con 0x5616cbe245c0 2023-11-16T16:54:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.788+0000 7ff13bca5b80 10 client.25801 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:54:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.788+0000 7ff13bca5b80 10 client.25801 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:54:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.788+0000 7ff13bca5b80 20 client.25801 populate_metadata read hostname 'smithi002' 2023-11-16T16:54:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.788+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/3176620385 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5616cbe9fe50 0x5616cbea2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:54:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.788+0000 7ff13bca5b80 4 client.25801 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:54:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.788+0000 7ff13bca5b80 1 -- 172.21.15.2:0/3176620385 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5616cbd40600 con 0x5616cbe9fe50 2023-11-16T16:54:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.789+0000 7ff12599a700 1 --2- 172.21.15.2:0/3176620385 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5616cbe9fe50 0x5616cbea2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:54:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.789+0000 7ff12599a700 1 --2- 172.21.15.2:0/3176620385 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5616cbe9fe50 0x5616cbea2260 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e401d7bdd9f2114a server_cookie=b898c4e988ef3289 in_seq=0 out_seq=0 2023-11-16T16:54:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:44.789+0000 7ff109ffb700 10 client.25801 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:54:44.868 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:44 smithi155 ceph-mon[131607]: pgmap v148: 129 pgs: 129 active+clean; 439 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 369 op/s 2023-11-16T16:54:44.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:44 smithi002 ceph-mon[128712]: pgmap v148: 129 pgs: 129 active+clean; 439 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 369 op/s 2023-11-16T16:54:45.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:44 smithi141 ceph-mon[131184]: pgmap v148: 129 pgs: 129 active+clean; 439 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 369 op/s 2023-11-16T16:54:46.746 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.744+0000 7ff109ffb700 1 -- 172.21.15.2:0/3176620385 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5616cbd40600 con 0x5616cbe9fe50 2023-11-16T16:54:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.744+0000 7ff109ffb700 10 client.25801 handle_command_reply: tid=0 2023-11-16T16:54:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.744+0000 7ff13bca5b80 1 client.25801 shutdown 2023-11-16T16:54:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.744+0000 7ff13bca5b80 1 -- 172.21.15.2:0/3176620385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe245c0 msgr2=0x5616cbe97750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.744+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/3176620385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe245c0 0x5616cbe97750 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff11c030040 tx=0x7ff11c0054d0 comp rx=0 tx=0).stop 2023-11-16T16:54:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.745+0000 7ff13bca5b80 1 -- 172.21.15.2:0/3176620385 shutdown_connections 2023-11-16T16:54:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.745+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/3176620385 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5616cbe26b90 0x5616cbe97fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.745+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/3176620385 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5616cbc7c6b0 0x5616cbe97210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.745+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/3176620385 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5616cbe9fe50 0x5616cbea2260 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.745+0000 7ff13bca5b80 1 --2- 172.21.15.2:0/3176620385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5616cbe245c0 0x5616cbe97750 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:46.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.745+0000 7ff109ffb700 0 client.25801 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:54:46.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.745+0000 7ff13bca5b80 1 -- 172.21.15.2:0/3176620385 >> 172.21.15.2:0/3176620385 conn(0x5616cbe15f50 msgr2=0x5616cbe25590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:46.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.745+0000 7ff13bca5b80 1 -- 172.21.15.2:0/3176620385 shutdown_connections 2023-11-16T16:54:46.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.745+0000 7ff13bca5b80 1 -- 172.21.15.2:0/3176620385 wait complete. 2023-11-16T16:54:46.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.746+0000 7ff13bca5b80 20 client.25801 trim_cache size 0 max 16384 2023-11-16T16:54:46.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:54:46.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:54:46.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:54:46.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:54:46.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.752+0000 7ff1283ff700 1 -- 172.21.15.2:0/1746617612 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ff10c0a1340 msgr2=0x7ff10c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:46.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.752+0000 7ff1283ff700 1 --2- 172.21.15.2:0/1746617612 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ff10c0a1340 0x7ff10c0a37f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff118096080 tx=0x7ff118004c50 comp rx=0 tx=0).stop 2023-11-16T16:54:46.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.752+0000 7ff1283ff700 1 -- 172.21.15.2:0/1746617612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff12010a810 msgr2=0x7ff12019cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:54:46.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.752+0000 7ff1283ff700 1 --2- 172.21.15.2:0/1746617612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff12010a810 0x7ff12019cf60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff11800efc0 tx=0x7ff11800c3a0 comp rx=0 tx=0).stop 2023-11-16T16:54:46.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.752+0000 7ff1283ff700 1 -- 172.21.15.2:0/1746617612 shutdown_connections 2023-11-16T16:54:46.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.752+0000 7ff1283ff700 1 --2- 172.21.15.2:0/1746617612 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ff10c0a1340 0x7ff10c0a37f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:46.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.752+0000 7ff1283ff700 1 --2- 172.21.15.2:0/1746617612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff12010a810 0x7ff12019cf60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:46.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.752+0000 7ff1283ff700 1 --2- 172.21.15.2:0/1746617612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff120101440 0x7ff1201985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:46.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.752+0000 7ff1283ff700 1 --2- 172.21.15.2:0/1746617612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff120101d50 0x7ff120198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:54:46.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.752+0000 7ff1283ff700 1 -- 172.21.15.2:0/1746617612 >> 172.21.15.2:0/1746617612 conn(0x7ff1200763a0 msgr2=0x7ff1200fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:54:46.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.753+0000 7ff1283ff700 1 -- 172.21.15.2:0/1746617612 shutdown_connections 2023-11-16T16:54:46.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:46.753+0000 7ff1283ff700 1 -- 172.21.15.2:0/1746617612 wait complete. 2023-11-16T16:54:46.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:46 smithi002 ceph-mon[128712]: pgmap v149: 129 pgs: 129 active+clean; 439 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 369 op/s 2023-11-16T16:54:47.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:46 smithi141 ceph-mon[131184]: pgmap v149: 129 pgs: 129 active+clean; 439 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 369 op/s 2023-11-16T16:54:47.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:46 smithi155 ceph-mon[131607]: pgmap v149: 129 pgs: 129 active+clean; 439 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 369 op/s 2023-11-16T16:54:48.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:48 smithi002 ceph-mon[128712]: pgmap v150: 129 pgs: 129 active+clean; 459 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.6 MiB/s wr, 744 op/s 2023-11-16T16:54:49.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:48 smithi141 ceph-mon[131184]: pgmap v150: 129 pgs: 129 active+clean; 459 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.6 MiB/s wr, 744 op/s 2023-11-16T16:54:49.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:48 smithi155 ceph-mon[131607]: pgmap v150: 129 pgs: 129 active+clean; 459 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.6 MiB/s wr, 744 op/s 2023-11-16T16:54:50.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:50 smithi002 ceph-mon[128712]: pgmap v151: 129 pgs: 129 active+clean; 459 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 599 op/s 2023-11-16T16:54:51.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:50 smithi141 ceph-mon[131184]: pgmap v151: 129 pgs: 129 active+clean; 459 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 599 op/s 2023-11-16T16:54:51.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:50 smithi155 ceph-mon[131607]: pgmap v151: 129 pgs: 129 active+clean; 459 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.6 MiB/s wr, 599 op/s 2023-11-16T16:54:52.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:52 smithi002 ceph-mon[128712]: pgmap v152: 129 pgs: 129 active+clean; 461 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.6 MiB/s wr, 927 op/s 2023-11-16T16:54:53.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:52 smithi141 ceph-mon[131184]: pgmap v152: 129 pgs: 129 active+clean; 461 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.6 MiB/s wr, 927 op/s 2023-11-16T16:54:53.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:52 smithi155 ceph-mon[131607]: pgmap v152: 129 pgs: 129 active+clean; 461 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.6 MiB/s wr, 927 op/s 2023-11-16T16:54:55.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:54 smithi141 ceph-mon[131184]: pgmap v153: 129 pgs: 129 active+clean; 461 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 703 op/s 2023-11-16T16:54:55.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:54 smithi155 ceph-mon[131607]: pgmap v153: 129 pgs: 129 active+clean; 461 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 703 op/s 2023-11-16T16:54:55.159 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:54 smithi002 ceph-mon[128712]: pgmap v153: 129 pgs: 129 active+clean; 461 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 703 op/s 2023-11-16T16:54:57.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:56 smithi141 ceph-mon[131184]: pgmap v154: 129 pgs: 129 active+clean; 461 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 704 op/s 2023-11-16T16:54:57.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:56 smithi155 ceph-mon[131607]: pgmap v154: 129 pgs: 129 active+clean; 461 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 704 op/s 2023-11-16T16:54:57.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:56 smithi002 ceph-mon[128712]: pgmap v154: 129 pgs: 129 active+clean; 461 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 704 op/s 2023-11-16T16:54:59.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:54:58 smithi141 ceph-mon[131184]: pgmap v155: 129 pgs: 129 active+clean; 487 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.9 MiB/s wr, 953 op/s 2023-11-16T16:54:59.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:54:58 smithi155 ceph-mon[131607]: pgmap v155: 129 pgs: 129 active+clean; 487 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.9 MiB/s wr, 953 op/s 2023-11-16T16:54:59.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:54:58 smithi002 ceph-mon[128712]: pgmap v155: 129 pgs: 129 active+clean; 487 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.9 MiB/s wr, 953 op/s 2023-11-16T16:54:59.964 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.962+0000 7fc25ebff700 1 -- 172.21.15.2:0/2748492703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2580746a0 msgr2=0x7fc258072d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:05.400 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.962+0000 7fc25ebff700 1 --2- 172.21.15.2:0/2748492703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2580746a0 0x7fc258072d20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009990 tx=0x7fc24c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.400 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.962+0000 7fc25ebff700 1 -- 172.21.15.2:0/2748492703 shutdown_connections 2023-11-16T16:55:05.400 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.962+0000 7fc25ebff700 1 --2- 172.21.15.2:0/2748492703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2580746a0 0x7fc258072d20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.400 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.962+0000 7fc25ebff700 1 --2- 172.21.15.2:0/2748492703 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc25810ce50 0x7fc25810f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.400 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.962+0000 7fc25ebff700 1 --2- 172.21.15.2:0/2748492703 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc258074b00 0x7fc258073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.400 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.962+0000 7fc25ebff700 1 -- 172.21.15.2:0/2748492703 >> 172.21.15.2:0/2748492703 conn(0x7fc258077550 msgr2=0x7fc258077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:05.400 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.962+0000 7fc25ebff700 1 -- 172.21.15.2:0/2748492703 shutdown_connections 2023-11-16T16:55:05.400 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.962+0000 7fc25ebff700 1 -- 172.21.15.2:0/2748492703 wait complete. 2023-11-16T16:55:05.402 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:00 smithi155 ceph-mon[131607]: pgmap v156: 129 pgs: 129 active+clean; 487 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 578 op/s 2023-11-16T16:55:05.402 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:02 smithi155 ceph-mon[131607]: pgmap v157: 129 pgs: 129 active+clean; 500 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.1 MiB/s wr, 879 op/s 2023-11-16T16:55:05.402 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:04 smithi155 ceph-mon[131607]: pgmap v158: 129 pgs: 129 active+clean; 500 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 551 op/s 2023-11-16T16:55:05.402 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:00 smithi141 ceph-mon[131184]: pgmap v156: 129 pgs: 129 active+clean; 487 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 578 op/s 2023-11-16T16:55:05.403 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:02 smithi141 ceph-mon[131184]: pgmap v157: 129 pgs: 129 active+clean; 500 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.1 MiB/s wr, 879 op/s 2023-11-16T16:55:05.403 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:04 smithi141 ceph-mon[131184]: pgmap v158: 129 pgs: 129 active+clean; 500 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 551 op/s 2023-11-16T16:55:05.412 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.963+0000 7fc25ebff700 1 Processor -- start 2023-11-16T16:55:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.964+0000 7fc25ebff700 1 -- start start 2023-11-16T16:55:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.964+0000 7fc25ebff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2580746a0 0x7fc2581985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.964+0000 7fc25ebff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc258074b00 0x7fc258198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.964+0000 7fc25ebff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc25810ce50 0x7fc25819cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.964+0000 7fc25ebff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25819d5d0 con 0x7fc258074b00 2023-11-16T16:55:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.964+0000 7fc25ebff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25819d740 con 0x7fc25810ce50 2023-11-16T16:55:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.964+0000 7fc25ebff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25819d900 con 0x7fc2580746a0 2023-11-16T16:55:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.964+0000 7fc25c99b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2580746a0 0x7fc2581985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.964+0000 7fc25d19c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc25810ce50 0x7fc25819cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.964+0000 7fc257fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc258074b00 0x7fc258198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.964+0000 7fc25c99b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2580746a0 0x7fc2581985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:51586/0 (socket says 172.21.15.2:51586) 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.964+0000 7fc25c99b700 1 -- 172.21.15.2:0/1862686165 learned_addr learned my addr 172.21.15.2:0/1862686165 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.965+0000 7fc25d19c700 1 -- 172.21.15.2:0/1862686165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2580746a0 msgr2=0x7fc2581985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.965+0000 7fc25d19c700 1 --2- 172.21.15.2:0/1862686165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2580746a0 0x7fc2581985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.965+0000 7fc25d19c700 1 -- 172.21.15.2:0/1862686165 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc258074b00 msgr2=0x7fc258198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.965+0000 7fc25d19c700 1 --2- 172.21.15.2:0/1862686165 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc258074b00 0x7fc258198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.965+0000 7fc25d19c700 1 -- 172.21.15.2:0/1862686165 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc24c0095f0 con 0x7fc25810ce50 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.965+0000 7fc25d19c700 1 --2- 172.21.15.2:0/1862686165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc25810ce50 0x7fc25819cf90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc25000ef90 tx=0x7fc25000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.965+0000 7fc255ffb700 1 -- 172.21.15.2:0/1862686165 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc250018070 con 0x7fc25810ce50 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.966+0000 7fc25ebff700 1 -- 172.21.15.2:0/1862686165 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc258112590 con 0x7fc25810ce50 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.966+0000 7fc255ffb700 1 -- 172.21.15.2:0/1862686165 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc25000f040 con 0x7fc25810ce50 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.966+0000 7fc255ffb700 1 -- 172.21.15.2:0/1862686165 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25000cc30 con 0x7fc25810ce50 2023-11-16T16:55:05.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.966+0000 7fc25ebff700 1 -- 172.21.15.2:0/1862686165 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc258112400 con 0x7fc25810ce50 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.970+0000 7fc255ffb700 1 -- 172.21.15.2:0/1862686165 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fc250003a00 con 0x7fc25810ce50 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.971+0000 7fc255ffb700 1 --2- 172.21.15.2:0/1862686165 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fc2480a12c0 0x7fc2480a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.971+0000 7fc27244db80 1 Processor -- start 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.972+0000 7fc255ffb700 1 -- 172.21.15.2:0/1862686165 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fc2500ccbc0 con 0x7fc25810ce50 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.972+0000 7fc27244db80 1 -- start start 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.972+0000 7fc25c99b700 1 --2- 172.21.15.2:0/1862686165 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fc2480a12c0 0x7fc2480a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.972+0000 7fc27244db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b47306b0 0x55c8b48da1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.972+0000 7fc27244db80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b48da720 0x55c8b48d4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.972+0000 7fc27244db80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b48dab90 0x55c8b48d4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.972+0000 7fc27244db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b47da000 con 0x55c8b47306b0 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.972+0000 7fc27244db80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b482c840 con 0x55c8b48da720 2023-11-16T16:55:05.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.972+0000 7fc27244db80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b4825cd0 con 0x55c8b48dab90 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.973+0000 7fc25d19c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b48dab90 0x55c8b48d4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.973+0000 7fc25d19c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b48dab90 0x55c8b48d4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:51596/0 (socket says 172.21.15.2:51596) 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.973+0000 7fc25c99b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b47306b0 0x55c8b48da1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.973+0000 7fc25d19c700 1 -- 172.21.15.2:0/492458961 learned_addr learned my addr 172.21.15.2:0/492458961 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.973+0000 7fc257fff700 1 --2- 172.21.15.2:0/492458961 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b48da720 0x55c8b48d4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.973+0000 7fc25d19c700 1 -- 172.21.15.2:0/492458961 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b48da720 msgr2=0x55c8b48d4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.973+0000 7fc25c99b700 1 --2- 172.21.15.2:0/1862686165 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fc2480a12c0 0x7fc2480a3770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc24c006010 tx=0x7fc24c00b610 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.974+0000 7fc25d19c700 1 --2- 172.21.15.2:0/492458961 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b48da720 0x55c8b48d4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.974+0000 7fc25d19c700 1 -- 172.21.15.2:0/492458961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b47306b0 msgr2=0x55c8b48da1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.974+0000 7fc25d19c700 1 --2- 172.21.15.2:0/492458961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b47306b0 0x55c8b48da1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.974+0000 7fc25d19c700 1 -- 172.21.15.2:0/492458961 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8b46a75d0 con 0x55c8b48dab90 2023-11-16T16:55:05.416 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.974+0000 7fc25c99b700 1 --2- 172.21.15.2:0/492458961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b47306b0 0x55c8b48da1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:55:05.417 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.974+0000 7fc257fff700 1 --2- 172.21.15.2:0/492458961 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b48da720 0x55c8b48d4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:55:05.417 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.974+0000 7fc25d19c700 1 --2- 172.21.15.2:0/492458961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b48dab90 0x55c8b48d4b00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc25001b000 tx=0x7fc250099510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69a5377b1527af2a server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:05.417 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.974+0000 7fc242ffd700 1 -- 172.21.15.2:0/492458961 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25009d980 con 0x55c8b48dab90 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.974+0000 7fc242ffd700 1 -- 172.21.15.2:0/492458961 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc2500a1d10 con 0x55c8b48dab90 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.974+0000 7fc242ffd700 1 -- 172.21.15.2:0/492458961 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25009de40 con 0x55c8b48dab90 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.975+0000 7fc27244db80 1 -- 172.21.15.2:0/492458961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b48dab90 msgr2=0x55c8b48d4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.975+0000 7fc27244db80 1 --2- 172.21.15.2:0/492458961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b48dab90 0x55c8b48d4b00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc25001b000 tx=0x7fc250099510 comp rx=0 tx=0).stop 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.975+0000 7fc27244db80 1 -- 172.21.15.2:0/492458961 shutdown_connections 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.975+0000 7fc27244db80 1 --2- 172.21.15.2:0/492458961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b48dab90 0x55c8b48d4b00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.975+0000 7fc27244db80 1 --2- 172.21.15.2:0/492458961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b47306b0 0x55c8b48da1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.975+0000 7fc27244db80 1 --2- 172.21.15.2:0/492458961 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b48da720 0x55c8b48d4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.975+0000 7fc27244db80 1 -- 172.21.15.2:0/492458961 >> 172.21.15.2:0/492458961 conn(0x55c8b48c9f50 msgr2=0x55c8b48cf270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.975+0000 7fc27244db80 1 -- 172.21.15.2:0/492458961 shutdown_connections 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.975+0000 7fc27244db80 1 -- 172.21.15.2:0/492458961 wait complete. 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.976+0000 7fc27244db80 1 -- start start 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.976+0000 7fc27244db80 1 Processor -- start 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.976+0000 7fc27244db80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b47306b0 0x55c8b48e0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.976+0000 7fc27244db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b48da720 0x55c8b48ddd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.977+0000 7fc27244db80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b48dab90 0x55c8b48de4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.977+0000 7fc27244db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b47c7880 con 0x55c8b48da720 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.977+0000 7fc27244db80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b47c7520 con 0x55c8b48dab90 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.977+0000 7fc27244db80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b46a98c0 con 0x55c8b47306b0 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.977+0000 7fc27244db80 10 client.0 _ll_register_callbacks cb 0x55c8b47e5720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.977+0000 7fc25c99b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b47306b0 0x55c8b48e0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.977+0000 7fc25c99b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b47306b0 0x55c8b48e0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:51612/0 (socket says 172.21.15.2:51612) 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.977+0000 7fc25c99b700 1 -- 172.21.15.2:0/249558422 learned_addr learned my addr 172.21.15.2:0/249558422 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.977+0000 7fc25d19c700 1 --2- 172.21.15.2:0/249558422 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b48dab90 0x55c8b48de4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.977+0000 7fc257fff700 1 --2- 172.21.15.2:0/249558422 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b48da720 0x55c8b48ddd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.977+0000 7fc25c99b700 1 -- 172.21.15.2:0/249558422 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b48dab90 msgr2=0x55c8b48de4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.977+0000 7fc25c99b700 1 --2- 172.21.15.2:0/249558422 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b48dab90 0x55c8b48de4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.977+0000 7fc25c99b700 1 -- 172.21.15.2:0/249558422 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b48da720 msgr2=0x55c8b48ddd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.977+0000 7fc25c99b700 1 --2- 172.21.15.2:0/249558422 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b48da720 0x55c8b48ddd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.977+0000 7fc25c99b700 1 -- 172.21.15.2:0/249558422 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc258112400 con 0x55c8b47306b0 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.977+0000 7fc25c99b700 1 --2- 172.21.15.2:0/249558422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b47306b0 0x55c8b48e0b00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc24c005490 tx=0x7fc24c009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c00b30133cd8f3f8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.977+0000 7fc237fff700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:55:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.978+0000 7fc237fff700 1 -- 172.21.15.2:0/249558422 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c01d040 con 0x55c8b47306b0 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.978+0000 7fc27244db80 1 -- 172.21.15.2:0/249558422 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c8b48c7710 con 0x55c8b47306b0 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.978+0000 7fc237fff700 1 -- 172.21.15.2:0/249558422 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fc24c02c040 con 0x55c8b47306b0 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.978+0000 7fc237fff700 1 -- 172.21.15.2:0/249558422 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc24c004b70 con 0x55c8b47306b0 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.978+0000 7fc27244db80 10 client.25813 fetch_fsmap learned FSMap version 35 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.978+0000 7fc27244db80 1 -- 172.21.15.2:0/249558422 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x55c8b47ec130 con 0x55c8b47306b0 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.978+0000 7fc237fff700 1 -- 172.21.15.2:0/249558422 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c8b48c7710 con 0x55c8b47306b0 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.979+0000 7fc237fff700 1 -- 172.21.15.2:0/249558422 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fc24c030450 con 0x55c8b47306b0 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.979+0000 7fc27244db80 10 client.25813 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.979+0000 7fc27244db80 10 client.25813 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.979+0000 7fc27244db80 20 client.25813 populate_metadata read hostname 'smithi002' 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.979+0000 7fc27244db80 1 --2- 172.21.15.2:0/249558422 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55c8b48f66c0 0x55c8b48f8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.979+0000 7fc27244db80 4 client.25813 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.979+0000 7fc27244db80 1 -- 172.21.15.2:0/249558422 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c8b47e8040 con 0x55c8b48f66c0 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.979+0000 7fc257fff700 1 --2- 172.21.15.2:0/249558422 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55c8b48f66c0 0x55c8b48f8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.980+0000 7fc257fff700 1 --2- 172.21.15.2:0/249558422 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55c8b48f66c0 0x55c8b48f8ad0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=456d12fee24106e0 server_cookie=5c2822d82d704136 in_seq=0 out_seq=0 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.980+0000 7fc237fff700 10 client.25813 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.981+0000 7fc237fff700 1 -- 172.21.15.2:0/249558422 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55c8b47e8040 con 0x55c8b48f66c0 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.981+0000 7fc237fff700 10 client.25813 handle_command_reply: tid=0 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.981+0000 7fc27244db80 1 client.25813 shutdown 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.981+0000 7fc27244db80 1 -- 172.21.15.2:0/249558422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b47306b0 msgr2=0x55c8b48e0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.981+0000 7fc27244db80 1 --2- 172.21.15.2:0/249558422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b47306b0 0x55c8b48e0b00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc24c005490 tx=0x7fc24c009ca0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.981+0000 7fc27244db80 1 -- 172.21.15.2:0/249558422 shutdown_connections 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.981+0000 7fc27244db80 1 --2- 172.21.15.2:0/249558422 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55c8b48f66c0 0x55c8b48f8ad0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.981+0000 7fc27244db80 1 --2- 172.21.15.2:0/249558422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b47306b0 0x55c8b48e0b00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.981+0000 7fc237fff700 0 client.25813 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.981+0000 7fc27244db80 1 --2- 172.21.15.2:0/249558422 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b48dab90 0x55c8b48de4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.981+0000 7fc27244db80 1 --2- 172.21.15.2:0/249558422 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b48da720 0x55c8b48ddd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.981+0000 7fc27244db80 1 -- 172.21.15.2:0/249558422 >> 172.21.15.2:0/249558422 conn(0x55c8b48c9f50 msgr2=0x55c8b48ce7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.981+0000 7fc27244db80 1 -- 172.21.15.2:0/249558422 shutdown_connections 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.981+0000 7fc27244db80 1 -- 172.21.15.2:0/249558422 wait complete. 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:54:59.982+0000 7fc27244db80 20 client.25813 trim_cache size 0 max 16384 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.001+0000 7fc27244db80 1 Processor -- start 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.001+0000 7fc27244db80 1 -- start start 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.002+0000 7fc27244db80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b47306b0 0x55c8b48ed8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.002+0000 7fc27244db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b48da720 0x55c8b48cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.002+0000 7fc27244db80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b48dab90 0x55c8b48e06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.002+0000 7fc27244db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b48deb40 con 0x55c8b48da720 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.002+0000 7fc27244db80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b48ded70 con 0x55c8b47306b0 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.002+0000 7fc27244db80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b48dc300 con 0x55c8b48dab90 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.003+0000 7fc25d19c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b48dab90 0x55c8b48e06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.003+0000 7fc25c99b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b47306b0 0x55c8b48ed8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.003+0000 7fc25d19c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b48dab90 0x55c8b48e06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:51620/0 (socket says 172.21.15.2:51620) 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.003+0000 7fc25c99b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b47306b0 0x55c8b48ed8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:35856/0 (socket says 172.21.15.2:35856) 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.004+0000 7fc25d19c700 1 -- 172.21.15.2:0/2804057121 learned_addr learned my addr 172.21.15.2:0/2804057121 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.004+0000 7fc257fff700 1 --2- 172.21.15.2:0/2804057121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b48da720 0x55c8b48cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.004+0000 7fc25d19c700 1 -- 172.21.15.2:0/2804057121 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b47306b0 msgr2=0x55c8b48ed8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.004+0000 7fc25d19c700 1 --2- 172.21.15.2:0/2804057121 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b47306b0 0x55c8b48ed8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.004+0000 7fc25d19c700 1 -- 172.21.15.2:0/2804057121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b48da720 msgr2=0x55c8b48cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.004+0000 7fc25d19c700 1 --2- 172.21.15.2:0/2804057121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b48da720 0x55c8b48cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.005+0000 7fc25d19c700 1 -- 172.21.15.2:0/2804057121 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8b47ebe80 con 0x55c8b48dab90 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.005+0000 7fc257fff700 1 --2- 172.21.15.2:0/2804057121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b48da720 0x55c8b48cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.005+0000 7fc25c99b700 1 --2- 172.21.15.2:0/2804057121 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b47306b0 0x55c8b48ed8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.006+0000 7fc25d19c700 1 --2- 172.21.15.2:0/2804057121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b48dab90 0x55c8b48e06a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc2500998e0 tx=0x7fc2500a1e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85d85c1567abb1e7 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.008+0000 7fc2437fe700 1 -- 172.21.15.2:0/2804057121 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25000cdb0 con 0x55c8b48dab90 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.008+0000 7fc2437fe700 1 -- 172.21.15.2:0/2804057121 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc2500c6070 con 0x55c8b48dab90 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.008+0000 7fc2437fe700 1 -- 172.21.15.2:0/2804057121 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc250099bb0 con 0x55c8b48dab90 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.010+0000 7fc27244db80 1 -- 172.21.15.2:0/2804057121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b48dab90 msgr2=0x55c8b48e06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.010+0000 7fc27244db80 1 --2- 172.21.15.2:0/2804057121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b48dab90 0x55c8b48e06a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc2500998e0 tx=0x7fc2500a1e80 comp rx=0 tx=0).stop 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.010+0000 7fc27244db80 1 -- 172.21.15.2:0/2804057121 shutdown_connections 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.010+0000 7fc27244db80 1 --2- 172.21.15.2:0/2804057121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b48dab90 0x55c8b48e06a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.010+0000 7fc27244db80 1 --2- 172.21.15.2:0/2804057121 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b47306b0 0x55c8b48ed8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.010+0000 7fc27244db80 1 --2- 172.21.15.2:0/2804057121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b48da720 0x55c8b48cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.010+0000 7fc27244db80 1 -- 172.21.15.2:0/2804057121 >> 172.21.15.2:0/2804057121 conn(0x55c8b48c9f50 msgr2=0x55c8b48d7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.010+0000 7fc27244db80 1 -- 172.21.15.2:0/2804057121 shutdown_connections 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.011+0000 7fc27244db80 1 -- 172.21.15.2:0/2804057121 wait complete. 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.011+0000 7fc27244db80 1 -- start start 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.011+0000 7fc27244db80 1 Processor -- start 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.012+0000 7fc27244db80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b47306b0 0x55c8b48d7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.012+0000 7fc27244db80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b48da720 0x55c8b48d7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.012+0000 7fc27244db80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b48dab90 0x55c8b4945770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.013+0000 7fc27244db80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b4945e80 con 0x55c8b48da720 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.013+0000 7fc27244db80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b4945ff0 con 0x55c8b48dab90 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.013+0000 7fc27244db80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b49461b0 con 0x55c8b47306b0 2023-11-16T16:55:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.013+0000 7fc27244db80 10 client.0 _ll_register_callbacks cb 0x55c8b47e5720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.013+0000 7fc25d19c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b48dab90 0x55c8b4945770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.013+0000 7fc25d19c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b48dab90 0x55c8b4945770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:35872/0 (socket says 172.21.15.2:35872) 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.013+0000 7fc25d19c700 1 -- 172.21.15.2:0/3936016144 learned_addr learned my addr 172.21.15.2:0/3936016144 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.013+0000 7fc257fff700 1 --2- 172.21.15.2:0/3936016144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b48da720 0x55c8b48d7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.013+0000 7fc25c99b700 1 --2- 172.21.15.2:0/3936016144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b47306b0 0x55c8b48d7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.015+0000 7fc25c99b700 1 -- 172.21.15.2:0/3936016144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b48dab90 msgr2=0x55c8b4945770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.015+0000 7fc25c99b700 1 --2- 172.21.15.2:0/3936016144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b48dab90 0x55c8b4945770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.015+0000 7fc25c99b700 1 -- 172.21.15.2:0/3936016144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b48da720 msgr2=0x55c8b48d7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.015+0000 7fc25c99b700 1 --2- 172.21.15.2:0/3936016144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b48da720 0x55c8b48d7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.015+0000 7fc25c99b700 1 -- 172.21.15.2:0/3936016144 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c8b47ec130 con 0x55c8b47306b0 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.015+0000 7fc25c99b700 1 --2- 172.21.15.2:0/3936016144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b47306b0 0x55c8b48d7100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc24c00b6f0 tx=0x7fc24c004850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aee8fd81c9158e2f server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.015+0000 7fc237fff700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.016+0000 7fc237fff700 1 -- 172.21.15.2:0/3936016144 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c01d070 con 0x55c8b47306b0 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.016+0000 7fc237fff700 1 -- 172.21.15.2:0/3936016144 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fc24c004b70 con 0x55c8b47306b0 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.016+0000 7fc27244db80 1 -- 172.21.15.2:0/3936016144 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc2500a1d10 con 0x55c8b47306b0 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.016+0000 7fc237fff700 1 -- 172.21.15.2:0/3936016144 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc24c004d30 con 0x55c8b47306b0 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.016+0000 7fc27244db80 10 client.25819 fetch_fsmap learned FSMap version 35 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.016+0000 7fc27244db80 1 -- 172.21.15.2:0/3936016144 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x7fc244009d60 con 0x55c8b47306b0 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.017+0000 7fc237fff700 1 -- 172.21.15.2:0/3936016144 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc2500a1d10 con 0x55c8b47306b0 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.017+0000 7fc237fff700 1 -- 172.21.15.2:0/3936016144 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fc24c042430 con 0x55c8b47306b0 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.017+0000 7fc27244db80 10 client.25819 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.017+0000 7fc27244db80 10 client.25819 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.017+0000 7fc27244db80 20 client.25819 populate_metadata read hostname 'smithi002' 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.017+0000 7fc27244db80 1 --2- 172.21.15.2:0/3936016144 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55c8b48f66c0 0x55c8b48ec0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.017+0000 7fc27244db80 4 client.25819 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:55:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.017+0000 7fc27244db80 1 -- 172.21.15.2:0/3936016144 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c8b47f4600 con 0x55c8b48f66c0 2023-11-16T16:55:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.018+0000 7fc257fff700 1 --2- 172.21.15.2:0/3936016144 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55c8b48f66c0 0x55c8b48ec0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.024+0000 7fc257fff700 1 --2- 172.21.15.2:0/3936016144 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55c8b48f66c0 0x55c8b48ec0e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55abc01826f32d69 server_cookie=f54431274ae139a3 in_seq=0 out_seq=0 2023-11-16T16:55:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.024+0000 7fc237fff700 10 client.25819 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.958+0000 7fc237fff700 1 -- 172.21.15.2:0/3936016144 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c8b47f4600 con 0x55c8b48f66c0 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.959+0000 7fc237fff700 10 client.25819 handle_command_reply: tid=0 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.959+0000 7fc27244db80 1 client.25819 shutdown 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.959+0000 7fc27244db80 1 -- 172.21.15.2:0/3936016144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b47306b0 msgr2=0x55c8b48d7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.959+0000 7fc27244db80 1 --2- 172.21.15.2:0/3936016144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b47306b0 0x55c8b48d7100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc24c00b6f0 tx=0x7fc24c004850 comp rx=0 tx=0).stop 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.961+0000 7fc27244db80 1 -- 172.21.15.2:0/3936016144 shutdown_connections 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.961+0000 7fc27244db80 1 --2- 172.21.15.2:0/3936016144 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x55c8b48f66c0 0x55c8b48ec0e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.961+0000 7fc27244db80 1 --2- 172.21.15.2:0/3936016144 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c8b47306b0 0x55c8b48d7100 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.961+0000 7fc27244db80 1 --2- 172.21.15.2:0/3936016144 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x55c8b48dab90 0x55c8b4945770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.961+0000 7fc27244db80 1 --2- 172.21.15.2:0/3936016144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c8b48da720 0x55c8b48d7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.961+0000 7fc27244db80 1 -- 172.21.15.2:0/3936016144 >> 172.21.15.2:0/3936016144 conn(0x55c8b48c9f50 msgr2=0x55c8b48d8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.961+0000 7fc237fff700 0 client.25819 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.962+0000 7fc27244db80 1 -- 172.21.15.2:0/3936016144 shutdown_connections 2023-11-16T16:55:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.962+0000 7fc27244db80 1 -- 172.21.15.2:0/3936016144 wait complete. 2023-11-16T16:55:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.963+0000 7fc27244db80 20 client.25819 trim_cache size 0 max 16384 2023-11-16T16:55:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:55:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:55:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:55:05.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:55:05.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.964+0000 7fc25ebff700 1 -- 172.21.15.2:0/1862686165 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fc2480a12c0 msgr2=0x7fc2480a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:05.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.964+0000 7fc25ebff700 1 --2- 172.21.15.2:0/1862686165 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fc2480a12c0 0x7fc2480a3770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc24c006010 tx=0x7fc24c00b610 comp rx=0 tx=0).stop 2023-11-16T16:55:05.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.964+0000 7fc25ebff700 1 -- 172.21.15.2:0/1862686165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc25810ce50 msgr2=0x7fc25819cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:05.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.964+0000 7fc25ebff700 1 --2- 172.21.15.2:0/1862686165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc25810ce50 0x7fc25819cf90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc25000ef90 tx=0x7fc25000c3a0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.965+0000 7fc25ebff700 1 -- 172.21.15.2:0/1862686165 shutdown_connections 2023-11-16T16:55:05.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.965+0000 7fc25ebff700 1 --2- 172.21.15.2:0/1862686165 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fc2480a12c0 0x7fc2480a3770 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.965+0000 7fc25ebff700 1 --2- 172.21.15.2:0/1862686165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2580746a0 0x7fc2581985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.965+0000 7fc25ebff700 1 --2- 172.21.15.2:0/1862686165 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc25810ce50 0x7fc25819cf90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.965+0000 7fc25ebff700 1 --2- 172.21.15.2:0/1862686165 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc258074b00 0x7fc258198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:05.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.965+0000 7fc25ebff700 1 -- 172.21.15.2:0/1862686165 >> 172.21.15.2:0/1862686165 conn(0x7fc258077550 msgr2=0x7fc258104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:05.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.965+0000 7fc25ebff700 1 -- 172.21.15.2:0/1862686165 shutdown_connections 2023-11-16T16:55:05.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:00.965+0000 7fc25ebff700 1 -- 172.21.15.2:0/1862686165 wait complete. 2023-11-16T16:55:05.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:00 smithi002 ceph-mon[128712]: pgmap v156: 129 pgs: 129 active+clean; 487 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 578 op/s 2023-11-16T16:55:05.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:02 smithi002 ceph-mon[128712]: pgmap v157: 129 pgs: 129 active+clean; 500 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.1 MiB/s wr, 879 op/s 2023-11-16T16:55:05.473 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:04 smithi002 ceph-mon[128712]: pgmap v158: 129 pgs: 129 active+clean; 500 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 551 op/s 2023-11-16T16:55:07.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:06 smithi141 ceph-mon[131184]: pgmap v159: 129 pgs: 129 active+clean; 500 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 551 op/s 2023-11-16T16:55:07.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:06 smithi155 ceph-mon[131607]: pgmap v159: 129 pgs: 129 active+clean; 500 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 551 op/s 2023-11-16T16:55:07.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:06 smithi002 ceph-mon[128712]: pgmap v159: 129 pgs: 129 active+clean; 500 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 551 op/s 2023-11-16T16:55:09.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:08 smithi141 ceph-mon[131184]: pgmap v160: 129 pgs: 129 active+clean; 509 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.9 MiB/s wr, 697 op/s 2023-11-16T16:55:09.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:08 smithi155 ceph-mon[131607]: pgmap v160: 129 pgs: 129 active+clean; 509 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.9 MiB/s wr, 697 op/s 2023-11-16T16:55:09.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:08 smithi002 ceph-mon[128712]: pgmap v160: 129 pgs: 129 active+clean; 509 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.9 MiB/s wr, 697 op/s 2023-11-16T16:55:11.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:10 smithi141 ceph-mon[131184]: pgmap v161: 129 pgs: 129 active+clean; 509 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 447 op/s 2023-11-16T16:55:11.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:10 smithi155 ceph-mon[131607]: pgmap v161: 129 pgs: 129 active+clean; 509 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 447 op/s 2023-11-16T16:55:11.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:10 smithi002 ceph-mon[128712]: pgmap v161: 129 pgs: 129 active+clean; 509 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 447 op/s 2023-11-16T16:55:13.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:12 smithi141 ceph-mon[131184]: pgmap v162: 129 pgs: 129 active+clean; 529 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.4 MiB/s wr, 673 op/s 2023-11-16T16:55:13.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:12 smithi155 ceph-mon[131607]: pgmap v162: 129 pgs: 129 active+clean; 529 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.4 MiB/s wr, 673 op/s 2023-11-16T16:55:13.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:12 smithi002 ceph-mon[128712]: pgmap v162: 129 pgs: 129 active+clean; 529 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.4 MiB/s wr, 673 op/s 2023-11-16T16:55:14.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.201+0000 7f67a39ff700 1 -- 172.21.15.2:0/903304146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f679c104630 msgr2=0x7f679c104ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:14.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.201+0000 7f67a39ff700 1 --2- 172.21.15.2:0/903304146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f679c104630 0x7f679c104ad0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f679400b0d0 tx=0x7f679400b3e0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.202+0000 7f67a39ff700 1 -- 172.21.15.2:0/903304146 shutdown_connections 2023-11-16T16:55:14.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.202+0000 7f67a39ff700 1 --2- 172.21.15.2:0/903304146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f679c104630 0x7f679c104ad0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.202+0000 7f67a39ff700 1 --2- 172.21.15.2:0/903304146 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f679c0ff230 0x7f679c0ff600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.202+0000 7f67a39ff700 1 --2- 172.21.15.2:0/903304146 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f679c0ffb40 0x7f679c104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.202+0000 7f67a39ff700 1 -- 172.21.15.2:0/903304146 >> 172.21.15.2:0/903304146 conn(0x7f679c0763a0 msgr2=0x7f679c0767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:14.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.203+0000 7f67a39ff700 1 -- 172.21.15.2:0/903304146 shutdown_connections 2023-11-16T16:55:14.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.203+0000 7f67a39ff700 1 -- 172.21.15.2:0/903304146 wait complete. 2023-11-16T16:55:14.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.205+0000 7f67a39ff700 1 Processor -- start 2023-11-16T16:55:14.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.206+0000 7f67a39ff700 1 -- start start 2023-11-16T16:55:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.207+0000 7f67a39ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f679c0ff230 0x7f679c10c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.207+0000 7f67a39ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f679c0ffb40 0x7f679c10c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.207+0000 7f67a39ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f679c104630 0x7f679c10faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.207+0000 7f67a39ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c110150 con 0x7f679c0ffb40 2023-11-16T16:55:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.207+0000 7f67a39ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c1102c0 con 0x7f679c104630 2023-11-16T16:55:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.207+0000 7f67a39ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c110480 con 0x7f679c0ff230 2023-11-16T16:55:14.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.207+0000 7f67a1f9c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f679c104630 0x7f679c10faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:14.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.208+0000 7f67a179b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f679c0ff230 0x7f679c10c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:14.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.208+0000 7f67a179b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f679c0ff230 0x7f679c10c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:54468/0 (socket says 172.21.15.2:54468) 2023-11-16T16:55:14.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.208+0000 7f67a179b700 1 -- 172.21.15.2:0/3746945373 learned_addr learned my addr 172.21.15.2:0/3746945373 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:14.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.208+0000 7f67a0f9a700 1 --2- 172.21.15.2:0/3746945373 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f679c0ffb40 0x7f679c10c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:14.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.209+0000 7f67a1f9c700 1 -- 172.21.15.2:0/3746945373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f679c0ff230 msgr2=0x7f679c10c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:14.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.209+0000 7f67a1f9c700 1 --2- 172.21.15.2:0/3746945373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f679c0ff230 0x7f679c10c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.209+0000 7f67a1f9c700 1 -- 172.21.15.2:0/3746945373 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f679c0ffb40 msgr2=0x7f679c10c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:14.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.209+0000 7f67a1f9c700 1 --2- 172.21.15.2:0/3746945373 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f679c0ffb40 0x7f679c10c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.209+0000 7f67a1f9c700 1 -- 172.21.15.2:0/3746945373 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6794009d00 con 0x7f679c104630 2023-11-16T16:55:14.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.210+0000 7f67a1f9c700 1 --2- 172.21.15.2:0/3746945373 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f679c104630 0x7f679c10faf0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6794009fd0 tx=0x7f6794007860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.211+0000 7f67927fc700 1 -- 172.21.15.2:0/3746945373 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679400e040 con 0x7f679c104630 2023-11-16T16:55:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.211+0000 7f67927fc700 1 -- 172.21.15.2:0/3746945373 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f679400bcb0 con 0x7f679c104630 2023-11-16T16:55:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.211+0000 7f67927fc700 1 -- 172.21.15.2:0/3746945373 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67940058d0 con 0x7f679c104630 2023-11-16T16:55:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.211+0000 7f67a39ff700 1 -- 172.21.15.2:0/3746945373 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679c1168d0 con 0x7f679c104630 2023-11-16T16:55:14.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.211+0000 7f67a39ff700 1 -- 172.21.15.2:0/3746945373 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679c116740 con 0x7f679c104630 2023-11-16T16:55:14.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.217+0000 7f67927fc700 1 -- 172.21.15.2:0/3746945373 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f6794004a50 con 0x7f679c104630 2023-11-16T16:55:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.222+0000 7f67b7320b80 1 Processor -- start 2023-11-16T16:55:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.222+0000 7f67927fc700 1 --2- 172.21.15.2:0/3746945373 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f67880a1410 0x7f67880a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.223+0000 7f67927fc700 1 -- 172.21.15.2:0/3746945373 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f67940ce160 con 0x7f679c104630 2023-11-16T16:55:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.223+0000 7f67b7320b80 1 -- start start 2023-11-16T16:55:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.224+0000 7f67b7320b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654791c76b0 0x56547937c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.224+0000 7f67b7320b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56547937cca0 0x565479380df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.224+0000 7f67b7320b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56547937d110 0x565479381330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.224+0000 7f67a0f9a700 1 --2- 172.21.15.2:0/3746945373 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f67880a1410 0x7f67880a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.224+0000 7f67b7320b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x565479271000 con 0x56547937d110 2023-11-16T16:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.224+0000 7f67b7320b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654792c3840 con 0x5654791c76b0 2023-11-16T16:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.224+0000 7f67b7320b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654792bccd0 con 0x56547937cca0 2023-11-16T16:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.225+0000 7f67a179b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654791c76b0 0x56547937c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.225+0000 7f67a179b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654791c76b0 0x56547937c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:40604/0 (socket says 172.21.15.2:40604) 2023-11-16T16:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.225+0000 7f67a179b700 1 -- 172.21.15.2:0/4008520612 learned_addr learned my addr 172.21.15.2:0/4008520612 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.226+0000 7f67a179b700 1 -- 172.21.15.2:0/4008520612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56547937cca0 msgr2=0x565479380df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-16T16:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.226+0000 7f67a179b700 1 --2- 172.21.15.2:0/4008520612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56547937cca0 0x565479380df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.226+0000 7f67a179b700 1 -- 172.21.15.2:0/4008520612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56547937d110 msgr2=0x565479381330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.226+0000 7f67a179b700 1 --2- 172.21.15.2:0/4008520612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56547937d110 0x565479381330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.226+0000 7f67a0f9a700 1 --2- 172.21.15.2:0/3746945373 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f67880a1410 0x7f67880a38c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f679c10d770 tx=0x7f678c009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.227+0000 7f67a179b700 1 -- 172.21.15.2:0/4008520612 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56547913e5d0 con 0x5654791c76b0 2023-11-16T16:55:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.227+0000 7f67a179b700 1 --2- 172.21.15.2:0/4008520612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654791c76b0 0x56547937c760 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f679800ae40 tx=0x7f679800f6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d561489cf64abfbf server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.228+0000 7f67877fe700 1 -- 172.21.15.2:0/4008520612 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679800a940 con 0x5654791c76b0 2023-11-16T16:55:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.228+0000 7f67877fe700 1 -- 172.21.15.2:0/4008520612 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f679800fec0 con 0x5654791c76b0 2023-11-16T16:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.228+0000 7f67877fe700 1 -- 172.21.15.2:0/4008520612 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679800a940 con 0x5654791c76b0 2023-11-16T16:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.229+0000 7f67b7320b80 1 -- 172.21.15.2:0/4008520612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654791c76b0 msgr2=0x56547937c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.229+0000 7f67b7320b80 1 --2- 172.21.15.2:0/4008520612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654791c76b0 0x56547937c760 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f679800ae40 tx=0x7f679800f6a0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.229+0000 7f67b7320b80 1 -- 172.21.15.2:0/4008520612 shutdown_connections 2023-11-16T16:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.229+0000 7f67b7320b80 1 --2- 172.21.15.2:0/4008520612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56547937d110 0x565479381330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.229+0000 7f67b7320b80 1 --2- 172.21.15.2:0/4008520612 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654791c76b0 0x56547937c760 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.229+0000 7f67b7320b80 1 --2- 172.21.15.2:0/4008520612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56547937cca0 0x565479380df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.229+0000 7f67b7320b80 1 -- 172.21.15.2:0/4008520612 >> 172.21.15.2:0/4008520612 conn(0x565479360f50 msgr2=0x565479366270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:14.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.231+0000 7f67b7320b80 1 -- 172.21.15.2:0/4008520612 shutdown_connections 2023-11-16T16:55:14.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.232+0000 7f67b7320b80 1 -- 172.21.15.2:0/4008520612 wait complete. 2023-11-16T16:55:14.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.232+0000 7f67b7320b80 1 -- start start 2023-11-16T16:55:14.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.233+0000 7f67b7320b80 1 Processor -- start 2023-11-16T16:55:14.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.234+0000 7f67b7320b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654791c76b0 0x565479385d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:14.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.234+0000 7f67b7320b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56547937cca0 0x5654793862c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:14.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.234+0000 7f67b7320b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56547937d110 0x565479386800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.235+0000 7f67b7320b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56547925e880 con 0x56547937d110 2023-11-16T16:55:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.235+0000 7f67b7320b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x56547925e520 con 0x56547937cca0 2023-11-16T16:55:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.235+0000 7f67b7320b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654791408c0 con 0x5654791c76b0 2023-11-16T16:55:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.235+0000 7f67b7320b80 10 client.0 _ll_register_callbacks cb 0x56547927c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.234+0000 7f67a0f9a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56547937cca0 0x5654793862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.234+0000 7f67a179b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654791c76b0 0x565479385d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.235+0000 7f67a179b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654791c76b0 0x565479385d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:33078/0 (socket says 172.21.15.2:33078) 2023-11-16T16:55:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.235+0000 7f67a179b700 1 -- 172.21.15.2:0/475908031 learned_addr learned my addr 172.21.15.2:0/475908031 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.235+0000 7f67a1f9c700 1 --2- 172.21.15.2:0/475908031 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56547937d110 0x565479386800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.236+0000 7f67a179b700 1 -- 172.21.15.2:0/475908031 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56547937cca0 msgr2=0x5654793862c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.236+0000 7f67a179b700 1 --2- 172.21.15.2:0/475908031 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56547937cca0 0x5654793862c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.236+0000 7f67a179b700 1 -- 172.21.15.2:0/475908031 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56547937d110 msgr2=0x565479386800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.236+0000 7f67a179b700 1 --2- 172.21.15.2:0/475908031 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56547937d110 0x565479386800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.236+0000 7f67a179b700 1 -- 172.21.15.2:0/475908031 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679800aaa0 con 0x5654791c76b0 2023-11-16T16:55:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.236+0000 7f67a179b700 1 --2- 172.21.15.2:0/475908031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654791c76b0 0x565479385d80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f679801b040 tx=0x7f6798011ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=265f67b8e9a85123 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.237+0000 7f6784ff9700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:55:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.237+0000 7f6784ff9700 1 -- 172.21.15.2:0/475908031 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679800a940 con 0x5654791c76b0 2023-11-16T16:55:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.237+0000 7f6784ff9700 1 -- 172.21.15.2:0/475908031 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6798005250 con 0x5654791c76b0 2023-11-16T16:55:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.238+0000 7f67b7320b80 1 -- 172.21.15.2:0/475908031 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56547927f040 con 0x5654791c76b0 2023-11-16T16:55:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.239+0000 7f6784ff9700 1 -- 172.21.15.2:0/475908031 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6798005410 con 0x5654791c76b0 2023-11-16T16:55:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.239+0000 7f67b7320b80 10 client.25828 fetch_fsmap learned FSMap version 35 2023-11-16T16:55:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.239+0000 7f67b7320b80 1 -- 172.21.15.2:0/475908031 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x565479283130 con 0x5654791c76b0 2023-11-16T16:55:14.240 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.240+0000 7f6784ff9700 1 -- 172.21.15.2:0/475908031 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56547927f040 con 0x5654791c76b0 2023-11-16T16:55:14.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.240+0000 7f6784ff9700 1 -- 172.21.15.2:0/475908031 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f6798018760 con 0x5654791c76b0 2023-11-16T16:55:14.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.240+0000 7f67b7320b80 10 client.25828 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:55:14.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.240+0000 7f67b7320b80 10 client.25828 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:14.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.240+0000 7f67b7320b80 20 client.25828 populate_metadata read hostname 'smithi002' 2023-11-16T16:55:14.241 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.240+0000 7f67b7320b80 1 --2- 172.21.15.2:0/475908031 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5654793960c0 0x5654793984d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.240+0000 7f67b7320b80 4 client.25828 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:55:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.240+0000 7f67b7320b80 1 -- 172.21.15.2:0/475908031 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56547928b600 con 0x5654793960c0 2023-11-16T16:55:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.241+0000 7f67a0f9a700 1 --2- 172.21.15.2:0/475908031 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5654793960c0 0x5654793984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.242+0000 7f6784ff9700 10 client.25828 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.242+0000 7f67a0f9a700 1 --2- 172.21.15.2:0/475908031 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5654793960c0 0x5654793984d0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=881b05e22bd4b07b server_cookie=b85602b44c41eff3 in_seq=0 out_seq=0 2023-11-16T16:55:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.243+0000 7f6784ff9700 1 -- 172.21.15.2:0/475908031 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56547928b600 con 0x5654793960c0 2023-11-16T16:55:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.243+0000 7f6784ff9700 10 client.25828 handle_command_reply: tid=0 2023-11-16T16:55:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.244+0000 7f67b7320b80 1 client.25828 shutdown 2023-11-16T16:55:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.245+0000 7f67b7320b80 1 -- 172.21.15.2:0/475908031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654791c76b0 msgr2=0x565479385d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:14.245 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.245+0000 7f67b7320b80 1 --2- 172.21.15.2:0/475908031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654791c76b0 0x565479385d80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f679801b040 tx=0x7f6798011ef0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.247+0000 7f67b7320b80 1 -- 172.21.15.2:0/475908031 shutdown_connections 2023-11-16T16:55:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.247+0000 7f67b7320b80 1 --2- 172.21.15.2:0/475908031 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5654793960c0 0x5654793984d0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.247+0000 7f67b7320b80 1 --2- 172.21.15.2:0/475908031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5654791c76b0 0x565479385d80 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.247+0000 7f67b7320b80 1 --2- 172.21.15.2:0/475908031 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56547937d110 0x565479386800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.247+0000 7f67b7320b80 1 --2- 172.21.15.2:0/475908031 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56547937cca0 0x5654793862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.247+0000 7f67b7320b80 1 -- 172.21.15.2:0/475908031 >> 172.21.15.2:0/475908031 conn(0x565479360f50 msgr2=0x5654793827e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:14.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.247+0000 7f6784ff9700 0 client.25828 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:14.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.247+0000 7f67b7320b80 1 -- 172.21.15.2:0/475908031 shutdown_connections 2023-11-16T16:55:14.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.248+0000 7f67b7320b80 1 -- 172.21.15.2:0/475908031 wait complete. 2023-11-16T16:55:14.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.248+0000 7f67b7320b80 20 client.25828 trim_cache size 0 max 16384 2023-11-16T16:55:14.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.281+0000 7f67b7320b80 1 Processor -- start 2023-11-16T16:55:14.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.282+0000 7f67b7320b80 1 -- start start 2023-11-16T16:55:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.284+0000 7f67b7320b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654791c76b0 0x56547938d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.284+0000 7f67b7320b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56547937cca0 0x565479384e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.284+0000 7f67b7320b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56547937d110 0x56547937fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.284+0000 7f67b7320b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x565479367a00 con 0x56547937cca0 2023-11-16T16:55:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.284+0000 7f67b7320b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x565479381ba0 con 0x5654791c76b0 2023-11-16T16:55:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.284+0000 7f67b7320b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654793805c0 con 0x56547937d110 2023-11-16T16:55:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.285+0000 7f67a1f9c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56547937d110 0x56547937fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.285+0000 7f67a1f9c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56547937d110 0x56547937fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:33080/0 (socket says 172.21.15.2:33080) 2023-11-16T16:55:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.285+0000 7f67a1f9c700 1 -- 172.21.15.2:0/2788531066 learned_addr learned my addr 172.21.15.2:0/2788531066 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.285+0000 7f67a179b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654791c76b0 0x56547938d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.286+0000 7f67a0f9a700 1 --2- 172.21.15.2:0/2788531066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56547937cca0 0x565479384e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.286+0000 7f67a0f9a700 1 -- 172.21.15.2:0/2788531066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56547937d110 msgr2=0x56547937fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.286+0000 7f67a0f9a700 1 --2- 172.21.15.2:0/2788531066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56547937d110 0x56547937fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.286+0000 7f67a0f9a700 1 -- 172.21.15.2:0/2788531066 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654791c76b0 msgr2=0x56547938d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.286+0000 7f67a0f9a700 1 --2- 172.21.15.2:0/2788531066 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654791c76b0 0x56547938d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.286+0000 7f67a0f9a700 1 -- 172.21.15.2:0/2788531066 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654793858f0 con 0x56547937cca0 2023-11-16T16:55:14.288 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.287+0000 7f67a0f9a700 1 --2- 172.21.15.2:0/2788531066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56547937cca0 0x565479384e00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x565479398d70 tx=0x7f678c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73503d79e4a8f1cb server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:14.290 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.289+0000 7f6787fff700 1 -- 172.21.15.2:0/2788531066 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678c025070 con 0x56547937cca0 2023-11-16T16:55:14.290 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.290+0000 7f6787fff700 1 -- 172.21.15.2:0/2788531066 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56547928b600 con 0x56547937cca0 2023-11-16T16:55:14.290 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.290+0000 7f6787fff700 1 -- 172.21.15.2:0/2788531066 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678c020b50 con 0x56547937cca0 2023-11-16T16:55:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.292+0000 7f67b7320b80 1 -- 172.21.15.2:0/2788531066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56547937cca0 msgr2=0x565479384e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.292+0000 7f67b7320b80 1 --2- 172.21.15.2:0/2788531066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56547937cca0 0x565479384e00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x565479398d70 tx=0x7f678c014000 comp rx=0 tx=0).stop 2023-11-16T16:55:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.292+0000 7f67b7320b80 1 -- 172.21.15.2:0/2788531066 shutdown_connections 2023-11-16T16:55:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.292+0000 7f67b7320b80 1 --2- 172.21.15.2:0/2788531066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56547937d110 0x56547937fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.292+0000 7f67b7320b80 1 --2- 172.21.15.2:0/2788531066 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5654791c76b0 0x56547938d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.292+0000 7f67b7320b80 1 --2- 172.21.15.2:0/2788531066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56547937cca0 0x565479384e00 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.292+0000 7f67b7320b80 1 -- 172.21.15.2:0/2788531066 >> 172.21.15.2:0/2788531066 conn(0x565479360f50 msgr2=0x565479392280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:14.294 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.293+0000 7f67b7320b80 1 -- 172.21.15.2:0/2788531066 shutdown_connections 2023-11-16T16:55:14.294 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.293+0000 7f67b7320b80 1 -- 172.21.15.2:0/2788531066 wait complete. 2023-11-16T16:55:14.294 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.294+0000 7f67b7320b80 1 -- start start 2023-11-16T16:55:14.294 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.294+0000 7f67b7320b80 1 Processor -- start 2023-11-16T16:55:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.296+0000 7f67b7320b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654791c76b0 0x56547938d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:14.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.296+0000 7f67b7320b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56547937cca0 0x56547938b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:14.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.296+0000 7f67b7320b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56547937d110 0x56547938b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:14.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.296+0000 7f67b7320b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56547938bb10 con 0x5654791c76b0 2023-11-16T16:55:14.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.296+0000 7f67b7320b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x56547938bc80 con 0x56547937cca0 2023-11-16T16:55:14.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.296+0000 7f67b7320b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x56547938be40 con 0x56547937d110 2023-11-16T16:55:14.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.296+0000 7f67b7320b80 10 client.0 _ll_register_callbacks cb 0x7f679800acc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:14.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.297+0000 7f67a179b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654791c76b0 0x56547938d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:14.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.297+0000 7f67a1f9c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56547937d110 0x56547938b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:14.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.297+0000 7f67a1f9c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56547937d110 0x56547938b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:33090/0 (socket says 172.21.15.2:33090) 2023-11-16T16:55:14.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.297+0000 7f67a1f9c700 1 -- 172.21.15.2:0/567216839 learned_addr learned my addr 172.21.15.2:0/567216839 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:14.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.299+0000 7f67a1f9c700 1 -- 172.21.15.2:0/567216839 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56547937cca0 msgr2=0x56547938b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:14.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.299+0000 7f67a1f9c700 1 --2- 172.21.15.2:0/567216839 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56547937cca0 0x56547938b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.299+0000 7f67a1f9c700 1 -- 172.21.15.2:0/567216839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654791c76b0 msgr2=0x56547938d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:14.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.299+0000 7f67a1f9c700 1 --2- 172.21.15.2:0/567216839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654791c76b0 0x56547938d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.299+0000 7f67a1f9c700 1 -- 172.21.15.2:0/567216839 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678c017070 con 0x56547937d110 2023-11-16T16:55:14.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.299+0000 7f67a179b700 1 --2- 172.21.15.2:0/567216839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654791c76b0 0x56547938d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:55:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.300+0000 7f67a1f9c700 1 --2- 172.21.15.2:0/567216839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56547937d110 0x56547938b5d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f67940cbb40 tx=0x7f679409aa90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffd17bc224e62644 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.300+0000 7f6784ff9700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:55:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.300+0000 7f6784ff9700 1 -- 172.21.15.2:0/567216839 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679402d2f0 con 0x56547937d110 2023-11-16T16:55:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.300+0000 7f6784ff9700 1 -- 172.21.15.2:0/567216839 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f6794005000 con 0x56547937d110 2023-11-16T16:55:14.301 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.300+0000 7f67b7320b80 1 -- 172.21.15.2:0/567216839 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56547935e710 con 0x56547937d110 2023-11-16T16:55:14.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.301+0000 7f6784ff9700 1 -- 172.21.15.2:0/567216839 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f679409af50 con 0x56547937d110 2023-11-16T16:55:14.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.301+0000 7f67b7320b80 10 client.25834 fetch_fsmap learned FSMap version 35 2023-11-16T16:55:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.301+0000 7f67b7320b80 1 -- 172.21.15.2:0/567216839 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x5654790ead80 con 0x56547937d110 2023-11-16T16:55:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.301+0000 7f6784ff9700 1 -- 172.21.15.2:0/567216839 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56547935e710 con 0x56547937d110 2023-11-16T16:55:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.302+0000 7f6784ff9700 1 -- 172.21.15.2:0/567216839 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f679409b1b0 con 0x56547937d110 2023-11-16T16:55:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.302+0000 7f67b7320b80 10 client.25834 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:55:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.302+0000 7f67b7320b80 10 client.25834 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.302+0000 7f67b7320b80 20 client.25834 populate_metadata read hostname 'smithi002' 2023-11-16T16:55:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.302+0000 7f67b7320b80 1 --2- 172.21.15.2:0/567216839 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5654793ea1c0 0x5654793ec5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.302+0000 7f67b7320b80 4 client.25834 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:55:14.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.302+0000 7f67b7320b80 1 -- 172.21.15.2:0/567216839 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56547910bf10 con 0x5654793ea1c0 2023-11-16T16:55:14.304 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.302+0000 7f67a179b700 1 --2- 172.21.15.2:0/567216839 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5654793ea1c0 0x5654793ec5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.304+0000 7f67a179b700 1 --2- 172.21.15.2:0/567216839 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5654793ea1c0 0x5654793ec5d0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a05a0fc2704074 server_cookie=f7e8ca4c48d3700b in_seq=0 out_seq=0 2023-11-16T16:55:14.305 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.305+0000 7f6784ff9700 10 client.25834 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:14.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.995+0000 7f6784ff9700 1 -- 172.21.15.2:0/567216839 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56547910bf10 con 0x5654793ea1c0 2023-11-16T16:55:14.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.995+0000 7f6784ff9700 10 client.25834 handle_command_reply: tid=0 2023-11-16T16:55:14.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.995+0000 7f67b7320b80 1 client.25834 shutdown 2023-11-16T16:55:14.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.996+0000 7f67b7320b80 1 -- 172.21.15.2:0/567216839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56547937d110 msgr2=0x56547938b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:14.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.996+0000 7f67b7320b80 1 --2- 172.21.15.2:0/567216839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56547937d110 0x56547938b5d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f67940cbb40 tx=0x7f679409aa90 comp rx=0 tx=0).stop 2023-11-16T16:55:14.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.996+0000 7f67b7320b80 1 -- 172.21.15.2:0/567216839 shutdown_connections 2023-11-16T16:55:14.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.996+0000 7f67b7320b80 1 --2- 172.21.15.2:0/567216839 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5654793ea1c0 0x5654793ec5d0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.996+0000 7f67b7320b80 1 --2- 172.21.15.2:0/567216839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56547937d110 0x56547938b5d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.996+0000 7f67b7320b80 1 --2- 172.21.15.2:0/567216839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5654791c76b0 0x56547938d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.998 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.996+0000 7f67b7320b80 1 --2- 172.21.15.2:0/567216839 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x56547937cca0 0x56547938b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:14.998 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.996+0000 7f67b7320b80 1 -- 172.21.15.2:0/567216839 >> 172.21.15.2:0/567216839 conn(0x565479360f50 msgr2=0x56547937fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:14.998 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.996+0000 7f6784ff9700 0 client.25834 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:14.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.998+0000 7f67b7320b80 1 -- 172.21.15.2:0/567216839 shutdown_connections 2023-11-16T16:55:14.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.998+0000 7f67b7320b80 1 -- 172.21.15.2:0/567216839 wait complete. 2023-11-16T16:55:14.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:14.999+0000 7f67b7320b80 20 client.25834 trim_cache size 0 max 16384 2023-11-16T16:55:14.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:55:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:55:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:55:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:55:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:15.000+0000 7f67a39ff700 1 -- 172.21.15.2:0/3746945373 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f67880a1410 msgr2=0x7f67880a38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:15.000+0000 7f67a39ff700 1 --2- 172.21.15.2:0/3746945373 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f67880a1410 0x7f67880a38c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f679c10d770 tx=0x7f678c009260 comp rx=0 tx=0).stop 2023-11-16T16:55:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:15.000+0000 7f67a39ff700 1 -- 172.21.15.2:0/3746945373 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f679c104630 msgr2=0x7f679c10faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:15.000+0000 7f67a39ff700 1 --2- 172.21.15.2:0/3746945373 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f679c104630 0x7f679c10faf0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6794009fd0 tx=0x7f6794007860 comp rx=0 tx=0).stop 2023-11-16T16:55:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:15.000+0000 7f67a39ff700 1 -- 172.21.15.2:0/3746945373 shutdown_connections 2023-11-16T16:55:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:15.000+0000 7f67a39ff700 1 --2- 172.21.15.2:0/3746945373 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f67880a1410 0x7f67880a38c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:15.000+0000 7f67a39ff700 1 --2- 172.21.15.2:0/3746945373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f679c0ff230 0x7f679c10c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:15.000+0000 7f67a39ff700 1 --2- 172.21.15.2:0/3746945373 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f679c104630 0x7f679c10faf0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:15.000+0000 7f67a39ff700 1 --2- 172.21.15.2:0/3746945373 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f679c0ffb40 0x7f679c10c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:15.000+0000 7f67a39ff700 1 -- 172.21.15.2:0/3746945373 >> 172.21.15.2:0/3746945373 conn(0x7f679c0763a0 msgr2=0x7f679c0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:15.000+0000 7f67a39ff700 1 -- 172.21.15.2:0/3746945373 shutdown_connections 2023-11-16T16:55:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:15.000+0000 7f67a39ff700 1 -- 172.21.15.2:0/3746945373 wait complete. 2023-11-16T16:55:15.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:14 smithi141 ceph-mon[131184]: pgmap v163: 129 pgs: 129 active+clean; 529 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 372 op/s 2023-11-16T16:55:15.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:14 smithi155 ceph-mon[131607]: pgmap v163: 129 pgs: 129 active+clean; 529 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 372 op/s 2023-11-16T16:55:15.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:14 smithi002 ceph-mon[128712]: pgmap v163: 129 pgs: 129 active+clean; 529 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 372 op/s 2023-11-16T16:55:17.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:16 smithi141 ceph-mon[131184]: pgmap v164: 129 pgs: 129 active+clean; 529 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 372 op/s 2023-11-16T16:55:17.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:16 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:55:17.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:16 smithi155 ceph-mon[131607]: pgmap v164: 129 pgs: 129 active+clean; 529 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 372 op/s 2023-11-16T16:55:17.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:16 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:55:17.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:16 smithi002 ceph-mon[128712]: pgmap v164: 129 pgs: 129 active+clean; 529 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 372 op/s 2023-11-16T16:55:17.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:16 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:55:19.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:18 smithi141 ceph-mon[131184]: pgmap v165: 129 pgs: 129 active+clean; 533 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.3 MiB/s wr, 773 op/s 2023-11-16T16:55:19.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:18 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:55:19.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:18 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:55:19.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:18 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:55:19.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:18 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:55:19.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:18 smithi141 ceph-mon[131184]: Service mds.all has no related daemon already placed. 2023-11-16T16:55:19.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:18 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:55:19.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:18 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:55:19.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:18 smithi155 ceph-mon[131607]: pgmap v165: 129 pgs: 129 active+clean; 533 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.3 MiB/s wr, 773 op/s 2023-11-16T16:55:19.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:18 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:55:19.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:18 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:55:19.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:18 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:55:19.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:18 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:55:19.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:18 smithi155 ceph-mon[131607]: Service mds.all has no related daemon already placed. 2023-11-16T16:55:19.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:18 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:55:19.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:18 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:55:19.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:18 smithi002 ceph-mon[128712]: pgmap v165: 129 pgs: 129 active+clean; 533 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.3 MiB/s wr, 773 op/s 2023-11-16T16:55:19.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:18 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:55:19.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:18 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:55:19.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:18 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:55:19.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:18 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:55:19.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:18 smithi002 ceph-mon[128712]: Service mds.all has no related daemon already placed. 2023-11-16T16:55:19.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:18 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:55:19.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:18 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:55:21.017 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:20 smithi141 ceph-mon[131184]: pgmap v166: 129 pgs: 129 active+clean; 533 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.5 MiB/s wr, 626 op/s 2023-11-16T16:55:21.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:20 smithi155 ceph-mon[131607]: pgmap v166: 129 pgs: 129 active+clean; 533 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.5 MiB/s wr, 626 op/s 2023-11-16T16:55:21.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:20 smithi002 ceph-mon[128712]: pgmap v166: 129 pgs: 129 active+clean; 533 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.5 MiB/s wr, 626 op/s 2023-11-16T16:55:23.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:22 smithi141 ceph-mon[131184]: pgmap v167: 129 pgs: 129 active+clean; 553 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.2 MiB/s wr, 738 op/s 2023-11-16T16:55:23.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:22 smithi155 ceph-mon[131607]: pgmap v167: 129 pgs: 129 active+clean; 553 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.2 MiB/s wr, 738 op/s 2023-11-16T16:55:23.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:22 smithi002 ceph-mon[128712]: pgmap v167: 129 pgs: 129 active+clean; 553 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.2 MiB/s wr, 738 op/s 2023-11-16T16:55:25.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:24 smithi141 ceph-mon[131184]: pgmap v168: 129 pgs: 129 active+clean; 553 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 511 op/s 2023-11-16T16:55:25.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:24 smithi155 ceph-mon[131607]: pgmap v168: 129 pgs: 129 active+clean; 553 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 511 op/s 2023-11-16T16:55:25.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:24 smithi002 ceph-mon[128712]: pgmap v168: 129 pgs: 129 active+clean; 553 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 511 op/s 2023-11-16T16:55:27.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:26 smithi155 ceph-mon[131607]: pgmap v169: 129 pgs: 129 active+clean; 553 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 511 op/s 2023-11-16T16:55:27.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:26 smithi002 ceph-mon[128712]: pgmap v169: 129 pgs: 129 active+clean; 553 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 511 op/s 2023-11-16T16:55:27.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:26 smithi141 ceph-mon[131184]: pgmap v169: 129 pgs: 129 active+clean; 553 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 511 op/s 2023-11-16T16:55:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.183+0000 7f60b75ff700 1 -- 172.21.15.2:0/2239129953 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60b0100c50 msgr2=0x7f60b0101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.183+0000 7f60b75ff700 1 --2- 172.21.15.2:0/2239129953 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60b0100c50 0x7f60b0101020 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009990 tx=0x7f60ac009ca0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.183+0000 7f60b75ff700 1 -- 172.21.15.2:0/2239129953 shutdown_connections 2023-11-16T16:55:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.183+0000 7f60b75ff700 1 --2- 172.21.15.2:0/2239129953 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60b0105560 0x7f60b010d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.183+0000 7f60b75ff700 1 --2- 172.21.15.2:0/2239129953 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60b0100c50 0x7f60b0101020 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.183+0000 7f60b75ff700 1 --2- 172.21.15.2:0/2239129953 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60b01015f0 0x7f60b0104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.183+0000 7f60b75ff700 1 -- 172.21.15.2:0/2239129953 >> 172.21.15.2:0/2239129953 conn(0x7f60b0077550 msgr2=0x7f60b0077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.183+0000 7f60b75ff700 1 -- 172.21.15.2:0/2239129953 shutdown_connections 2023-11-16T16:55:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.183+0000 7f60b75ff700 1 -- 172.21.15.2:0/2239129953 wait complete. 2023-11-16T16:55:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.185+0000 7f60b75ff700 1 Processor -- start 2023-11-16T16:55:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.185+0000 7f60b75ff700 1 -- start start 2023-11-16T16:55:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.185+0000 7f60b75ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60b0100c50 0x7f60b0198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.185+0000 7f60b75ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60b01015f0 0x7f60b0198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.185+0000 7f60b75ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60b0105560 0x7f60b019cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.186+0000 7f60b75ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b019d610 con 0x7f60b01015f0 2023-11-16T16:55:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.186+0000 7f60b75ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b019d780 con 0x7f60b0100c50 2023-11-16T16:55:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.186+0000 7f60b75ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b019d940 con 0x7f60b0105560 2023-11-16T16:55:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.186+0000 7f60b539b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60b0100c50 0x7f60b0198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.186+0000 7f60b539b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60b0100c50 0x7f60b0198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:33522/0 (socket says 172.21.15.2:33522) 2023-11-16T16:55:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.186+0000 7f60b539b700 1 -- 172.21.15.2:0/2738346060 learned_addr learned my addr 172.21.15.2:0/2738346060 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.186+0000 7f60b4b9a700 1 --2- 172.21.15.2:0/2738346060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60b01015f0 0x7f60b0198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:28.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.186+0000 7f60b539b700 1 -- 172.21.15.2:0/2738346060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60b0105560 msgr2=0x7f60b019cfd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-16T16:55:28.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.186+0000 7f60b539b700 1 --2- 172.21.15.2:0/2738346060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60b0105560 0x7f60b019cfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.186+0000 7f60b539b700 1 -- 172.21.15.2:0/2738346060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60b01015f0 msgr2=0x7f60b0198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:28.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.186+0000 7f60b539b700 1 --2- 172.21.15.2:0/2738346060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60b01015f0 0x7f60b0198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.186+0000 7f60b539b700 1 -- 172.21.15.2:0/2738346060 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac0095f0 con 0x7f60b0100c50 2023-11-16T16:55:28.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.186+0000 7f60b4b9a700 1 --2- 172.21.15.2:0/2738346060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60b01015f0 0x7f60b0198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:55:28.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.186+0000 7f60b539b700 1 --2- 172.21.15.2:0/2738346060 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60b0100c50 0x7f60b0198630 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f60ac005910 tx=0x7f60ac0057d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:28.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.187+0000 7f60a67fc700 1 -- 172.21.15.2:0/2738346060 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac01d070 con 0x7f60b0100c50 2023-11-16T16:55:28.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.187+0000 7f60a67fc700 1 -- 172.21.15.2:0/2738346060 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60ac022470 con 0x7f60b0100c50 2023-11-16T16:55:28.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.187+0000 7f60a67fc700 1 -- 172.21.15.2:0/2738346060 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac017650 con 0x7f60b0100c50 2023-11-16T16:55:28.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.187+0000 7f60b75ff700 1 -- 172.21.15.2:0/2738346060 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b01103b0 con 0x7f60b0100c50 2023-11-16T16:55:28.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.187+0000 7f60b75ff700 1 -- 172.21.15.2:0/2738346060 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b0110220 con 0x7f60b0100c50 2023-11-16T16:55:28.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.192+0000 7f60cafbdb80 1 Processor -- start 2023-11-16T16:55:28.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.192+0000 7f60cafbdb80 1 -- start start 2023-11-16T16:55:28.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.192+0000 7f60a67fc700 1 -- 172.21.15.2:0/2738346060 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f60ac00fb00 con 0x7f60b0100c50 2023-11-16T16:55:28.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.193+0000 7f60cafbdb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649ae66b0 0x559649c901e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.193+0000 7f60cafbdb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649c90720 0x559649c8a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.193+0000 7f60cafbdb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649c90b90 0x559649c8ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.193+0000 7f60cafbdb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559649b90000 con 0x559649c90b90 2023-11-16T16:55:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.193+0000 7f60cafbdb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x559649be2840 con 0x559649ae66b0 2023-11-16T16:55:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.193+0000 7f60cafbdb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x559649bdbcd0 con 0x559649c90720 2023-11-16T16:55:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.193+0000 7f60b5b9c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649ae66b0 0x559649c901e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:28.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.193+0000 7f60b5b9c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649ae66b0 0x559649c901e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:33524/0 (socket says 172.21.15.2:33524) 2023-11-16T16:55:28.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.193+0000 7f60b5b9c700 1 -- 172.21.15.2:0/2951202963 learned_addr learned my addr 172.21.15.2:0/2951202963 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:28.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.193+0000 7f60b539b700 1 --2- 172.21.15.2:0/2951202963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649c90720 0x559649c8a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:28.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.194+0000 7f60b5b9c700 1 -- 172.21.15.2:0/2951202963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649c90720 msgr2=0x559649c8a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.194+0000 7f60b5b9c700 1 --2- 172.21.15.2:0/2951202963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649c90720 0x559649c8a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.194+0000 7f60b5b9c700 1 -- 172.21.15.2:0/2951202963 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649c90b90 msgr2=0x559649c8ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.194+0000 7f60b4b9a700 1 --2- 172.21.15.2:0/2951202963 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649c90b90 0x559649c8ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.194+0000 7f60b5b9c700 1 --2- 172.21.15.2:0/2951202963 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649c90b90 0x559649c8ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.194+0000 7f60b5b9c700 1 -- 172.21.15.2:0/2951202963 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559649a5d5d0 con 0x559649ae66b0 2023-11-16T16:55:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.194+0000 7f60b5b9c700 1 --2- 172.21.15.2:0/2951202963 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649ae66b0 0x559649c901e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f60a8014b10 tx=0x7f60a8009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=904b54f042420bd server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.195+0000 7f609b7fe700 1 -- 172.21.15.2:0/2951202963 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8019070 con 0x559649ae66b0 2023-11-16T16:55:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.195+0000 7f609b7fe700 1 -- 172.21.15.2:0/2951202963 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60a80093a0 con 0x559649ae66b0 2023-11-16T16:55:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.195+0000 7f609b7fe700 1 -- 172.21.15.2:0/2951202963 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a800b700 con 0x559649ae66b0 2023-11-16T16:55:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.195+0000 7f60a67fc700 1 --2- 172.21.15.2:0/2738346060 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f609c0a12c0 0x7f609c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:28.198 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.195+0000 7f60a67fc700 1 -- 172.21.15.2:0/2738346060 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f60ac0ce460 con 0x7f60b0100c50 2023-11-16T16:55:28.198 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.196+0000 7f60cafbdb80 1 -- 172.21.15.2:0/2951202963 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649ae66b0 msgr2=0x559649c901e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.196+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/2951202963 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649ae66b0 0x559649c901e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f60a8014b10 tx=0x7f60a8009920 comp rx=0 tx=0).stop 2023-11-16T16:55:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.196+0000 7f60b539b700 1 --2- 172.21.15.2:0/2738346060 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f609c0a12c0 0x7f609c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.196+0000 7f60cafbdb80 1 -- 172.21.15.2:0/2951202963 shutdown_connections 2023-11-16T16:55:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.196+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/2951202963 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649c90b90 0x559649c8ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.196+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/2951202963 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649ae66b0 0x559649c901e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.196+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/2951202963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649c90720 0x559649c8a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.196+0000 7f60cafbdb80 1 -- 172.21.15.2:0/2951202963 >> 172.21.15.2:0/2951202963 conn(0x559649c7ff50 msgr2=0x559649c85270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.196+0000 7f60cafbdb80 1 -- 172.21.15.2:0/2951202963 shutdown_connections 2023-11-16T16:55:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.196+0000 7f60cafbdb80 1 -- 172.21.15.2:0/2951202963 wait complete. 2023-11-16T16:55:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.196+0000 7f60b539b700 1 --2- 172.21.15.2:0/2738346060 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f609c0a12c0 0x7f609c0a3770 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f60ac0cb8f0 tx=0x7f60ac022c50 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:28.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.197+0000 7f60cafbdb80 1 -- start start 2023-11-16T16:55:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.197+0000 7f60cafbdb80 1 Processor -- start 2023-11-16T16:55:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.198+0000 7f60cafbdb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649ae66b0 0x559649c96af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.198+0000 7f60cafbdb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649c90720 0x559649c93d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.198+0000 7f60cafbdb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649c90b90 0x559649c943f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.198+0000 7f60cafbdb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559649b7d880 con 0x559649ae66b0 2023-11-16T16:55:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.198+0000 7f60cafbdb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x559649b7d520 con 0x559649c90720 2023-11-16T16:55:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.198+0000 7f60cafbdb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x559649a5f8c0 con 0x559649c90b90 2023-11-16T16:55:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.198+0000 7f60cafbdb80 10 client.0 _ll_register_callbacks cb 0x559649b9b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:28.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.198+0000 7f60b5b9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649ae66b0 0x559649c96af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.198+0000 7f60b5b9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649ae66b0 0x559649c96af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57150/0 (socket says 172.21.15.2:57150) 2023-11-16T16:55:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.199+0000 7f60b5b9c700 1 -- 172.21.15.2:0/1955815575 learned_addr learned my addr 172.21.15.2:0/1955815575 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.199+0000 7f60b4b9a700 1 --2- 172.21.15.2:0/1955815575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649c90720 0x559649c93d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.199+0000 7f60b5b9c700 1 --2- 172.21.15.2:0/1955815575 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649c90b90 0x559649c943f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.199+0000 7f60b4b9a700 1 -- 172.21.15.2:0/1955815575 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649c90b90 msgr2=0x559649c943f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.199+0000 7f60b4b9a700 1 --2- 172.21.15.2:0/1955815575 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649c90b90 0x559649c943f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.199+0000 7f60b4b9a700 1 -- 172.21.15.2:0/1955815575 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649ae66b0 msgr2=0x559649c96af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.199+0000 7f60b4b9a700 1 --2- 172.21.15.2:0/1955815575 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649ae66b0 0x559649c96af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.199+0000 7f60b4b9a700 1 -- 172.21.15.2:0/1955815575 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a8014850 con 0x559649c90720 2023-11-16T16:55:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.199+0000 7f60b4b9a700 1 --2- 172.21.15.2:0/1955815575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649c90720 0x559649c93d00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f60b0199c40 tx=0x7f60a0012a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b8bdcb5885f0adf server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:28.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.199+0000 7f6098ff9700 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2023-11-16T16:55:28.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.200+0000 7f6098ff9700 1 -- 172.21.15.2:0/1955815575 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a00096d0 con 0x559649c90720 2023-11-16T16:55:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.200+0000 7f6098ff9700 1 -- 172.21.15.2:0/1955815575 <== mon.2 v2:172.21.15.141:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f60a0004030 con 0x559649c90720 2023-11-16T16:55:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.200+0000 7f60cafbdb80 1 -- 172.21.15.2:0/1955815575 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559649c7d710 con 0x559649c90720 2023-11-16T16:55:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.200+0000 7f6098ff9700 1 -- 172.21.15.2:0/1955815575 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60a0004230 con 0x559649c90720 2023-11-16T16:55:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.200+0000 7f60cafbdb80 10 client.25865 fetch_fsmap learned FSMap version 35 2023-11-16T16:55:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.200+0000 7f60cafbdb80 1 -- 172.21.15.2:0/1955815575 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x559649ba2130 con 0x559649c90720 2023-11-16T16:55:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.201+0000 7f6098ff9700 1 -- 172.21.15.2:0/1955815575 <== mon.2 v2:172.21.15.141:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559649c7d710 con 0x559649c90720 2023-11-16T16:55:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.201+0000 7f6098ff9700 1 -- 172.21.15.2:0/1955815575 <== mon.2 v2:172.21.15.141:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f60a000f740 con 0x559649c90720 2023-11-16T16:55:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.201+0000 7f60cafbdb80 10 client.25865 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:55:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.201+0000 7f60cafbdb80 10 client.25865 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.201+0000 7f60cafbdb80 20 client.25865 populate_metadata read hostname 'smithi002' 2023-11-16T16:55:28.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.201+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/1955815575 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x559649ca4020 0x559649ca6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:28.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.201+0000 7f60cafbdb80 4 client.25865 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:55:28.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.201+0000 7f60cafbdb80 1 -- 172.21.15.2:0/1955815575 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559649b9e040 con 0x559649ca4020 2023-11-16T16:55:28.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.201+0000 7f60b5b9c700 1 --2- 172.21.15.2:0/1955815575 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x559649ca4020 0x559649ca6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:28.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.202+0000 7f60b5b9c700 1 --2- 172.21.15.2:0/1955815575 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x559649ca4020 0x559649ca6430 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1bd8641dc80a0e server_cookie=2832ecd62deaa464 in_seq=0 out_seq=0 2023-11-16T16:55:28.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.202+0000 7f6098ff9700 10 client.25865 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.203+0000 7f6098ff9700 1 -- 172.21.15.2:0/1955815575 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x559649b9e040 con 0x559649ca4020 2023-11-16T16:55:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.203+0000 7f6098ff9700 10 client.25865 handle_command_reply: tid=0 2023-11-16T16:55:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.203+0000 7f60cafbdb80 1 client.25865 shutdown 2023-11-16T16:55:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.203+0000 7f60cafbdb80 1 -- 172.21.15.2:0/1955815575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649c90720 msgr2=0x559649c93d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.203+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/1955815575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649c90720 0x559649c93d00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f60b0199c40 tx=0x7f60a0012a30 comp rx=0 tx=0).stop 2023-11-16T16:55:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.203+0000 7f60cafbdb80 1 -- 172.21.15.2:0/1955815575 shutdown_connections 2023-11-16T16:55:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.203+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/1955815575 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x559649ca4020 0x559649ca6430 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.203+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/1955815575 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649c90b90 0x559649c943f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.203+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/1955815575 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649ae66b0 0x559649c96af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.203+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/1955815575 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649c90720 0x559649c93d00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.203+0000 7f6098ff9700 0 client.25865 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.203+0000 7f60cafbdb80 1 -- 172.21.15.2:0/1955815575 >> 172.21.15.2:0/1955815575 conn(0x559649c7ff50 msgr2=0x559649c847e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.203+0000 7f60cafbdb80 1 -- 172.21.15.2:0/1955815575 shutdown_connections 2023-11-16T16:55:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.203+0000 7f60cafbdb80 1 -- 172.21.15.2:0/1955815575 wait complete. 2023-11-16T16:55:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.203+0000 7f60cafbdb80 20 client.25865 trim_cache size 0 max 16384 2023-11-16T16:55:28.224 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.223+0000 7f60cafbdb80 1 Processor -- start 2023-11-16T16:55:28.224 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.223+0000 7f60cafbdb80 1 -- start start 2023-11-16T16:55:28.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.223+0000 7f60cafbdb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649ae66b0 0x559649c84f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:28.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.223+0000 7f60cafbdb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649c90720 0x559649c935c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:28.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.223+0000 7f60cafbdb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649c90b90 0x559649c93b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:28.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.223+0000 7f60cafbdb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559649ca1700 con 0x559649c90720 2023-11-16T16:55:28.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.223+0000 7f60cafbdb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x559649c94040 con 0x559649c90b90 2023-11-16T16:55:28.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.223+0000 7f60cafbdb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x559649c8ea10 con 0x559649ae66b0 2023-11-16T16:55:28.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.224+0000 7f60b5b9c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649ae66b0 0x559649c84f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:28.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.224+0000 7f60b5b9c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649ae66b0 0x559649c84f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:60938/0 (socket says 172.21.15.2:60938) 2023-11-16T16:55:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.224+0000 7f60b5b9c700 1 -- 172.21.15.2:0/972318076 learned_addr learned my addr 172.21.15.2:0/972318076 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.224+0000 7f60b5b9c700 1 --2- 172.21.15.2:0/972318076 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649c90b90 0x559649c93b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.224+0000 7f60b4b9a700 1 --2- 172.21.15.2:0/972318076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649c90720 0x559649c935c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.224+0000 7f60b5b9c700 1 -- 172.21.15.2:0/972318076 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649c90b90 msgr2=0x559649c93b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.224+0000 7f60b5b9c700 1 --2- 172.21.15.2:0/972318076 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649c90b90 0x559649c93b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.224+0000 7f60b5b9c700 1 -- 172.21.15.2:0/972318076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649c90720 msgr2=0x559649c935c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.224+0000 7f60b5b9c700 1 --2- 172.21.15.2:0/972318076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649c90720 0x559649c935c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.224+0000 7f60b5b9c700 1 -- 172.21.15.2:0/972318076 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559649c816f0 con 0x559649ae66b0 2023-11-16T16:55:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.224+0000 7f60b5b9c700 1 --2- 172.21.15.2:0/972318076 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649ae66b0 0x559649c84f60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f60a8000f80 tx=0x7f60a8006000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bf27b49a930ece6 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.225+0000 7f609bfff700 1 -- 172.21.15.2:0/972318076 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8019050 con 0x559649ae66b0 2023-11-16T16:55:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.225+0000 7f609bfff700 1 -- 172.21.15.2:0/972318076 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60a8023040 con 0x559649ae66b0 2023-11-16T16:55:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.225+0000 7f609bfff700 1 -- 172.21.15.2:0/972318076 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a801e470 con 0x559649ae66b0 2023-11-16T16:55:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.225+0000 7f60cafbdb80 1 -- 172.21.15.2:0/972318076 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649ae66b0 msgr2=0x559649c84f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.225+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/972318076 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649ae66b0 0x559649c84f60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f60a8000f80 tx=0x7f60a8006000 comp rx=0 tx=0).stop 2023-11-16T16:55:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.225+0000 7f60cafbdb80 1 -- 172.21.15.2:0/972318076 shutdown_connections 2023-11-16T16:55:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.225+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/972318076 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649ae66b0 0x559649c84f60 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.225+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/972318076 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649c90b90 0x559649c93b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.225+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/972318076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649c90720 0x559649c935c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.226+0000 7f60cafbdb80 1 -- 172.21.15.2:0/972318076 >> 172.21.15.2:0/972318076 conn(0x559649c7ff50 msgr2=0x559649c8dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.226+0000 7f60cafbdb80 1 -- 172.21.15.2:0/972318076 shutdown_connections 2023-11-16T16:55:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.226+0000 7f60cafbdb80 1 -- 172.21.15.2:0/972318076 wait complete. 2023-11-16T16:55:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.226+0000 7f60cafbdb80 1 -- start start 2023-11-16T16:55:28.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.226+0000 7f60cafbdb80 1 Processor -- start 2023-11-16T16:55:28.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.227+0000 7f60cafbdb80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649ae66b0 0x559649c97620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:28.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.227+0000 7f60cafbdb80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649c90720 0x559649c97b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:28.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.227+0000 7f60cafbdb80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649c90b90 0x559649cf6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:28.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.227+0000 7f60cafbdb80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559649c98450 con 0x559649ae66b0 2023-11-16T16:55:28.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.227+0000 7f60cafbdb80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x559649cf73e0 con 0x559649c90b90 2023-11-16T16:55:28.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.227+0000 7f60cafbdb80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x559649cf7520 con 0x559649c90720 2023-11-16T16:55:28.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.227+0000 7f60cafbdb80 10 client.0 _ll_register_callbacks cb 0x7f60a0009bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.227+0000 7f60b5b9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649ae66b0 0x559649c97620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.227+0000 7f60b5b9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649ae66b0 0x559649c97620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57162/0 (socket says 172.21.15.2:57162) 2023-11-16T16:55:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.227+0000 7f60b5b9c700 1 -- 172.21.15.2:0/3823328908 learned_addr learned my addr 172.21.15.2:0/3823328908 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.228+0000 7f60b5b9c700 1 --2- 172.21.15.2:0/3823328908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649c90b90 0x559649cf6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.228+0000 7f60b4b9a700 1 --2- 172.21.15.2:0/3823328908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649c90720 0x559649c97b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.228+0000 7f60b5b9c700 1 -- 172.21.15.2:0/3823328908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649c90720 msgr2=0x559649c97b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.228+0000 7f60b5b9c700 1 --2- 172.21.15.2:0/3823328908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649c90720 0x559649c97b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.228+0000 7f60b5b9c700 1 -- 172.21.15.2:0/3823328908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649c90b90 msgr2=0x559649cf6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.228+0000 7f60b5b9c700 1 --2- 172.21.15.2:0/3823328908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649c90b90 0x559649cf6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.228+0000 7f60b5b9c700 1 -- 172.21.15.2:0/3823328908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a8025040 con 0x559649ae66b0 2023-11-16T16:55:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.228+0000 7f60b5b9c700 1 --2- 172.21.15.2:0/3823328908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649ae66b0 0x559649c97620 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f60a8014b10 tx=0x7f60a8004f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b904943fd3326579 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.228+0000 7f6098ff9700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:55:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.229+0000 7f6098ff9700 1 -- 172.21.15.2:0/3823328908 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8019030 con 0x559649ae66b0 2023-11-16T16:55:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.229+0000 7f6098ff9700 1 -- 172.21.15.2:0/3823328908 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f60a802eb30 con 0x559649ae66b0 2023-11-16T16:55:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.229+0000 7f60cafbdb80 1 -- 172.21.15.2:0/3823328908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f60a80093a0 con 0x559649ae66b0 2023-11-16T16:55:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.229+0000 7f6098ff9700 1 -- 172.21.15.2:0/3823328908 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60a8035040 con 0x559649ae66b0 2023-11-16T16:55:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.229+0000 7f60cafbdb80 10 client.15822 fetch_fsmap learned FSMap version 35 2023-11-16T16:55:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.229+0000 7f60cafbdb80 1 -- 172.21.15.2:0/3823328908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x559649a09d80 con 0x559649ae66b0 2023-11-16T16:55:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.229+0000 7f6098ff9700 1 -- 172.21.15.2:0/3823328908 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60a80093a0 con 0x559649ae66b0 2023-11-16T16:55:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.229+0000 7f6098ff9700 1 -- 172.21.15.2:0/3823328908 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f60a800b0b0 con 0x559649ae66b0 2023-11-16T16:55:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.230+0000 7f60cafbdb80 10 client.15822 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:55:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.230+0000 7f60cafbdb80 10 client.15822 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.230+0000 7f60cafbdb80 20 client.15822 populate_metadata read hostname 'smithi002' 2023-11-16T16:55:28.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.230+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/3823328908 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x559649cf8860 0x559649cfac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:28.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.230+0000 7f60cafbdb80 4 client.15822 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:55:28.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.230+0000 7f60cafbdb80 1 -- 172.21.15.2:0/3823328908 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559649baa600 con 0x559649cf8860 2023-11-16T16:55:28.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.230+0000 7f60b4b9a700 1 --2- 172.21.15.2:0/3823328908 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x559649cf8860 0x559649cfac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:28.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.231+0000 7f60b4b9a700 1 --2- 172.21.15.2:0/3823328908 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x559649cf8860 0x559649cfac70 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9837c15e883ebd7 server_cookie=565c16f5165f6100 in_seq=0 out_seq=0 2023-11-16T16:55:28.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:28.231+0000 7f6098ff9700 10 client.15822 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:29.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:28 smithi155 ceph-mon[131607]: pgmap v170: 129 pgs: 129 active+clean; 574 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.3 MiB/s wr, 618 op/s 2023-11-16T16:55:29.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:28 smithi002 ceph-mon[128712]: pgmap v170: 129 pgs: 129 active+clean; 574 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.3 MiB/s wr, 618 op/s 2023-11-16T16:55:29.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:28 smithi141 ceph-mon[131184]: pgmap v170: 129 pgs: 129 active+clean; 574 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.3 MiB/s wr, 618 op/s 2023-11-16T16:55:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.885+0000 7f6098ff9700 1 -- 172.21.15.2:0/3823328908 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559649baa600 con 0x559649cf8860 2023-11-16T16:55:29.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.885+0000 7f6098ff9700 10 client.15822 handle_command_reply: tid=0 2023-11-16T16:55:29.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.885+0000 7f60cafbdb80 1 client.15822 shutdown 2023-11-16T16:55:29.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.885+0000 7f60cafbdb80 1 -- 172.21.15.2:0/3823328908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649ae66b0 msgr2=0x559649c97620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:29.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.885+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/3823328908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649ae66b0 0x559649c97620 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f60a8014b10 tx=0x7f60a8004f10 comp rx=0 tx=0).stop 2023-11-16T16:55:29.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.886+0000 7f60cafbdb80 1 -- 172.21.15.2:0/3823328908 shutdown_connections 2023-11-16T16:55:29.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.886+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/3823328908 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x559649c90b90 0x559649cf6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:29.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.886+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/3823328908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559649ae66b0 0x559649c97620 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:29.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.886+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/3823328908 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x559649cf8860 0x559649cfac70 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:29.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.886+0000 7f60cafbdb80 1 --2- 172.21.15.2:0/3823328908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559649c90720 0x559649c97b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:29.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.886+0000 7f6098ff9700 0 client.15822 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.886+0000 7f60cafbdb80 1 -- 172.21.15.2:0/3823328908 >> 172.21.15.2:0/3823328908 conn(0x559649c7ff50 msgr2=0x559649c8ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.888+0000 7f60cafbdb80 1 -- 172.21.15.2:0/3823328908 shutdown_connections 2023-11-16T16:55:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.888+0000 7f60cafbdb80 1 -- 172.21.15.2:0/3823328908 wait complete. 2023-11-16T16:55:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.889+0000 7f60cafbdb80 20 client.15822 trim_cache size 0 max 16384 2023-11-16T16:55:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:55:29.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:55:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:55:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:55:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.889+0000 7f60b75ff700 1 -- 172.21.15.2:0/2738346060 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f609c0a12c0 msgr2=0x7f609c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.889+0000 7f60b75ff700 1 --2- 172.21.15.2:0/2738346060 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f609c0a12c0 0x7f609c0a3770 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f60ac0cb8f0 tx=0x7f60ac022c50 comp rx=0 tx=0).stop 2023-11-16T16:55:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.889+0000 7f60b75ff700 1 -- 172.21.15.2:0/2738346060 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60b0100c50 msgr2=0x7f60b0198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.889+0000 7f60b75ff700 1 --2- 172.21.15.2:0/2738346060 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60b0100c50 0x7f60b0198630 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f60ac005910 tx=0x7f60ac0057d0 comp rx=0 tx=0).stop 2023-11-16T16:55:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.890+0000 7f60b75ff700 1 -- 172.21.15.2:0/2738346060 shutdown_connections 2023-11-16T16:55:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.890+0000 7f60b75ff700 1 --2- 172.21.15.2:0/2738346060 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f609c0a12c0 0x7f609c0a3770 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.890+0000 7f60b75ff700 1 --2- 172.21.15.2:0/2738346060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60b0105560 0x7f60b019cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.890+0000 7f60b75ff700 1 --2- 172.21.15.2:0/2738346060 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f60b0100c50 0x7f60b0198630 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.890+0000 7f60b75ff700 1 --2- 172.21.15.2:0/2738346060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60b01015f0 0x7f60b0198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.890+0000 7f60b75ff700 1 -- 172.21.15.2:0/2738346060 >> 172.21.15.2:0/2738346060 conn(0x7f60b0077550 msgr2=0x7f60b00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.890+0000 7f60b75ff700 1 -- 172.21.15.2:0/2738346060 shutdown_connections 2023-11-16T16:55:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:29.890+0000 7f60b75ff700 1 -- 172.21.15.2:0/2738346060 wait complete. 2023-11-16T16:55:31.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:30 smithi155 ceph-mon[131607]: pgmap v171: 129 pgs: 129 active+clean; 574 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.5 MiB/s wr, 217 op/s 2023-11-16T16:55:31.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:30 smithi002 ceph-mon[128712]: pgmap v171: 129 pgs: 129 active+clean; 574 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.5 MiB/s wr, 217 op/s 2023-11-16T16:55:31.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:30 smithi141 ceph-mon[131184]: pgmap v171: 129 pgs: 129 active+clean; 574 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.5 MiB/s wr, 217 op/s 2023-11-16T16:55:33.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:32 smithi155 ceph-mon[131607]: pgmap v172: 129 pgs: 129 active+clean; 582 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.9 MiB/s wr, 501 op/s 2023-11-16T16:55:33.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:32 smithi002 ceph-mon[128712]: pgmap v172: 129 pgs: 129 active+clean; 582 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.9 MiB/s wr, 501 op/s 2023-11-16T16:55:33.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:32 smithi141 ceph-mon[131184]: pgmap v172: 129 pgs: 129 active+clean; 582 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.9 MiB/s wr, 501 op/s 2023-11-16T16:55:35.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:34 smithi155 ceph-mon[131607]: pgmap v173: 129 pgs: 129 active+clean; 582 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 390 op/s 2023-11-16T16:55:35.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:34 smithi002 ceph-mon[128712]: pgmap v173: 129 pgs: 129 active+clean; 582 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 390 op/s 2023-11-16T16:55:35.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:34 smithi141 ceph-mon[131184]: pgmap v173: 129 pgs: 129 active+clean; 582 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 390 op/s 2023-11-16T16:55:37.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:36 smithi155 ceph-mon[131607]: pgmap v174: 129 pgs: 129 active+clean; 582 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 390 op/s 2023-11-16T16:55:37.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:36 smithi002 ceph-mon[128712]: pgmap v174: 129 pgs: 129 active+clean; 582 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 390 op/s 2023-11-16T16:55:37.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:36 smithi141 ceph-mon[131184]: pgmap v174: 129 pgs: 129 active+clean; 582 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 390 op/s 2023-11-16T16:55:39.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:38 smithi155 ceph-mon[131607]: pgmap v175: 129 pgs: 129 active+clean; 598 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.6 MiB/s wr, 567 op/s 2023-11-16T16:55:39.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:38 smithi002 ceph-mon[128712]: pgmap v175: 129 pgs: 129 active+clean; 598 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.6 MiB/s wr, 567 op/s 2023-11-16T16:55:39.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:38 smithi141 ceph-mon[131184]: pgmap v175: 129 pgs: 129 active+clean; 598 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.6 MiB/s wr, 567 op/s 2023-11-16T16:55:40.020 INFO:tasks.workunit.client.0.smithi002.stdout:cleaned up hierarchy 2023-11-16T16:55:41.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:40 smithi155 ceph-mon[131607]: pgmap v176: 129 pgs: 129 active+clean; 598 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 460 op/s 2023-11-16T16:55:41.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:40 smithi002 ceph-mon[128712]: pgmap v176: 129 pgs: 129 active+clean; 598 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 460 op/s 2023-11-16T16:55:41.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:40 smithi141 ceph-mon[131184]: pgmap v176: 129 pgs: 129 active+clean; 598 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 460 op/s 2023-11-16T16:55:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.056+0000 7f741ebff700 1 -- 172.21.15.2:0/3064638183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7418100c50 msgr2=0x7f7418101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.056+0000 7f741ebff700 1 --2- 172.21.15.2:0/3064638183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7418100c50 0x7f7418101020 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f740c0099c0 tx=0x7f740c009cd0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.056+0000 7f741ebff700 1 -- 172.21.15.2:0/3064638183 shutdown_connections 2023-11-16T16:55:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.056+0000 7f741ebff700 1 --2- 172.21.15.2:0/3064638183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7418100c50 0x7f7418101020 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.056+0000 7f741ebff700 1 --2- 172.21.15.2:0/3064638183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7418105560 0x7f741810d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.056+0000 7f741ebff700 1 --2- 172.21.15.2:0/3064638183 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f74181015f0 0x7f7418104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.056+0000 7f741ebff700 1 -- 172.21.15.2:0/3064638183 >> 172.21.15.2:0/3064638183 conn(0x7f7418077550 msgr2=0x7f7418077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.056+0000 7f741ebff700 1 -- 172.21.15.2:0/3064638183 shutdown_connections 2023-11-16T16:55:43.059 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.056+0000 7f741ebff700 1 -- 172.21.15.2:0/3064638183 wait complete. 2023-11-16T16:55:43.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.057+0000 7f741ebff700 1 Processor -- start 2023-11-16T16:55:43.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.057+0000 7f741ebff700 1 -- start start 2023-11-16T16:55:43.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.058+0000 7f741ebff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7418100c50 0x7f7418198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:43.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.058+0000 7f741ebff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74181015f0 0x7f7418198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:43.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.058+0000 7f741ebff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7418105560 0x7f741819d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:43.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.058+0000 7f741ebff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741819d650 con 0x7f74181015f0 2023-11-16T16:55:43.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.058+0000 7f741ebff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741819d7c0 con 0x7f7418105560 2023-11-16T16:55:43.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.058+0000 7f741ebff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741819d980 con 0x7f7418100c50 2023-11-16T16:55:43.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.058+0000 7f741c99b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7418100c50 0x7f7418198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:43.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.058+0000 7f7417fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74181015f0 0x7f7418198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:43.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.058+0000 7f741c99b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7418100c50 0x7f7418198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:40774/0 (socket says 172.21.15.2:40774) 2023-11-16T16:55:43.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.058+0000 7f741c99b700 1 -- 172.21.15.2:0/736235500 learned_addr learned my addr 172.21.15.2:0/736235500 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:43.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.058+0000 7f741d19c700 1 --2- 172.21.15.2:0/736235500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7418105560 0x7f741819d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:43.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.058+0000 7f741c99b700 1 -- 172.21.15.2:0/736235500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7418105560 msgr2=0x7f741819d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:43.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.058+0000 7f741c99b700 1 --2- 172.21.15.2:0/736235500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7418105560 0x7f741819d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.058+0000 7f741c99b700 1 -- 172.21.15.2:0/736235500 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74181015f0 msgr2=0x7f7418198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:43.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.058+0000 7f741c99b700 1 --2- 172.21.15.2:0/736235500 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74181015f0 0x7f7418198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.058+0000 7f741c99b700 1 -- 172.21.15.2:0/736235500 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7410009c40 con 0x7f7418100c50 2023-11-16T16:55:43.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.058+0000 7f741d19c700 1 --2- 172.21.15.2:0/736235500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7418105560 0x7f741819d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:55:43.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.058+0000 7f7417fff700 1 --2- 172.21.15.2:0/736235500 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74181015f0 0x7f7418198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:55:43.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.059+0000 7f741c99b700 1 --2- 172.21.15.2:0/736235500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7418100c50 0x7f7418198670 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f740c00b580 tx=0x7f740c00bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:43.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.059+0000 7f7415ffb700 1 -- 172.21.15.2:0/736235500 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740c01d070 con 0x7f7418100c50 2023-11-16T16:55:43.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.059+0000 7f7415ffb700 1 -- 172.21.15.2:0/736235500 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f740c005050 con 0x7f7418100c50 2023-11-16T16:55:43.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.059+0000 7f7415ffb700 1 -- 172.21.15.2:0/736235500 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740c004a40 con 0x7f7418100c50 2023-11-16T16:55:43.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.059+0000 7f741ebff700 1 -- 172.21.15.2:0/736235500 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f740c0095f0 con 0x7f7418100c50 2023-11-16T16:55:43.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.059+0000 7f741ebff700 1 -- 172.21.15.2:0/736235500 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74181103b0 con 0x7f7418100c50 2023-11-16T16:55:43.064 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.062+0000 7f7415ffb700 1 -- 172.21.15.2:0/736235500 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f740c0051c0 con 0x7f7418100c50 2023-11-16T16:55:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.064+0000 7f7415ffb700 1 --2- 172.21.15.2:0/736235500 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f74080a1340 0x7f74080a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:43.065 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.064+0000 7f7432489b80 1 Processor -- start 2023-11-16T16:55:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.064+0000 7f7415ffb700 1 -- 172.21.15.2:0/736235500 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f740c0cdea0 con 0x7f7418100c50 2023-11-16T16:55:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.064+0000 7f7417fff700 1 --2- 172.21.15.2:0/736235500 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f74080a1340 0x7f74080a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:43.066 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.064+0000 7f7432489b80 1 -- start start 2023-11-16T16:55:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.064+0000 7f7432489b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f66c86b0 0x5560f68721e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:43.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.064+0000 7f7432489b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f6872720 0x5560f686c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:43.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.065+0000 7f7432489b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f6872b90 0x5560f686cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:43.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.065+0000 7f7432489b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f6772000 con 0x5560f66c86b0 2023-11-16T16:55:43.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.065+0000 7f7432489b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f67c4840 con 0x5560f6872720 2023-11-16T16:55:43.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.065+0000 7f7432489b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f67bdcd0 con 0x5560f6872b90 2023-11-16T16:55:43.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.065+0000 7f7417fff700 1 --2- 172.21.15.2:0/736235500 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f74080a1340 0x7f74080a37f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7418199c80 tx=0x7f7404009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:43.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.065+0000 7f741c99b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f66c86b0 0x5560f68721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:43.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.065+0000 7f741c99b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f66c86b0 0x5560f68721e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44758/0 (socket says 172.21.15.2:44758) 2023-11-16T16:55:43.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.065+0000 7f741c99b700 1 -- 172.21.15.2:0/3238502310 learned_addr learned my addr 172.21.15.2:0/3238502310 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:43.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.065+0000 7f741d19c700 1 --2- 172.21.15.2:0/3238502310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f6872b90 0x5560f686cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:43.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.065+0000 7f7417fff700 1 --2- 172.21.15.2:0/3238502310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f6872720 0x5560f686c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:43.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.065+0000 7f741c99b700 1 -- 172.21.15.2:0/3238502310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f6872b90 msgr2=0x5560f686cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:43.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.065+0000 7f741c99b700 1 --2- 172.21.15.2:0/3238502310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f6872b90 0x5560f686cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.065+0000 7f741c99b700 1 -- 172.21.15.2:0/3238502310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f6872720 msgr2=0x5560f686c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:43.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.065+0000 7f741c99b700 1 --2- 172.21.15.2:0/3238502310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f6872720 0x5560f686c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.065+0000 7f741c99b700 1 -- 172.21.15.2:0/3238502310 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560f663f5d0 con 0x5560f66c86b0 2023-11-16T16:55:43.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.065+0000 7f741c99b700 1 --2- 172.21.15.2:0/3238502310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f66c86b0 0x5560f68721e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f740c0976a0 tx=0x7f740c09ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acfd23590a45815c server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:43.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.066+0000 7f7402ffd700 1 -- 172.21.15.2:0/3238502310 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740c09e070 con 0x5560f66c86b0 2023-11-16T16:55:43.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.066+0000 7f7402ffd700 1 -- 172.21.15.2:0/3238502310 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f740c0a2ff0 con 0x5560f66c86b0 2023-11-16T16:55:43.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.066+0000 7f7402ffd700 1 -- 172.21.15.2:0/3238502310 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740c09ae90 con 0x5560f66c86b0 2023-11-16T16:55:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.066+0000 7f7432489b80 1 -- 172.21.15.2:0/3238502310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f66c86b0 msgr2=0x5560f68721e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.066+0000 7f7432489b80 1 --2- 172.21.15.2:0/3238502310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f66c86b0 0x5560f68721e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f740c0976a0 tx=0x7f740c09ad20 comp rx=0 tx=0).stop 2023-11-16T16:55:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.067+0000 7f7432489b80 1 -- 172.21.15.2:0/3238502310 shutdown_connections 2023-11-16T16:55:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.067+0000 7f7432489b80 1 --2- 172.21.15.2:0/3238502310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f6872b90 0x5560f686cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.067+0000 7f7432489b80 1 --2- 172.21.15.2:0/3238502310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f66c86b0 0x5560f68721e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.067+0000 7f7432489b80 1 --2- 172.21.15.2:0/3238502310 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f6872720 0x5560f686c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.067+0000 7f7432489b80 1 -- 172.21.15.2:0/3238502310 >> 172.21.15.2:0/3238502310 conn(0x5560f6861f50 msgr2=0x5560f6867270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.067+0000 7f7432489b80 1 -- 172.21.15.2:0/3238502310 shutdown_connections 2023-11-16T16:55:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.067+0000 7f7432489b80 1 -- 172.21.15.2:0/3238502310 wait complete. 2023-11-16T16:55:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.067+0000 7f7432489b80 1 -- start start 2023-11-16T16:55:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.067+0000 7f7432489b80 1 Processor -- start 2023-11-16T16:55:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.068+0000 7f7432489b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f66c86b0 0x5560f687b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.068+0000 7f7432489b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f6872720 0x5560f687bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.068+0000 7f7432489b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f6872b90 0x5560f6875a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.068+0000 7f7432489b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f675f880 con 0x5560f66c86b0 2023-11-16T16:55:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.068+0000 7f7432489b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f675f520 con 0x5560f6872b90 2023-11-16T16:55:43.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.068+0000 7f7432489b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f66418c0 con 0x5560f6872720 2023-11-16T16:55:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.068+0000 7f7432489b80 10 client.0 _ll_register_callbacks cb 0x5560f677d720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.068+0000 7f7417fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f6872720 0x5560f687bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.068+0000 7f7417fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f6872720 0x5560f687bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:40786/0 (socket says 172.21.15.2:40786) 2023-11-16T16:55:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.068+0000 7f7417fff700 1 -- 172.21.15.2:0/2276590121 learned_addr learned my addr 172.21.15.2:0/2276590121 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.068+0000 7f741c99b700 1 --2- 172.21.15.2:0/2276590121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f66c86b0 0x5560f687b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.069+0000 7f741d19c700 1 --2- 172.21.15.2:0/2276590121 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f6872b90 0x5560f6875a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.069+0000 7f7417fff700 1 -- 172.21.15.2:0/2276590121 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f6872b90 msgr2=0x5560f6875a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.069+0000 7f7417fff700 1 --2- 172.21.15.2:0/2276590121 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f6872b90 0x5560f6875a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.069+0000 7f7417fff700 1 -- 172.21.15.2:0/2276590121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f66c86b0 msgr2=0x5560f687b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.069+0000 7f7417fff700 1 --2- 172.21.15.2:0/2276590121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f66c86b0 0x5560f687b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.069+0000 7f7417fff700 1 -- 172.21.15.2:0/2276590121 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74181103b0 con 0x5560f6872720 2023-11-16T16:55:43.076 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.069+0000 7f741c99b700 1 --2- 172.21.15.2:0/2276590121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f66c86b0 0x5560f687b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:55:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.069+0000 7f7417fff700 1 --2- 172.21.15.2:0/2276590121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f6872720 0x5560f687bd40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f74040038c0 tx=0x7f74040047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1a69f231d9bac88 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.069+0000 7f73f7fff700 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2023-11-16T16:55:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.069+0000 7f73f7fff700 1 -- 172.21.15.2:0/2276590121 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740401fa10 con 0x5560f6872720 2023-11-16T16:55:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.069+0000 7f73f7fff700 1 -- 172.21.15.2:0/2276590121 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f740401fb70 con 0x5560f6872720 2023-11-16T16:55:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.069+0000 7f7432489b80 1 -- 172.21.15.2:0/2276590121 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5560f685f710 con 0x5560f6872720 2023-11-16T16:55:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.070+0000 7f73f7fff700 1 -- 172.21.15.2:0/2276590121 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f740401fd70 con 0x5560f6872720 2023-11-16T16:55:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.070+0000 7f7432489b80 10 client.25861 fetch_fsmap learned FSMap version 35 2023-11-16T16:55:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.070+0000 7f7432489b80 1 -- 172.21.15.2:0/2276590121 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x5560f6784130 con 0x5560f6872720 2023-11-16T16:55:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.070+0000 7f73f7fff700 1 -- 172.21.15.2:0/2276590121 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5560f685f710 con 0x5560f6872720 2023-11-16T16:55:43.077 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.070+0000 7f73f7fff700 1 -- 172.21.15.2:0/2276590121 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f7404027400 con 0x5560f6872720 2023-11-16T16:55:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.070+0000 7f7432489b80 10 client.25861 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:55:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.070+0000 7f7432489b80 10 client.25861 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.070+0000 7f7432489b80 20 client.25861 populate_metadata read hostname 'smithi002' 2023-11-16T16:55:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.070+0000 7f7432489b80 1 --2- 172.21.15.2:0/2276590121 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5560f6885d90 0x5560f68881a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.070+0000 7f7432489b80 4 client.25861 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:55:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.070+0000 7f7432489b80 1 -- 172.21.15.2:0/2276590121 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5560f6780040 con 0x5560f6885d90 2023-11-16T16:55:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.071+0000 7f741c99b700 1 --2- 172.21.15.2:0/2276590121 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5560f6885d90 0x5560f68881a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.071+0000 7f741c99b700 1 --2- 172.21.15.2:0/2276590121 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5560f6885d90 0x5560f68881a0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0868bd6c076e7f2 server_cookie=5f4f68dcaf9ecf3f in_seq=0 out_seq=0 2023-11-16T16:55:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.071+0000 7f73f7fff700 10 client.25861 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.072+0000 7f73f7fff700 1 -- 172.21.15.2:0/2276590121 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5560f6780040 con 0x5560f6885d90 2023-11-16T16:55:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.072+0000 7f73f7fff700 10 client.25861 handle_command_reply: tid=0 2023-11-16T16:55:43.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.072+0000 7f7432489b80 1 client.25861 shutdown 2023-11-16T16:55:43.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.072+0000 7f7432489b80 1 -- 172.21.15.2:0/2276590121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f6872720 msgr2=0x5560f687bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:43.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.072+0000 7f7432489b80 1 --2- 172.21.15.2:0/2276590121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f6872720 0x5560f687bd40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f74040038c0 tx=0x7f74040047c0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.072+0000 7f7432489b80 1 -- 172.21.15.2:0/2276590121 shutdown_connections 2023-11-16T16:55:43.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.072+0000 7f7432489b80 1 --2- 172.21.15.2:0/2276590121 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f6872b90 0x5560f6875a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.072+0000 7f7432489b80 1 --2- 172.21.15.2:0/2276590121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f66c86b0 0x5560f687b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.072+0000 7f7432489b80 1 --2- 172.21.15.2:0/2276590121 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5560f6885d90 0x5560f68881a0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.072+0000 7f7432489b80 1 --2- 172.21.15.2:0/2276590121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f6872720 0x5560f687bd40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.072+0000 7f7432489b80 1 -- 172.21.15.2:0/2276590121 >> 172.21.15.2:0/2276590121 conn(0x5560f6861f50 msgr2=0x5560f68667e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:43.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.072+0000 7f73f7fff700 0 client.25861 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:43.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.072+0000 7f7432489b80 1 -- 172.21.15.2:0/2276590121 shutdown_connections 2023-11-16T16:55:43.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.072+0000 7f7432489b80 1 -- 172.21.15.2:0/2276590121 wait complete. 2023-11-16T16:55:43.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.072+0000 7f7432489b80 20 client.25861 trim_cache size 0 max 16384 2023-11-16T16:55:43.090 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.088+0000 7f7432489b80 1 Processor -- start 2023-11-16T16:55:43.090 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.089+0000 7f7432489b80 1 -- start start 2023-11-16T16:55:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.089+0000 7f7432489b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f66c86b0 0x5560f6866f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.089+0000 7f7432489b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f6872720 0x5560f6872f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.089+0000 7f7432489b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f6872b90 0x5560f68734d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.089+0000 7f7432489b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f6868300 con 0x5560f6872b90 2023-11-16T16:55:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.089+0000 7f7432489b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f6861b70 con 0x5560f6872720 2023-11-16T16:55:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.089+0000 7f7432489b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f6883360 con 0x5560f66c86b0 2023-11-16T16:55:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.089+0000 7f7417fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f6872720 0x5560f6872f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.089+0000 7f7417fff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f6872720 0x5560f6872f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:56486/0 (socket says 172.21.15.2:56486) 2023-11-16T16:55:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.089+0000 7f7417fff700 1 -- 172.21.15.2:0/3377081548 learned_addr learned my addr 172.21.15.2:0/3377081548 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.089+0000 7f741c99b700 1 --2- 172.21.15.2:0/3377081548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f66c86b0 0x5560f6866f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:43.093 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.089+0000 7f741d19c700 1 --2- 172.21.15.2:0/3377081548 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f6872b90 0x5560f68734d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:43.093 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.089+0000 7f7417fff700 1 -- 172.21.15.2:0/3377081548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f66c86b0 msgr2=0x5560f6866f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:43.093 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.089+0000 7f7417fff700 1 --2- 172.21.15.2:0/3377081548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f66c86b0 0x5560f6866f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.093 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.089+0000 7f7417fff700 1 -- 172.21.15.2:0/3377081548 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f6872b90 msgr2=0x5560f68734d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:43.093 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.090+0000 7f7417fff700 1 --2- 172.21.15.2:0/3377081548 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f6872b90 0x5560f68734d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.093 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.090+0000 7f7417fff700 1 -- 172.21.15.2:0/3377081548 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560f68636f0 con 0x5560f6872720 2023-11-16T16:55:43.093 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.090+0000 7f741d19c700 1 --2- 172.21.15.2:0/3377081548 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f6872b90 0x5560f68734d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:55:43.093 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.090+0000 7f7417fff700 1 --2- 172.21.15.2:0/3377081548 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f6872720 0x5560f6872f90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7404000c00 tx=0x7f7404005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=138382bcefead74d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:43.093 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.090+0000 7f74037fe700 1 -- 172.21.15.2:0/3377081548 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74040279a0 con 0x5560f6872720 2023-11-16T16:55:43.093 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.090+0000 7f74037fe700 1 -- 172.21.15.2:0/3377081548 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f740401f6e0 con 0x5560f6872720 2023-11-16T16:55:43.093 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.090+0000 7f74037fe700 1 -- 172.21.15.2:0/3377081548 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74040366a0 con 0x5560f6872720 2023-11-16T16:55:43.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.091+0000 7f7432489b80 1 -- 172.21.15.2:0/3377081548 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f6872720 msgr2=0x5560f6872f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.091+0000 7f7432489b80 1 --2- 172.21.15.2:0/3377081548 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f6872720 0x5560f6872f90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7404000c00 tx=0x7f7404005880 comp rx=0 tx=0).stop 2023-11-16T16:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.091+0000 7f7432489b80 1 -- 172.21.15.2:0/3377081548 shutdown_connections 2023-11-16T16:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.091+0000 7f7432489b80 1 --2- 172.21.15.2:0/3377081548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f66c86b0 0x5560f6866f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.091+0000 7f7432489b80 1 --2- 172.21.15.2:0/3377081548 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f6872b90 0x5560f68734d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.091+0000 7f7432489b80 1 --2- 172.21.15.2:0/3377081548 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f6872720 0x5560f6872f90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.091+0000 7f7432489b80 1 -- 172.21.15.2:0/3377081548 >> 172.21.15.2:0/3377081548 conn(0x5560f6861f50 msgr2=0x5560f6876320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.091+0000 7f7432489b80 1 -- 172.21.15.2:0/3377081548 shutdown_connections 2023-11-16T16:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.091+0000 7f7432489b80 1 -- 172.21.15.2:0/3377081548 wait complete. 2023-11-16T16:55:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.091+0000 7f7432489b80 1 -- start start 2023-11-16T16:55:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.091+0000 7f7432489b80 1 Processor -- start 2023-11-16T16:55:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.092+0000 7f7432489b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f66c86b0 0x5560f6879410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.092+0000 7f7432489b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f6872720 0x5560f6879950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.092+0000 7f7432489b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f6872b90 0x5560f6879eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.092+0000 7f7432489b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f68d8dc0 con 0x5560f6872720 2023-11-16T16:55:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.092+0000 7f7432489b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f68d8f30 con 0x5560f66c86b0 2023-11-16T16:55:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.092+0000 7f7432489b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f68d90f0 con 0x5560f6872b90 2023-11-16T16:55:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.092+0000 7f7432489b80 10 client.0 _ll_register_callbacks cb 0x7f7404003680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.092+0000 7f7417fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f6872720 0x5560f6879950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.092+0000 7f741c99b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f66c86b0 0x5560f6879410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.092+0000 7f741c99b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f66c86b0 0x5560f6879410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:56498/0 (socket says 172.21.15.2:56498) 2023-11-16T16:55:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.092+0000 7f741c99b700 1 -- 172.21.15.2:0/2336843827 learned_addr learned my addr 172.21.15.2:0/2336843827 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.092+0000 7f741d19c700 1 --2- 172.21.15.2:0/2336843827 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f6872b90 0x5560f6879eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.093+0000 7f741c99b700 1 -- 172.21.15.2:0/2336843827 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f6872b90 msgr2=0x5560f6879eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.093+0000 7f741c99b700 1 --2- 172.21.15.2:0/2336843827 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f6872b90 0x5560f6879eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.093+0000 7f741c99b700 1 -- 172.21.15.2:0/2336843827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f6872720 msgr2=0x5560f6879950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.093+0000 7f741c99b700 1 --2- 172.21.15.2:0/2336843827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f6872720 0x5560f6879950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.093+0000 7f741c99b700 1 -- 172.21.15.2:0/2336843827 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5560f6784130 con 0x5560f66c86b0 2023-11-16T16:55:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.093+0000 7f7417fff700 1 --2- 172.21.15.2:0/2336843827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f6872720 0x5560f6879950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:55:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.093+0000 7f741d19c700 1 --2- 172.21.15.2:0/2336843827 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f6872b90 0x5560f6879eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:55:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.093+0000 7f741c99b700 1 --2- 172.21.15.2:0/2336843827 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f66c86b0 0x5560f6879410 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f740c09abc0 tx=0x7f740c09ad90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b93cf6dbbdf4bacb server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.093+0000 7f73f7fff700 10 client.0 ms_handle_connect on v2:172.21.15.141:3300/0 2023-11-16T16:55:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.093+0000 7f73f7fff700 1 -- 172.21.15.2:0/2336843827 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740c09e040 con 0x5560f66c86b0 2023-11-16T16:55:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.093+0000 7f73f7fff700 1 -- 172.21.15.2:0/2336843827 <== mon.2 v2:172.21.15.141:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f740c0b3140 con 0x5560f66c86b0 2023-11-16T16:55:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.093+0000 7f7432489b80 1 -- 172.21.15.2:0/2336843827 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f740c0a2ff0 con 0x5560f66c86b0 2023-11-16T16:55:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.093+0000 7f73f7fff700 1 -- 172.21.15.2:0/2336843827 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f740c0b3300 con 0x5560f66c86b0 2023-11-16T16:55:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.093+0000 7f7432489b80 10 client.25889 fetch_fsmap learned FSMap version 35 2023-11-16T16:55:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.093+0000 7f7432489b80 1 -- 172.21.15.2:0/2336843827 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x5560f65ebd80 con 0x5560f66c86b0 2023-11-16T16:55:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.093+0000 7f73f7fff700 1 -- 172.21.15.2:0/2336843827 <== mon.2 v2:172.21.15.141:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f740c07d080 con 0x5560f66c86b0 2023-11-16T16:55:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.094+0000 7f73f7fff700 1 -- 172.21.15.2:0/2336843827 <== mon.2 v2:172.21.15.141:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f740c09a240 con 0x5560f66c86b0 2023-11-16T16:55:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.094+0000 7f7432489b80 10 client.25889 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:55:43.100 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.094+0000 7f7432489b80 10 client.25889 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.094+0000 7f7432489b80 20 client.25889 populate_metadata read hostname 'smithi002' 2023-11-16T16:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.094+0000 7f7432489b80 1 --2- 172.21.15.2:0/2336843827 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5560f68da420 0x5560f68dc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.094+0000 7f7432489b80 4 client.25889 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.094+0000 7f7432489b80 1 -- 172.21.15.2:0/2336843827 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5560f678c600 con 0x5560f68da420 2023-11-16T16:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.094+0000 7f7417fff700 1 --2- 172.21.15.2:0/2336843827 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5560f68da420 0x5560f68dc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.094+0000 7f7417fff700 1 --2- 172.21.15.2:0/2336843827 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5560f68da420 0x5560f68dc830 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4aca5327a70a933b server_cookie=84330be6d5dab746 in_seq=0 out_seq=0 2023-11-16T16:55:43.101 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.094+0000 7f73f7fff700 10 client.25889 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:43.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:42 smithi155 ceph-mon[131607]: pgmap v177: 129 pgs: 129 active+clean; 602 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.9 MiB/s wr, 632 op/s 2023-11-16T16:55:43.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:42 smithi002 ceph-mon[128712]: pgmap v177: 129 pgs: 129 active+clean; 602 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.9 MiB/s wr, 632 op/s 2023-11-16T16:55:43.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:42 smithi141 ceph-mon[131184]: pgmap v177: 129 pgs: 129 active+clean; 602 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.9 MiB/s wr, 632 op/s 2023-11-16T16:55:43.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.411+0000 7f73f7fff700 1 -- 172.21.15.2:0/2336843827 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5560f678c600 con 0x5560f68da420 2023-11-16T16:55:43.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.411+0000 7f73f7fff700 10 client.25889 handle_command_reply: tid=0 2023-11-16T16:55:43.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.411+0000 7f7432489b80 1 client.25889 shutdown 2023-11-16T16:55:43.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.411+0000 7f7432489b80 1 -- 172.21.15.2:0/2336843827 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f66c86b0 msgr2=0x5560f6879410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:43.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.411+0000 7f7432489b80 1 --2- 172.21.15.2:0/2336843827 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f66c86b0 0x5560f6879410 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f740c09abc0 tx=0x7f740c09ad90 comp rx=0 tx=0).stop 2023-11-16T16:55:43.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.411+0000 7f7432489b80 1 -- 172.21.15.2:0/2336843827 shutdown_connections 2023-11-16T16:55:43.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.411+0000 7f7432489b80 1 --2- 172.21.15.2:0/2336843827 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5560f68da420 0x5560f68dc830 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.411+0000 7f7432489b80 1 --2- 172.21.15.2:0/2336843827 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5560f6872b90 0x5560f6879eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.411+0000 7f7432489b80 1 --2- 172.21.15.2:0/2336843827 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5560f66c86b0 0x5560f6879410 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.411+0000 7f73f7fff700 0 client.25889 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:43.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.411+0000 7f7432489b80 1 --2- 172.21.15.2:0/2336843827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5560f6872720 0x5560f6879950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.411+0000 7f7432489b80 1 -- 172.21.15.2:0/2336843827 >> 172.21.15.2:0/2336843827 conn(0x5560f6861f50 msgr2=0x5560f68752f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:43.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.412+0000 7f7432489b80 1 -- 172.21.15.2:0/2336843827 shutdown_connections 2023-11-16T16:55:43.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.412+0000 7f7432489b80 1 -- 172.21.15.2:0/2336843827 wait complete. 2023-11-16T16:55:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.412+0000 7f7432489b80 20 client.25889 trim_cache size 0 max 16384 2023-11-16T16:55:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:55:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:55:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:55:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:55:43.416 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.413+0000 7f741ebff700 1 -- 172.21.15.2:0/736235500 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f74080a1340 msgr2=0x7f74080a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:43.416 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.413+0000 7f741ebff700 1 --2- 172.21.15.2:0/736235500 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f74080a1340 0x7f74080a37f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7418199c80 tx=0x7f7404009260 comp rx=0 tx=0).stop 2023-11-16T16:55:43.416 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.413+0000 7f741ebff700 1 -- 172.21.15.2:0/736235500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7418100c50 msgr2=0x7f7418198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:43.416 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.413+0000 7f741ebff700 1 --2- 172.21.15.2:0/736235500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7418100c50 0x7f7418198670 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f740c00b580 tx=0x7f740c00bfd0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.416 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.413+0000 7f741ebff700 1 -- 172.21.15.2:0/736235500 shutdown_connections 2023-11-16T16:55:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.413+0000 7f741ebff700 1 --2- 172.21.15.2:0/736235500 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f74080a1340 0x7f74080a37f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.413+0000 7f741ebff700 1 --2- 172.21.15.2:0/736235500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7418100c50 0x7f7418198670 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.413+0000 7f741ebff700 1 --2- 172.21.15.2:0/736235500 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f7418105560 0x7f741819d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.413+0000 7f741ebff700 1 --2- 172.21.15.2:0/736235500 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74181015f0 0x7f7418198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.413+0000 7f741ebff700 1 -- 172.21.15.2:0/736235500 >> 172.21.15.2:0/736235500 conn(0x7f7418077550 msgr2=0x7f74180fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.413+0000 7f741ebff700 1 -- 172.21.15.2:0/736235500 shutdown_connections 2023-11-16T16:55:43.417 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:43.413+0000 7f741ebff700 1 -- 172.21.15.2:0/736235500 wait complete. 2023-11-16T16:55:45.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:44 smithi155 ceph-mon[131607]: pgmap v178: 129 pgs: 129 active+clean; 602 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 348 op/s 2023-11-16T16:55:45.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:44 smithi002 ceph-mon[128712]: pgmap v178: 129 pgs: 129 active+clean; 602 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 348 op/s 2023-11-16T16:55:45.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:44 smithi141 ceph-mon[131184]: pgmap v178: 129 pgs: 129 active+clean; 602 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 348 op/s 2023-11-16T16:55:47.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:46 smithi155 ceph-mon[131607]: pgmap v179: 129 pgs: 129 active+clean; 602 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 348 op/s 2023-11-16T16:55:47.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:46 smithi002 ceph-mon[128712]: pgmap v179: 129 pgs: 129 active+clean; 602 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 348 op/s 2023-11-16T16:55:47.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:46 smithi141 ceph-mon[131184]: pgmap v179: 129 pgs: 129 active+clean; 602 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 348 op/s 2023-11-16T16:55:49.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:48 smithi155 ceph-mon[131607]: pgmap v180: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.3 MiB/s wr, 747 op/s 2023-11-16T16:55:49.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:48 smithi002 ceph-mon[128712]: pgmap v180: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.3 MiB/s wr, 747 op/s 2023-11-16T16:55:49.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:48 smithi141 ceph-mon[131184]: pgmap v180: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.3 MiB/s wr, 747 op/s 2023-11-16T16:55:49.902 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T16:55:49.902 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T16:55:50.057 INFO:tasks.workunit:Running workunit fs/misc/filelock_deadlock.py... 2023-11-16T16:55:50.057 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/filelock_deadlock.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/filelock_deadlock.py 2023-11-16T16:55:51.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:50 smithi155 ceph-mon[131607]: pgmap v181: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 570 op/s 2023-11-16T16:55:51.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:50 smithi002 ceph-mon[128712]: pgmap v181: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 570 op/s 2023-11-16T16:55:51.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:50 smithi141 ceph-mon[131184]: pgmap v181: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 570 op/s 2023-11-16T16:55:53.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:52 smithi155 ceph-mon[131607]: pgmap v182: 129 pgs: 129 active+clean; 606 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.1 MiB/s wr, 629 op/s 2023-11-16T16:55:53.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:52 smithi002 ceph-mon[128712]: pgmap v182: 129 pgs: 129 active+clean; 606 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.1 MiB/s wr, 629 op/s 2023-11-16T16:55:53.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:52 smithi141 ceph-mon[131184]: pgmap v182: 129 pgs: 129 active+clean; 606 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.1 MiB/s wr, 629 op/s 2023-11-16T16:55:55.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:54 smithi155 ceph-mon[131607]: pgmap v183: 129 pgs: 129 active+clean; 606 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 457 op/s 2023-11-16T16:55:55.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:54 smithi002 ceph-mon[128712]: pgmap v183: 129 pgs: 129 active+clean; 606 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 457 op/s 2023-11-16T16:55:55.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:54 smithi141 ceph-mon[131184]: pgmap v183: 129 pgs: 129 active+clean; 606 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 457 op/s 2023-11-16T16:55:56.576 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.574+0000 7fb5bc3ff700 1 -- 172.21.15.2:0/3519233665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5b4100c50 msgr2=0x7fb5b4101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:56.576 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.574+0000 7fb5bc3ff700 1 --2- 172.21.15.2:0/3519233665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5b4100c50 0x7fb5b4101020 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb5b00099c0 tx=0x7fb5b0009cd0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.574+0000 7fb5bc3ff700 1 -- 172.21.15.2:0/3519233665 shutdown_connections 2023-11-16T16:55:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.574+0000 7fb5bc3ff700 1 --2- 172.21.15.2:0/3519233665 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5b4105560 0x7fb5b410d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.574+0000 7fb5bc3ff700 1 --2- 172.21.15.2:0/3519233665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5b4100c50 0x7fb5b4101020 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.574+0000 7fb5bc3ff700 1 --2- 172.21.15.2:0/3519233665 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb5b41015f0 0x7fb5b4104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.574+0000 7fb5bc3ff700 1 -- 172.21.15.2:0/3519233665 >> 172.21.15.2:0/3519233665 conn(0x7fb5b4077550 msgr2=0x7fb5b4077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.574+0000 7fb5bc3ff700 1 -- 172.21.15.2:0/3519233665 shutdown_connections 2023-11-16T16:55:56.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.574+0000 7fb5bc3ff700 1 -- 172.21.15.2:0/3519233665 wait complete. 2023-11-16T16:55:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.575+0000 7fb5bc3ff700 1 Processor -- start 2023-11-16T16:55:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.576+0000 7fb5bc3ff700 1 -- start start 2023-11-16T16:55:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.576+0000 7fb5bc3ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb5b4100c50 0x7fb5b41985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.576+0000 7fb5bc3ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5b41015f0 0x7fb5b4198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.576+0000 7fb5bc3ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5b4105560 0x7fb5b419cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.576+0000 7fb5bc3ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b419d5a0 con 0x7fb5b4105560 2023-11-16T16:55:56.579 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.576+0000 7fb5bc3ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b419d710 con 0x7fb5b4100c50 2023-11-16T16:55:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.576+0000 7fb5bc3ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b419d8d0 con 0x7fb5b41015f0 2023-11-16T16:55:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.576+0000 7fb5ba99c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5b4105560 0x7fb5b419cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.576+0000 7fb5ba99c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5b4105560 0x7fb5b419cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40478/0 (socket says 172.21.15.2:40478) 2023-11-16T16:55:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.576+0000 7fb5ba99c700 1 -- 172.21.15.2:0/2694380653 learned_addr learned my addr 172.21.15.2:0/2694380653 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.576+0000 7fb5ba19b700 1 --2- 172.21.15.2:0/2694380653 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb5b4100c50 0x7fb5b41985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.576+0000 7fb5b999a700 1 --2- 172.21.15.2:0/2694380653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5b41015f0 0x7fb5b4198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.577+0000 7fb5ba99c700 1 -- 172.21.15.2:0/2694380653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5b41015f0 msgr2=0x7fb5b4198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.577+0000 7fb5ba99c700 1 --2- 172.21.15.2:0/2694380653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5b41015f0 0x7fb5b4198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.577+0000 7fb5ba99c700 1 -- 172.21.15.2:0/2694380653 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb5b4100c50 msgr2=0x7fb5b41985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.577+0000 7fb5ba99c700 1 --2- 172.21.15.2:0/2694380653 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb5b4100c50 0x7fb5b41985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.577+0000 7fb5ba99c700 1 -- 172.21.15.2:0/2694380653 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b00095f0 con 0x7fb5b4105560 2023-11-16T16:55:56.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.577+0000 7fb5ba99c700 1 --2- 172.21.15.2:0/2694380653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5b4105560 0x7fb5b419cf60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00ef30 tx=0x7fb5ac00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.577+0000 7fb5ab7fe700 1 -- 172.21.15.2:0/2694380653 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac018070 con 0x7fb5b4105560 2023-11-16T16:55:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.577+0000 7fb5ab7fe700 1 -- 172.21.15.2:0/2694380653 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5ac00f040 con 0x7fb5b4105560 2023-11-16T16:55:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.577+0000 7fb5ab7fe700 1 -- 172.21.15.2:0/2694380653 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac00cd00 con 0x7fb5b4105560 2023-11-16T16:55:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.577+0000 7fb5bc3ff700 1 -- 172.21.15.2:0/2694380653 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b41103b0 con 0x7fb5b4105560 2023-11-16T16:55:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.577+0000 7fb5bc3ff700 1 -- 172.21.15.2:0/2694380653 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b4110220 con 0x7fb5b4105560 2023-11-16T16:55:56.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.581+0000 7fb5ab7fe700 1 -- 172.21.15.2:0/2694380653 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fb5ac003a00 con 0x7fb5b4105560 2023-11-16T16:55:56.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.582+0000 7fb5cfc69b80 1 Processor -- start 2023-11-16T16:55:56.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.583+0000 7fb5cfc69b80 1 -- start start 2023-11-16T16:55:56.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.583+0000 7fb5cfc69b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f93156b0 0x5637f94bf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:56.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.583+0000 7fb5cfc69b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 0x5637f94b9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:56.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.583+0000 7fb5cfc69b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bfb90 0x5637f94b9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:56.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.583+0000 7fb5cfc69b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637f93bf000 con 0x5637f94bf720 2023-11-16T16:55:56.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.583+0000 7fb5cfc69b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637f9411840 con 0x5637f93156b0 2023-11-16T16:55:56.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.583+0000 7fb5cfc69b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637f940acd0 con 0x5637f94bfb90 2023-11-16T16:55:56.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.583+0000 7fb5ab7fe700 1 --2- 172.21.15.2:0/2694380653 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fb5a00a1340 0x7fb5a00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:56.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.583+0000 7fb5ba19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 0x5637f94b9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:56.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.583+0000 7fb5ab7fe700 1 -- 172.21.15.2:0/2694380653 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fb5ac0ccb00 con 0x7fb5b4105560 2023-11-16T16:55:56.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.583+0000 7fb5b999a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f93156b0 0x5637f94bf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:56.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.583+0000 7fb5ba19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 0x5637f94b9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40494/0 (socket says 172.21.15.2:40494) 2023-11-16T16:55:56.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.583+0000 7fb5b999a700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f93156b0 0x5637f94bf1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:60976/0 (socket says 172.21.15.2:60976) 2023-11-16T16:55:56.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.583+0000 7fb5ba19b700 1 -- 172.21.15.2:0/3289236029 learned_addr learned my addr 172.21.15.2:0/3289236029 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:56.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.583+0000 7fb5ba99c700 1 --2- 172.21.15.2:0/2694380653 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fb5a00a1340 0x7fb5a00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.584+0000 7fb5b999a700 1 --2- 172.21.15.2:0/3289236029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bfb90 0x5637f94b9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.584+0000 7fb5ba19b700 1 -- 172.21.15.2:0/3289236029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bfb90 msgr2=0x5637f94b9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.584+0000 7fb5ba19b700 1 --2- 172.21.15.2:0/3289236029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bfb90 0x5637f94b9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.584+0000 7fb5ba19b700 1 -- 172.21.15.2:0/3289236029 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f93156b0 msgr2=0x5637f94bf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.584+0000 7fb5ba19b700 1 --2- 172.21.15.2:0/3289236029 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f93156b0 0x5637f94bf1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.584+0000 7fb5ba19b700 1 -- 172.21.15.2:0/3289236029 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5637f928c5d0 con 0x5637f94bf720 2023-11-16T16:55:56.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.584+0000 7fb5ba19b700 1 --2- 172.21.15.2:0/3289236029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 0x5637f94b9560 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb5b00058e0 tx=0x7fb5b0005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f959cd090806498 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:56.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.584+0000 7fb5ba99c700 1 --2- 172.21.15.2:0/2694380653 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fb5a00a1340 0x7fb5a00a37f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0c9ee0 tx=0x7fb5ac0092e0 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:56.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.584+0000 7fb5a8bfe700 1 -- 172.21.15.2:0/3289236029 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b000ba50 con 0x5637f94bf720 2023-11-16T16:55:56.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.584+0000 7fb5a8bfe700 1 -- 172.21.15.2:0/3289236029 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5b001e040 con 0x5637f94bf720 2023-11-16T16:55:56.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.584+0000 7fb5a8bfe700 1 -- 172.21.15.2:0/3289236029 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b00223f0 con 0x5637f94bf720 2023-11-16T16:55:56.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.585+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/3289236029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 msgr2=0x5637f94b9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:56.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.585+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/3289236029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 0x5637f94b9560 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb5b00058e0 tx=0x7fb5b0005f00 comp rx=0 tx=0).stop 2023-11-16T16:55:56.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.585+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/3289236029 shutdown_connections 2023-11-16T16:55:56.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.585+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/3289236029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bfb90 0x5637f94b9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.585+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/3289236029 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f93156b0 0x5637f94bf1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.585+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/3289236029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 0x5637f94b9560 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.585+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/3289236029 >> 172.21.15.2:0/3289236029 conn(0x5637f94aef50 msgr2=0x5637f94b4270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:56.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.585+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/3289236029 shutdown_connections 2023-11-16T16:55:56.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.585+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/3289236029 wait complete. 2023-11-16T16:55:56.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.586+0000 7fb5cfc69b80 1 -- start start 2023-11-16T16:55:56.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.586+0000 7fb5cfc69b80 1 Processor -- start 2023-11-16T16:55:56.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.586+0000 7fb5cfc69b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f93156b0 0x5637f94c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:56.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.586+0000 7fb5cfc69b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bf720 0x5637f94c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:56.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.586+0000 7fb5cfc69b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f94bfb90 0x5637f94c2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:56.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.586+0000 7fb5cfc69b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637f93ac880 con 0x5637f93156b0 2023-11-16T16:55:56.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.587+0000 7fb5cfc69b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637f93ac520 con 0x5637f94bfb90 2023-11-16T16:55:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.587+0000 7fb5cfc69b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637f928e8c0 con 0x5637f94bf720 2023-11-16T16:55:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.587+0000 7fb5cfc69b80 10 client.0 _ll_register_callbacks cb 0x5637f93ca720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.587+0000 7fb5b999a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f93156b0 0x5637f94c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.587+0000 7fb5b999a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f93156b0 0x5637f94c8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40510/0 (socket says 172.21.15.2:40510) 2023-11-16T16:55:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.587+0000 7fb5b999a700 1 -- 172.21.15.2:0/4097747560 learned_addr learned my addr 172.21.15.2:0/4097747560 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.587+0000 7fb5ba19b700 1 --2- 172.21.15.2:0/4097747560 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bf720 0x5637f94c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.587+0000 7fb5b999a700 1 --2- 172.21.15.2:0/4097747560 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f94bfb90 0x5637f94c2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.587+0000 7fb5b999a700 1 -- 172.21.15.2:0/4097747560 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bf720 msgr2=0x5637f94c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.587+0000 7fb5b999a700 1 --2- 172.21.15.2:0/4097747560 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bf720 0x5637f94c8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.587+0000 7fb5b999a700 1 -- 172.21.15.2:0/4097747560 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f94bfb90 msgr2=0x5637f94c2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.587+0000 7fb5b999a700 1 --2- 172.21.15.2:0/4097747560 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f94bfb90 0x5637f94c2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.587+0000 7fb5b999a700 1 -- 172.21.15.2:0/4097747560 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b410d9b0 con 0x5637f93156b0 2023-11-16T16:55:56.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.587+0000 7fb5b999a700 1 --2- 172.21.15.2:0/4097747560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f93156b0 0x5637f94c8a40 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009ee0 tx=0x7fb5a40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93ffa0f497808c9 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:56.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.587+0000 7fb59dffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:55:56.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.588+0000 7fb59dffb700 1 -- 172.21.15.2:0/4097747560 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4016070 con 0x5637f93156b0 2023-11-16T16:55:56.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.588+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/4097747560 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5637f94ac710 con 0x5637f93156b0 2023-11-16T16:55:56.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.588+0000 7fb59dffb700 1 -- 172.21.15.2:0/4097747560 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fb5a4012e40 con 0x5637f93156b0 2023-11-16T16:55:56.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.588+0000 7fb59dffb700 1 -- 172.21.15.2:0/4097747560 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5a400fd70 con 0x5637f93156b0 2023-11-16T16:55:56.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.588+0000 7fb5cfc69b80 10 client.15852 fetch_fsmap learned FSMap version 35 2023-11-16T16:55:56.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.588+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/4097747560 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x5637f93d1130 con 0x5637f93156b0 2023-11-16T16:55:56.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.588+0000 7fb59dffb700 1 -- 172.21.15.2:0/4097747560 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5637f94ac710 con 0x5637f93156b0 2023-11-16T16:55:56.594 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.588+0000 7fb59dffb700 1 -- 172.21.15.2:0/4097747560 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fb5a401e030 con 0x5637f93156b0 2023-11-16T16:55:56.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.589+0000 7fb5cfc69b80 10 client.15852 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:55:56.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.589+0000 7fb5cfc69b80 10 client.15852 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:56.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.589+0000 7fb5cfc69b80 20 client.15852 populate_metadata read hostname 'smithi002' 2023-11-16T16:55:56.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.589+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/4097747560 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5637f94d2f70 0x5637f94d5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:56.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.589+0000 7fb5cfc69b80 4 client.15852 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:55:56.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.589+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/4097747560 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5637f93cd040 con 0x5637f94d2f70 2023-11-16T16:55:56.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.589+0000 7fb5ba19b700 1 --2- 172.21.15.2:0/4097747560 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5637f94d2f70 0x5637f94d5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:56.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.589+0000 7fb59dffb700 10 client.15852 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:56.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.589+0000 7fb5ba19b700 1 --2- 172.21.15.2:0/4097747560 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5637f94d2f70 0x5637f94d5380 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=107890157494f943 server_cookie=c3704dc37644ddf9 in_seq=0 out_seq=0 2023-11-16T16:55:56.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.590+0000 7fb59dffb700 1 -- 172.21.15.2:0/4097747560 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5637f93cd040 con 0x5637f94d2f70 2023-11-16T16:55:56.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.590+0000 7fb59dffb700 10 client.15852 handle_command_reply: tid=0 2023-11-16T16:55:56.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.590+0000 7fb5cfc69b80 1 client.15852 shutdown 2023-11-16T16:55:56.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.590+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/4097747560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f93156b0 msgr2=0x5637f94c8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:56.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.590+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/4097747560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f93156b0 0x5637f94c8a40 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009ee0 tx=0x7fb5a40047c0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.591+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/4097747560 shutdown_connections 2023-11-16T16:55:56.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.591+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/4097747560 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f94bfb90 0x5637f94c2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.591+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/4097747560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f93156b0 0x5637f94c8a40 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.591+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/4097747560 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5637f94d2f70 0x5637f94d5380 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.591+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/4097747560 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bf720 0x5637f94c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.591+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/4097747560 >> 172.21.15.2:0/4097747560 conn(0x5637f94aef50 msgr2=0x5637f94b37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:56.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.591+0000 7fb59dffb700 0 client.15852 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:56.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.591+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/4097747560 shutdown_connections 2023-11-16T16:55:56.598 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.591+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/4097747560 wait complete. 2023-11-16T16:55:56.598 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.591+0000 7fb5cfc69b80 20 client.15852 trim_cache size 0 max 16384 2023-11-16T16:55:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.606+0000 7fb5cfc69b80 1 Processor -- start 2023-11-16T16:55:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.607+0000 7fb5cfc69b80 1 -- start start 2023-11-16T16:55:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.607+0000 7fb5cfc69b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f93156b0 0x5637f94c2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.607+0000 7fb5cfc69b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 0x5637f94b9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.607+0000 7fb5cfc69b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bfb90 0x5637f94b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.607+0000 7fb5cfc69b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637f94ac310 con 0x5637f94bf720 2023-11-16T16:55:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.607+0000 7fb5cfc69b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637f94c12f0 con 0x5637f93156b0 2023-11-16T16:55:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.607+0000 7fb5cfc69b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637f94bda20 con 0x5637f94bfb90 2023-11-16T16:55:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.607+0000 7fb5ba19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 0x5637f94b9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.607+0000 7fb5ba19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 0x5637f94b9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40514/0 (socket says 172.21.15.2:40514) 2023-11-16T16:55:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.607+0000 7fb5ba19b700 1 -- 172.21.15.2:0/2224906247 learned_addr learned my addr 172.21.15.2:0/2224906247 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.607+0000 7fb5b999a700 1 --2- 172.21.15.2:0/2224906247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f93156b0 0x5637f94c2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.607+0000 7fb5b999a700 1 --2- 172.21.15.2:0/2224906247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bfb90 0x5637f94b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.607+0000 7fb5ba19b700 1 -- 172.21.15.2:0/2224906247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bfb90 msgr2=0x5637f94b3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.607+0000 7fb5ba19b700 1 --2- 172.21.15.2:0/2224906247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bfb90 0x5637f94b3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.607+0000 7fb5ba19b700 1 -- 172.21.15.2:0/2224906247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f93156b0 msgr2=0x5637f94c2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.607+0000 7fb5ba19b700 1 --2- 172.21.15.2:0/2224906247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f93156b0 0x5637f94c2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.607+0000 7fb5ba19b700 1 -- 172.21.15.2:0/2224906247 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5637f94ba2a0 con 0x5637f94bf720 2023-11-16T16:55:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.607+0000 7fb5b999a700 1 --2- 172.21.15.2:0/2224906247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bfb90 0x5637f94b3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-16T16:55:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.608+0000 7fb5ba19b700 1 --2- 172.21.15.2:0/2224906247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 0x5637f94b9560 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb5b00098e0 tx=0x7fb5b001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15028e81e1aa0ec8 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.608+0000 7fb5a93ff700 1 -- 172.21.15.2:0/2224906247 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b000b5b0 con 0x5637f94bf720 2023-11-16T16:55:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.608+0000 7fb5a93ff700 1 -- 172.21.15.2:0/2224906247 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5b000f4e0 con 0x5637f94bf720 2023-11-16T16:55:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.608+0000 7fb5a93ff700 1 -- 172.21.15.2:0/2224906247 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b000b850 con 0x5637f94bf720 2023-11-16T16:55:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.608+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/2224906247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 msgr2=0x5637f94b9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.609+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/2224906247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 0x5637f94b9560 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb5b00098e0 tx=0x7fb5b001a040 comp rx=0 tx=0).stop 2023-11-16T16:55:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.609+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/2224906247 shutdown_connections 2023-11-16T16:55:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.609+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/2224906247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bfb90 0x5637f94b3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.609+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/2224906247 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f93156b0 0x5637f94c2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.609+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/2224906247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 0x5637f94b9560 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.609+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/2224906247 >> 172.21.15.2:0/2224906247 conn(0x5637f94aef50 msgr2=0x5637f94c36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.609+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/2224906247 shutdown_connections 2023-11-16T16:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.609+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/2224906247 wait complete. 2023-11-16T16:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.609+0000 7fb5cfc69b80 1 -- start start 2023-11-16T16:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.609+0000 7fb5cfc69b80 1 Processor -- start 2023-11-16T16:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.610+0000 7fb5cfc69b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f93156b0 0x5637f94c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.610+0000 7fb5cfc69b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 0x5637f94c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.610+0000 7fb5cfc69b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bfb90 0x5637f94c7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.610+0000 7fb5cfc69b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637f9525df0 con 0x5637f94bf720 2023-11-16T16:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.610+0000 7fb5cfc69b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637f9525f60 con 0x5637f93156b0 2023-11-16T16:55:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.610+0000 7fb5cfc69b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637f9526120 con 0x5637f94bfb90 2023-11-16T16:55:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.610+0000 7fb5cfc69b80 10 client.0 _ll_register_callbacks cb 0x7fb5a40128a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:55:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.610+0000 7fb5ba19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 0x5637f94c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.610+0000 7fb5ba19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 0x5637f94c6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40526/0 (socket says 172.21.15.2:40526) 2023-11-16T16:55:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.610+0000 7fb5ba19b700 1 -- 172.21.15.2:0/229708899 learned_addr learned my addr 172.21.15.2:0/229708899 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:55:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.610+0000 7fb5b999a700 1 --2- 172.21.15.2:0/229708899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bfb90 0x5637f94c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.610+0000 7fb5b999a700 1 --2- 172.21.15.2:0/229708899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f93156b0 0x5637f94c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.610+0000 7fb5ba19b700 1 -- 172.21.15.2:0/229708899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bfb90 msgr2=0x5637f94c7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.610+0000 7fb5ba19b700 1 --2- 172.21.15.2:0/229708899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bfb90 0x5637f94c7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.610+0000 7fb5ba19b700 1 -- 172.21.15.2:0/229708899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f93156b0 msgr2=0x5637f94c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:55:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.610+0000 7fb5ba19b700 1 --2- 172.21.15.2:0/229708899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f93156b0 0x5637f94c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.610+0000 7fb5ba19b700 1 -- 172.21.15.2:0/229708899 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5637f928c5d0 con 0x5637f94bf720 2023-11-16T16:55:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.610+0000 7fb5b999a700 1 --2- 172.21.15.2:0/229708899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f93156b0 0x5637f94c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:55:56.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.611+0000 7fb5ba19b700 1 --2- 172.21.15.2:0/229708899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 0x5637f94c6b30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0009d10 tx=0x7fb5b0005a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4bc33b46cdec432 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.611+0000 7fb59dffb700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.611+0000 7fb59dffb700 1 -- 172.21.15.2:0/229708899 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b0035cd0 con 0x5637f94bf720 2023-11-16T16:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.611+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/229708899 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb5b001e040 con 0x5637f94bf720 2023-11-16T16:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.611+0000 7fb59dffb700 1 -- 172.21.15.2:0/229708899 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fb5b0035e30 con 0x5637f94bf720 2023-11-16T16:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.611+0000 7fb59dffb700 1 -- 172.21.15.2:0/229708899 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5b0035600 con 0x5637f94bf720 2023-11-16T16:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.611+0000 7fb5cfc69b80 10 client.15858 fetch_fsmap learned FSMap version 35 2023-11-16T16:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.611+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/229708899 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x5637f9238d80 con 0x5637f94bf720 2023-11-16T16:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.611+0000 7fb59dffb700 1 -- 172.21.15.2:0/229708899 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5b001e040 con 0x5637f94bf720 2023-11-16T16:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.612+0000 7fb59dffb700 1 -- 172.21.15.2:0/229708899 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fb5b0021070 con 0x5637f94bf720 2023-11-16T16:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.612+0000 7fb5cfc69b80 10 client.15858 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.612+0000 7fb5cfc69b80 10 client.15858 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.612+0000 7fb5cfc69b80 20 client.15858 populate_metadata read hostname 'smithi002' 2023-11-16T16:55:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.612+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/229708899 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5637f95273d0 0x5637f95297e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:55:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.612+0000 7fb5cfc69b80 4 client.15858 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:55:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.612+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/229708899 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5637f93d9600 con 0x5637f95273d0 2023-11-16T16:55:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.612+0000 7fb5b999a700 1 --2- 172.21.15.2:0/229708899 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5637f95273d0 0x5637f95297e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:55:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.612+0000 7fb5b999a700 1 --2- 172.21.15.2:0/229708899 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5637f95273d0 0x5637f95297e0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14029a229da08ba6 server_cookie=7c74d1c1c11caffc in_seq=0 out_seq=0 2023-11-16T16:55:56.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.612+0000 7fb59dffb700 10 client.15858 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.806+0000 7fb59dffb700 1 -- 172.21.15.2:0/229708899 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5637f93d9600 con 0x5637f95273d0 2023-11-16T16:55:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.806+0000 7fb59dffb700 10 client.15858 handle_command_reply: tid=0 2023-11-16T16:55:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.806+0000 7fb5cfc69b80 1 client.15858 shutdown 2023-11-16T16:55:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.806+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/229708899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 msgr2=0x5637f94c6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.806+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/229708899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 0x5637f94c6b30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0009d10 tx=0x7fb5b0005a40 comp rx=0 tx=0).stop 2023-11-16T16:55:56.808 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.806+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/229708899 shutdown_connections 2023-11-16T16:55:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.806+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/229708899 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5637f95273d0 0x5637f95297e0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.806+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/229708899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5637f94bfb90 0x5637f94c7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.806+0000 7fb59dffb700 0 client.15858 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:55:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.806+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/229708899 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5637f93156b0 0x5637f94c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.806+0000 7fb5cfc69b80 1 --2- 172.21.15.2:0/229708899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5637f94bf720 0x5637f94c6b30 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.806+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/229708899 >> 172.21.15.2:0/229708899 conn(0x5637f94aef50 msgr2=0x5637f94bff60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.806+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/229708899 shutdown_connections 2023-11-16T16:55:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.806+0000 7fb5cfc69b80 1 -- 172.21.15.2:0/229708899 wait complete. 2023-11-16T16:55:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.807+0000 7fb5cfc69b80 20 client.15858 trim_cache size 0 max 16384 2023-11-16T16:55:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:55:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:55:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:55:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:55:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.807+0000 7fb5bc3ff700 1 -- 172.21.15.2:0/2694380653 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fb5a00a1340 msgr2=0x7fb5a00a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.807+0000 7fb5bc3ff700 1 --2- 172.21.15.2:0/2694380653 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fb5a00a1340 0x7fb5a00a37f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0c9ee0 tx=0x7fb5ac0092e0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.807+0000 7fb5bc3ff700 1 -- 172.21.15.2:0/2694380653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5b4105560 msgr2=0x7fb5b419cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:55:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.807+0000 7fb5bc3ff700 1 --2- 172.21.15.2:0/2694380653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5b4105560 0x7fb5b419cf60 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00ef30 tx=0x7fb5ac00c3a0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.807+0000 7fb5bc3ff700 1 -- 172.21.15.2:0/2694380653 shutdown_connections 2023-11-16T16:55:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.807+0000 7fb5bc3ff700 1 --2- 172.21.15.2:0/2694380653 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fb5a00a1340 0x7fb5a00a37f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.807+0000 7fb5bc3ff700 1 --2- 172.21.15.2:0/2694380653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5b4105560 0x7fb5b419cf60 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.807+0000 7fb5bc3ff700 1 --2- 172.21.15.2:0/2694380653 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb5b4100c50 0x7fb5b41985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.807+0000 7fb5bc3ff700 1 --2- 172.21.15.2:0/2694380653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5b41015f0 0x7fb5b4198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:55:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.807+0000 7fb5bc3ff700 1 -- 172.21.15.2:0/2694380653 >> 172.21.15.2:0/2694380653 conn(0x7fb5b4077550 msgr2=0x7fb5b40fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:55:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.807+0000 7fb5bc3ff700 1 -- 172.21.15.2:0/2694380653 shutdown_connections 2023-11-16T16:55:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:55:56.807+0000 7fb5bc3ff700 1 -- 172.21.15.2:0/2694380653 wait complete. 2023-11-16T16:55:57.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:56 smithi155 ceph-mon[131607]: pgmap v184: 129 pgs: 129 active+clean; 606 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 457 op/s 2023-11-16T16:55:57.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:56 smithi002 ceph-mon[128712]: pgmap v184: 129 pgs: 129 active+clean; 606 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 457 op/s 2023-11-16T16:55:57.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:56 smithi141 ceph-mon[131184]: pgmap v184: 129 pgs: 129 active+clean; 606 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.1 MiB/s wr, 457 op/s 2023-11-16T16:55:59.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:55:58 smithi155 ceph-mon[131607]: pgmap v185: 129 pgs: 129 active+clean; 606 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.1 MiB/s wr, 461 op/s 2023-11-16T16:55:59.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:55:58 smithi002 ceph-mon[128712]: pgmap v185: 129 pgs: 129 active+clean; 606 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.1 MiB/s wr, 461 op/s 2023-11-16T16:55:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:55:58 smithi141 ceph-mon[131184]: pgmap v185: 129 pgs: 129 active+clean; 606 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.1 MiB/s wr, 461 op/s 2023-11-16T16:56:00.168 INFO:tasks.workunit.client.0.smithi002.stdout:ok 2023-11-16T16:56:00.174 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T16:56:00.175 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T16:56:00.270 INFO:tasks.workunit:Running workunit fs/misc/filelock_interrupt.py... 2023-11-16T16:56:00.270 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/filelock_interrupt.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/filelock_interrupt.py 2023-11-16T16:56:01.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:00 smithi155 ceph-mon[131607]: pgmap v186: 129 pgs: 129 active+clean; 606 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 265 KiB/s wr, 62 op/s 2023-11-16T16:56:01.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:00 smithi002 ceph-mon[128712]: pgmap v186: 129 pgs: 129 active+clean; 606 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 265 KiB/s wr, 62 op/s 2023-11-16T16:56:01.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:00 smithi141 ceph-mon[131184]: pgmap v186: 129 pgs: 129 active+clean; 606 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 265 KiB/s wr, 62 op/s 2023-11-16T16:56:03.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:02 smithi155 ceph-mon[131607]: pgmap v187: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 351 KiB/s wr, 256 op/s 2023-11-16T16:56:03.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:02 smithi002 ceph-mon[128712]: pgmap v187: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 351 KiB/s wr, 256 op/s 2023-11-16T16:56:03.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:02 smithi141 ceph-mon[131184]: pgmap v187: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 351 KiB/s wr, 256 op/s 2023-11-16T16:56:05.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:04 smithi155 ceph-mon[131607]: pgmap v188: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 89 KiB/s wr, 198 op/s 2023-11-16T16:56:05.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:04 smithi002 ceph-mon[128712]: pgmap v188: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 89 KiB/s wr, 198 op/s 2023-11-16T16:56:05.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:04 smithi141 ceph-mon[131184]: pgmap v188: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 89 KiB/s wr, 198 op/s 2023-11-16T16:56:07.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:06 smithi155 ceph-mon[131607]: pgmap v189: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 89 KiB/s wr, 198 op/s 2023-11-16T16:56:07.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:06 smithi002 ceph-mon[128712]: pgmap v189: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 89 KiB/s wr, 198 op/s 2023-11-16T16:56:07.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:06 smithi141 ceph-mon[131184]: pgmap v189: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 89 KiB/s wr, 198 op/s 2023-11-16T16:56:09.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:08 smithi155 ceph-mon[131607]: pgmap v190: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 90 KiB/s wr, 202 op/s 2023-11-16T16:56:09.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:08 smithi002 ceph-mon[128712]: pgmap v190: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 90 KiB/s wr, 202 op/s 2023-11-16T16:56:09.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:08 smithi141 ceph-mon[131184]: pgmap v190: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 90 KiB/s wr, 202 op/s 2023-11-16T16:56:09.973 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.971+0000 7f90cb5ff700 1 -- 172.21.15.2:0/602904994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90c41015f0 msgr2=0x7f90c4104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:09.973 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.971+0000 7f90cb5ff700 1 --2- 172.21.15.2:0/602904994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90c41015f0 0x7f90c4104e80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f90b40099c0 tx=0x7f90b4009cd0 comp rx=0 tx=0).stop 2023-11-16T16:56:09.973 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.971+0000 7f90cb5ff700 1 -- 172.21.15.2:0/602904994 shutdown_connections 2023-11-16T16:56:09.973 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.971+0000 7f90cb5ff700 1 --2- 172.21.15.2:0/602904994 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90c4105560 0x7f90c410d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:09.973 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.971+0000 7f90cb5ff700 1 --2- 172.21.15.2:0/602904994 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90c4100c50 0x7f90c4101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:09.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.971+0000 7f90cb5ff700 1 --2- 172.21.15.2:0/602904994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90c41015f0 0x7f90c4104e80 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:09.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.971+0000 7f90cb5ff700 1 -- 172.21.15.2:0/602904994 >> 172.21.15.2:0/602904994 conn(0x7f90c4077550 msgr2=0x7f90c4077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:09.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.971+0000 7f90cb5ff700 1 -- 172.21.15.2:0/602904994 shutdown_connections 2023-11-16T16:56:09.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.971+0000 7f90cb5ff700 1 -- 172.21.15.2:0/602904994 wait complete. 2023-11-16T16:56:09.975 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.972+0000 7f90cb5ff700 1 Processor -- start 2023-11-16T16:56:09.975 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.972+0000 7f90cb5ff700 1 -- start start 2023-11-16T16:56:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.973+0000 7f90cb5ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90c4100c50 0x7f90c4072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.973+0000 7f90cb5ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90c41015f0 0x7f90c4073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.973+0000 7f90cb5ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90c4105560 0x7f90c41a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.973+0000 7f90cb5ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c4073b60 con 0x7f90c41015f0 2023-11-16T16:56:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.973+0000 7f90cb5ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c41a4af0 con 0x7f90c4105560 2023-11-16T16:56:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.973+0000 7f90cb5ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c41a4c30 con 0x7f90c4100c50 2023-11-16T16:56:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.973+0000 7f90c8b9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90c41015f0 0x7f90c4073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.973+0000 7f90c8b9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90c41015f0 0x7f90c4073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49470/0 (socket says 172.21.15.2:49470) 2023-11-16T16:56:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.973+0000 7f90c8b9a700 1 -- 172.21.15.2:0/3625621023 learned_addr learned my addr 172.21.15.2:0/3625621023 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:56:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.973+0000 7f90c939b700 1 --2- 172.21.15.2:0/3625621023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90c4100c50 0x7f90c4072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.973+0000 7f90c9b9c700 1 --2- 172.21.15.2:0/3625621023 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90c4105560 0x7f90c41a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.974+0000 7f90c8b9a700 1 -- 172.21.15.2:0/3625621023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90c4100c50 msgr2=0x7f90c4072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.974+0000 7f90c8b9a700 1 --2- 172.21.15.2:0/3625621023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90c4100c50 0x7f90c4072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:09.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.974+0000 7f90c8b9a700 1 -- 172.21.15.2:0/3625621023 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90c4105560 msgr2=0x7f90c41a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:09.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.974+0000 7f90c8b9a700 1 --2- 172.21.15.2:0/3625621023 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90c4105560 0x7f90c41a45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:09.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.974+0000 7f90c8b9a700 1 -- 172.21.15.2:0/3625621023 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b40095f0 con 0x7f90c41015f0 2023-11-16T16:56:09.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.974+0000 7f90c939b700 1 --2- 172.21.15.2:0/3625621023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90c4100c50 0x7f90c4072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:56:09.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.974+0000 7f90c9b9c700 1 --2- 172.21.15.2:0/3625621023 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90c4105560 0x7f90c41a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:56:09.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.974+0000 7f90c8b9a700 1 --2- 172.21.15.2:0/3625621023 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90c41015f0 0x7f90c4073330 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f90b40098e0 tx=0x7f90b400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.974+0000 7f90ba7fc700 1 -- 172.21.15.2:0/3625621023 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b401d070 con 0x7f90c41015f0 2023-11-16T16:56:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.974+0000 7f90ba7fc700 1 -- 172.21.15.2:0/3625621023 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f90b4022470 con 0x7f90c41015f0 2023-11-16T16:56:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.974+0000 7f90ba7fc700 1 -- 172.21.15.2:0/3625621023 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b4017650 con 0x7f90c41015f0 2023-11-16T16:56:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.974+0000 7f90cb5ff700 1 -- 172.21.15.2:0/3625621023 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c4110370 con 0x7f90c41015f0 2023-11-16T16:56:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.974+0000 7f90cb5ff700 1 -- 172.21.15.2:0/3625621023 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c41101e0 con 0x7f90c41015f0 2023-11-16T16:56:09.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.978+0000 7f90ba7fc700 1 -- 172.21.15.2:0/3625621023 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f90b4022850 con 0x7f90c41015f0 2023-11-16T16:56:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.979+0000 7f90ba7fc700 1 --2- 172.21.15.2:0/3625621023 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f90b00a1340 0x7f90b00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.979+0000 7f90deeb0b80 1 Processor -- start 2023-11-16T16:56:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.979+0000 7f90ba7fc700 1 -- 172.21.15.2:0/3625621023 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f90b40cdf30 con 0x7f90c41015f0 2023-11-16T16:56:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.979+0000 7f90deeb0b80 1 -- start start 2023-11-16T16:56:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.979+0000 7f90c939b700 1 --2- 172.21.15.2:0/3625621023 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f90b00a1340 0x7f90b00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.980+0000 7f90deeb0b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b12bd6b0 0x5593b14671e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.980+0000 7f90deeb0b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467720 0x5593b1461560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.980+0000 7f90deeb0b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b1467b90 0x5593b1461b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.980+0000 7f90deeb0b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593b1367000 con 0x5593b12bd6b0 2023-11-16T16:56:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.980+0000 7f90deeb0b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593b13b9840 con 0x5593b1467b90 2023-11-16T16:56:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.980+0000 7f90deeb0b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593b13b2cd0 con 0x5593b1467720 2023-11-16T16:56:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.980+0000 7f90c939b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b12bd6b0 0x5593b14671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.980+0000 7f90c939b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b12bd6b0 0x5593b14671e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49478/0 (socket says 172.21.15.2:49478) 2023-11-16T16:56:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.980+0000 7f90c939b700 1 -- 172.21.15.2:0/4174077653 learned_addr learned my addr 172.21.15.2:0/4174077653 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:56:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.980+0000 7f90c8b9a700 1 --2- 172.21.15.2:0/4174077653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467720 0x5593b1461560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.980+0000 7f90c9b9c700 1 --2- 172.21.15.2:0/4174077653 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b1467b90 0x5593b1461b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.980+0000 7f90c8b9a700 1 -- 172.21.15.2:0/4174077653 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b1467b90 msgr2=0x5593b1461b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.980+0000 7f90c8b9a700 1 --2- 172.21.15.2:0/4174077653 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b1467b90 0x5593b1461b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.980+0000 7f90c8b9a700 1 -- 172.21.15.2:0/4174077653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b12bd6b0 msgr2=0x5593b14671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.980+0000 7f90c8b9a700 1 --2- 172.21.15.2:0/4174077653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b12bd6b0 0x5593b14671e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.980+0000 7f90c8b9a700 1 -- 172.21.15.2:0/4174077653 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593b12345d0 con 0x5593b1467720 2023-11-16T16:56:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.981+0000 7f90c939b700 1 --2- 172.21.15.2:0/3625621023 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f90b00a1340 0x7f90b00a37f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f90c00139a0 tx=0x7f90c0013d60 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.981+0000 7f90c8b9a700 1 --2- 172.21.15.2:0/4174077653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467720 0x5593b1461560 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f90b40cb2b0 tx=0x7f90b409a800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb2ebfdb67b48b53 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.981+0000 7f90af7fe700 1 -- 172.21.15.2:0/4174077653 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b409acc0 con 0x5593b1467720 2023-11-16T16:56:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.981+0000 7f90af7fe700 1 -- 172.21.15.2:0/4174077653 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f90b409ecf0 con 0x5593b1467720 2023-11-16T16:56:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.981+0000 7f90af7fe700 1 -- 172.21.15.2:0/4174077653 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b409a410 con 0x5593b1467720 2023-11-16T16:56:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.982+0000 7f90deeb0b80 1 -- 172.21.15.2:0/4174077653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467720 msgr2=0x5593b1461560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.982+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/4174077653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467720 0x5593b1461560 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f90b40cb2b0 tx=0x7f90b409a800 comp rx=0 tx=0).stop 2023-11-16T16:56:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.982+0000 7f90deeb0b80 1 -- 172.21.15.2:0/4174077653 shutdown_connections 2023-11-16T16:56:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.982+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/4174077653 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b1467b90 0x5593b1461b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.982+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/4174077653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b12bd6b0 0x5593b14671e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.982+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/4174077653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467720 0x5593b1461560 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.982+0000 7f90deeb0b80 1 -- 172.21.15.2:0/4174077653 >> 172.21.15.2:0/4174077653 conn(0x5593b1456f50 msgr2=0x5593b145c270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.982+0000 7f90deeb0b80 1 -- 172.21.15.2:0/4174077653 shutdown_connections 2023-11-16T16:56:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.982+0000 7f90deeb0b80 1 -- 172.21.15.2:0/4174077653 wait complete. 2023-11-16T16:56:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.983+0000 7f90deeb0b80 1 -- start start 2023-11-16T16:56:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.983+0000 7f90deeb0b80 1 Processor -- start 2023-11-16T16:56:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.983+0000 7f90deeb0b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b12bd6b0 0x5593b1470760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.983+0000 7f90deeb0b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b1467720 0x5593b1470ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.983+0000 7f90deeb0b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467b90 0x5593b146a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.983+0000 7f90deeb0b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593b1354880 con 0x5593b1467720 2023-11-16T16:56:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.983+0000 7f90deeb0b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593b1354520 con 0x5593b12bd6b0 2023-11-16T16:56:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.983+0000 7f90deeb0b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593b12368c0 con 0x5593b1467b90 2023-11-16T16:56:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.983+0000 7f90deeb0b80 10 client.0 _ll_register_callbacks cb 0x5593b1372720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:56:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.984+0000 7f90c8b9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b1467720 0x5593b1470ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.984+0000 7f90c8b9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b1467720 0x5593b1470ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49482/0 (socket says 172.21.15.2:49482) 2023-11-16T16:56:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.984+0000 7f90c8b9a700 1 -- 172.21.15.2:0/3329661713 learned_addr learned my addr 172.21.15.2:0/3329661713 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:56:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.984+0000 7f90c939b700 1 --2- 172.21.15.2:0/3329661713 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b12bd6b0 0x5593b1470760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.984+0000 7f90c9b9c700 1 --2- 172.21.15.2:0/3329661713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467b90 0x5593b146a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.984+0000 7f90c8b9a700 1 -- 172.21.15.2:0/3329661713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467b90 msgr2=0x5593b146a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.984+0000 7f90c8b9a700 1 --2- 172.21.15.2:0/3329661713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467b90 0x5593b146a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.984+0000 7f90c8b9a700 1 -- 172.21.15.2:0/3329661713 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b12bd6b0 msgr2=0x5593b1470760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.984+0000 7f90c8b9a700 1 --2- 172.21.15.2:0/3329661713 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b12bd6b0 0x5593b1470760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.984+0000 7f90c8b9a700 1 -- 172.21.15.2:0/3329661713 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c41101e0 con 0x5593b1467720 2023-11-16T16:56:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.984+0000 7f90c9b9c700 1 --2- 172.21.15.2:0/3329661713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467b90 0x5593b146a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:56:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.984+0000 7f90c939b700 1 --2- 172.21.15.2:0/3329661713 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b12bd6b0 0x5593b1470760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:56:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.984+0000 7f90c8b9a700 1 --2- 172.21.15.2:0/3329661713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b1467720 0x5593b1470ca0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f90b40971c0 tx=0x7f90b40a3080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab053eaf1eae68d server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.984+0000 7f90acff9700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:56:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.984+0000 7f90acff9700 1 -- 172.21.15.2:0/3329661713 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b409b120 con 0x5593b1467720 2023-11-16T16:56:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.985+0000 7f90deeb0b80 1 -- 172.21.15.2:0/3329661713 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5593b1454710 con 0x5593b1467720 2023-11-16T16:56:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.985+0000 7f90acff9700 1 -- 172.21.15.2:0/3329661713 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f90b40a2bf0 con 0x5593b1467720 2023-11-16T16:56:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.985+0000 7f90acff9700 1 -- 172.21.15.2:0/3329661713 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90b409ad40 con 0x5593b1467720 2023-11-16T16:56:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.985+0000 7f90deeb0b80 10 client.15870 fetch_fsmap learned FSMap version 35 2023-11-16T16:56:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.985+0000 7f90deeb0b80 1 -- 172.21.15.2:0/3329661713 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x5593b1379130 con 0x5593b1467720 2023-11-16T16:56:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.985+0000 7f90acff9700 1 -- 172.21.15.2:0/3329661713 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f90b407d080 con 0x5593b1467720 2023-11-16T16:56:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.985+0000 7f90acff9700 1 -- 172.21.15.2:0/3329661713 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f90b4079c20 con 0x5593b1467720 2023-11-16T16:56:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.985+0000 7f90deeb0b80 10 client.15870 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:56:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.985+0000 7f90deeb0b80 10 client.15870 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:56:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.985+0000 7f90deeb0b80 20 client.15870 populate_metadata read hostname 'smithi002' 2023-11-16T16:56:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.985+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/3329661713 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5593b147a6e0 0x5593b147cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.985+0000 7f90deeb0b80 4 client.15870 mds_command: new command op to 24590 tid=0[{"prefix": "get_command_descriptions"}] 2023-11-16T16:56:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.985+0000 7f90deeb0b80 1 -- 172.21.15.2:0/3329661713 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5593b1375040 con 0x5593b147a6e0 2023-11-16T16:56:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.986+0000 7f90c939b700 1 --2- 172.21.15.2:0/3329661713 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5593b147a6e0 0x5593b147cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.986+0000 7f90c939b700 1 --2- 172.21.15.2:0/3329661713 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5593b147a6e0 0x5593b147cac0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e88cf0b9d7f042cb server_cookie=7d7789f4f02808c3 in_seq=0 out_seq=0 2023-11-16T16:56:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.986+0000 7f90acff9700 10 client.15870 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:56:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.987+0000 7f90acff9700 1 -- 172.21.15.2:0/3329661713 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5593b1375040 con 0x5593b147a6e0 2023-11-16T16:56:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.987+0000 7f90acff9700 10 client.15870 handle_command_reply: tid=0 2023-11-16T16:56:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.987+0000 7f90deeb0b80 1 client.15870 shutdown 2023-11-16T16:56:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.987+0000 7f90deeb0b80 1 -- 172.21.15.2:0/3329661713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b1467720 msgr2=0x5593b1470ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.987+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/3329661713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b1467720 0x5593b1470ca0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f90b40971c0 tx=0x7f90b40a3080 comp rx=0 tx=0).stop 2023-11-16T16:56:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.987+0000 7f90deeb0b80 1 -- 172.21.15.2:0/3329661713 shutdown_connections 2023-11-16T16:56:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.987+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/3329661713 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5593b147a6e0 0x5593b147cac0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.987+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/3329661713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467b90 0x5593b146a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.987+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/3329661713 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b12bd6b0 0x5593b1470760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.987+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/3329661713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b1467720 0x5593b1470ca0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.987+0000 7f90deeb0b80 1 -- 172.21.15.2:0/3329661713 >> 172.21.15.2:0/3329661713 conn(0x5593b1456f50 msgr2=0x5593b145b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.987+0000 7f90acff9700 0 client.15870 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:56:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.988+0000 7f90deeb0b80 1 -- 172.21.15.2:0/3329661713 shutdown_connections 2023-11-16T16:56:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.988+0000 7f90deeb0b80 1 -- 172.21.15.2:0/3329661713 wait complete. 2023-11-16T16:56:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:09.988+0000 7f90deeb0b80 20 client.15870 trim_cache size 0 max 16384 2023-11-16T16:56:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.003+0000 7f90deeb0b80 1 Processor -- start 2023-11-16T16:56:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.003+0000 7f90deeb0b80 1 -- start start 2023-11-16T16:56:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.003+0000 7f90deeb0b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b12bd6b0 0x5593b145bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.003+0000 7f90deeb0b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467720 0x5593b1467f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.003+0000 7f90deeb0b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b1467b90 0x5593b14684d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.003+0000 7f90deeb0b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593b1456b70 con 0x5593b12bd6b0 2023-11-16T16:56:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.003+0000 7f90deeb0b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593b145dc50 con 0x5593b1467b90 2023-11-16T16:56:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.003+0000 7f90deeb0b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593b1465630 con 0x5593b1467720 2023-11-16T16:56:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.004+0000 7f90c939b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b12bd6b0 0x5593b145bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.004+0000 7f90c939b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b12bd6b0 0x5593b145bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49488/0 (socket says 172.21.15.2:49488) 2023-11-16T16:56:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.004+0000 7f90c939b700 1 -- 172.21.15.2:0/349167962 learned_addr learned my addr 172.21.15.2:0/349167962 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:56:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.004+0000 7f90c9b9c700 1 --2- 172.21.15.2:0/349167962 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b1467b90 0x5593b14684d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.004+0000 7f90c8b9a700 1 --2- 172.21.15.2:0/349167962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467720 0x5593b1467f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.004+0000 7f90c939b700 1 -- 172.21.15.2:0/349167962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467720 msgr2=0x5593b1467f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.004+0000 7f90c939b700 1 --2- 172.21.15.2:0/349167962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467720 0x5593b1467f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.004+0000 7f90c939b700 1 -- 172.21.15.2:0/349167962 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b1467b90 msgr2=0x5593b14684d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.004+0000 7f90c939b700 1 --2- 172.21.15.2:0/349167962 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b1467b90 0x5593b14684d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.004+0000 7f90c939b700 1 -- 172.21.15.2:0/349167962 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593b14586f0 con 0x5593b12bd6b0 2023-11-16T16:56:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.004+0000 7f90c939b700 1 --2- 172.21.15.2:0/349167962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b12bd6b0 0x5593b145bf60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f90c0009fd0 tx=0x7f90c001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae8403f380317f16 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.005+0000 7f90affff700 1 -- 172.21.15.2:0/349167962 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c0018040 con 0x5593b12bd6b0 2023-11-16T16:56:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.005+0000 7f90affff700 1 -- 172.21.15.2:0/349167962 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f90c0004420 con 0x5593b12bd6b0 2023-11-16T16:56:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.005+0000 7f90affff700 1 -- 172.21.15.2:0/349167962 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c0004920 con 0x5593b12bd6b0 2023-11-16T16:56:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.005+0000 7f90deeb0b80 1 -- 172.21.15.2:0/349167962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b12bd6b0 msgr2=0x5593b145bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.005+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/349167962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b12bd6b0 0x5593b145bf60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f90c0009fd0 tx=0x7f90c001f000 comp rx=0 tx=0).stop 2023-11-16T16:56:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.005+0000 7f90deeb0b80 1 -- 172.21.15.2:0/349167962 shutdown_connections 2023-11-16T16:56:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.005+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/349167962 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b1467b90 0x5593b14684d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.005+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/349167962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b12bd6b0 0x5593b145bf60 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.005+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/349167962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467720 0x5593b1467f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.005+0000 7f90deeb0b80 1 -- 172.21.15.2:0/349167962 >> 172.21.15.2:0/349167962 conn(0x5593b1456f50 msgr2=0x5593b146b340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.005+0000 7f90deeb0b80 1 -- 172.21.15.2:0/349167962 shutdown_connections 2023-11-16T16:56:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.005+0000 7f90deeb0b80 1 -- 172.21.15.2:0/349167962 wait complete. 2023-11-16T16:56:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.006+0000 7f90deeb0b80 1 -- start start 2023-11-16T16:56:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.006+0000 7f90deeb0b80 1 Processor -- start 2023-11-16T16:56:10.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.006+0000 7f90deeb0b80 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b12bd6b0 0x5593b1470bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.007+0000 7f90deeb0b80 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b1467720 0x5593b146e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.007+0000 7f90deeb0b80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467b90 0x5593b146e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.007+0000 7f90deeb0b80 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593b146f090 con 0x5593b1467720 2023-11-16T16:56:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.007+0000 7f90deeb0b80 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593b14cd500 con 0x5593b12bd6b0 2023-11-16T16:56:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.007+0000 7f90deeb0b80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593b14cd640 con 0x5593b1467b90 2023-11-16T16:56:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.007+0000 7f90deeb0b80 10 client.0 _ll_register_callbacks cb 0x7f90c0009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-16T16:56:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.007+0000 7f90c8b9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b1467720 0x5593b146e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.007+0000 7f90c8b9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b1467720 0x5593b146e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49494/0 (socket says 172.21.15.2:49494) 2023-11-16T16:56:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.007+0000 7f90c8b9a700 1 -- 172.21.15.2:0/4066391691 learned_addr learned my addr 172.21.15.2:0/4066391691 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:56:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.007+0000 7f90c939b700 1 --2- 172.21.15.2:0/4066391691 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b12bd6b0 0x5593b1470bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.007+0000 7f90c9b9c700 1 --2- 172.21.15.2:0/4066391691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467b90 0x5593b146e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.007+0000 7f90c8b9a700 1 -- 172.21.15.2:0/4066391691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467b90 msgr2=0x5593b146e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.007+0000 7f90c8b9a700 1 --2- 172.21.15.2:0/4066391691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467b90 0x5593b146e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.007+0000 7f90c8b9a700 1 -- 172.21.15.2:0/4066391691 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b12bd6b0 msgr2=0x5593b1470bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-16T16:56:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.007+0000 7f90c8b9a700 1 --2- 172.21.15.2:0/4066391691 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b12bd6b0 0x5593b1470bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.007+0000 7f90c8b9a700 1 -- 172.21.15.2:0/4066391691 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c000fd40 con 0x5593b1467720 2023-11-16T16:56:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.007+0000 7f90c939b700 1 --2- 172.21.15.2:0/4066391691 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b12bd6b0 0x5593b1470bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T16:56:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.007+0000 7f90c8b9a700 1 --2- 172.21.15.2:0/4066391691 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b1467720 0x5593b146e2d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f90b409a9c0 tx=0x7f90b40a2e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1203ea113613001 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:56:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.007+0000 7f90acff9700 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2023-11-16T16:56:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.008+0000 7f90acff9700 1 -- 172.21.15.2:0/4066391691 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b400fe60 con 0x5593b1467720 2023-11-16T16:56:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.008+0000 7f90acff9700 1 -- 172.21.15.2:0/4066391691 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f90b40cb170 con 0x5593b1467720 2023-11-16T16:56:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.008+0000 7f90deeb0b80 1 -- 172.21.15.2:0/4066391691 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f90b409ecf0 con 0x5593b1467720 2023-11-16T16:56:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.008+0000 7f90acff9700 1 -- 172.21.15.2:0/4066391691 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90b409a800 con 0x5593b1467720 2023-11-16T16:56:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.008+0000 7f90deeb0b80 10 client.15876 fetch_fsmap learned FSMap version 35 2023-11-16T16:56:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.008+0000 7f90deeb0b80 1 -- 172.21.15.2:0/4066391691 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x5593b11e0d80 con 0x5593b1467720 2023-11-16T16:56:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.008+0000 7f90acff9700 1 -- 172.21.15.2:0/4066391691 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f90b409ecf0 con 0x5593b1467720 2023-11-16T16:56:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.008+0000 7f90acff9700 1 -- 172.21.15.2:0/4066391691 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 35) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f90b40b3170 con 0x5593b1467720 2023-11-16T16:56:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.008+0000 7f90deeb0b80 10 client.15876 fetch_fsmap finished waiting for FSMap version 35 2023-11-16T16:56:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.008+0000 7f90deeb0b80 10 client.15876 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-11-16T16:56:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.008+0000 7f90deeb0b80 20 client.15876 populate_metadata read hostname 'smithi002' 2023-11-16T16:56:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.008+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/4066391691 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5593b14ce830 0x5593b14d0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:56:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.008+0000 7f90deeb0b80 4 client.15876 mds_command: new command op to 24590 tid=0[{"prefix": "flush journal"}] 2023-11-16T16:56:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.008+0000 7f90deeb0b80 1 -- 172.21.15.2:0/4066391691 --> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5593b1381600 con 0x5593b14ce830 2023-11-16T16:56:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.009+0000 7f90c939b700 1 --2- 172.21.15.2:0/4066391691 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5593b14ce830 0x5593b14d0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:56:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.009+0000 7f90c939b700 1 --2- 172.21.15.2:0/4066391691 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5593b14ce830 0x5593b14d0c10 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=928715af7b277513 server_cookie=79c06e96aa3901c8 in_seq=0 out_seq=0 2023-11-16T16:56:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.009+0000 7f90acff9700 10 client.15876 ms_handle_connect on v2:172.21.15.141:6838/3916358272 2023-11-16T16:56:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.018+0000 7f90acff9700 1 -- 172.21.15.2:0/4066391691 <== mds.0 v2:172.21.15.141:6838/3916358272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5593b1381600 con 0x5593b14ce830 2023-11-16T16:56:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.018+0000 7f90acff9700 10 client.15876 handle_command_reply: tid=0 2023-11-16T16:56:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.018+0000 7f90deeb0b80 1 client.15876 shutdown 2023-11-16T16:56:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.018+0000 7f90deeb0b80 1 -- 172.21.15.2:0/4066391691 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b1467720 msgr2=0x5593b146e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.018+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/4066391691 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b1467720 0x5593b146e2d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f90b409a9c0 tx=0x7f90b40a2e00 comp rx=0 tx=0).stop 2023-11-16T16:56:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.018+0000 7f90deeb0b80 1 -- 172.21.15.2:0/4066391691 shutdown_connections 2023-11-16T16:56:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.018+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/4066391691 >> [v2:172.21.15.141:6838/3916358272,v1:172.21.15.141:6839/3916358272] conn(0x5593b14ce830 0x5593b14d0c10 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.018+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/4066391691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5593b1467b90 0x5593b146e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.018+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/4066391691 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x5593b12bd6b0 0x5593b1470bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.018+0000 7f90deeb0b80 1 --2- 172.21.15.2:0/4066391691 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5593b1467720 0x5593b146e2d0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.018+0000 7f90acff9700 0 client.15876 ms_handle_reset on v2:172.21.15.141:6838/3916358272 2023-11-16T16:56:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.018+0000 7f90deeb0b80 1 -- 172.21.15.2:0/4066391691 >> 172.21.15.2:0/4066391691 conn(0x5593b1456f50 msgr2=0x5593b1467f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.018+0000 7f90deeb0b80 1 -- 172.21.15.2:0/4066391691 shutdown_connections 2023-11-16T16:56:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.018+0000 7f90deeb0b80 1 -- 172.21.15.2:0/4066391691 wait complete. 2023-11-16T16:56:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.018+0000 7f90deeb0b80 20 client.15876 trim_cache size 0 max 16384 2023-11-16T16:56:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2023-11-16T16:56:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2023-11-16T16:56:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2023-11-16T16:56:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2023-11-16T16:56:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.019+0000 7f90cb5ff700 1 -- 172.21.15.2:0/3625621023 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f90b00a1340 msgr2=0x7f90b00a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.019+0000 7f90cb5ff700 1 --2- 172.21.15.2:0/3625621023 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f90b00a1340 0x7f90b00a37f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f90c00139a0 tx=0x7f90c0013d60 comp rx=0 tx=0).stop 2023-11-16T16:56:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.019+0000 7f90cb5ff700 1 -- 172.21.15.2:0/3625621023 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90c41015f0 msgr2=0x7f90c4073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:56:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.019+0000 7f90cb5ff700 1 --2- 172.21.15.2:0/3625621023 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90c41015f0 0x7f90c4073330 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f90b40098e0 tx=0x7f90b400f690 comp rx=0 tx=0).stop 2023-11-16T16:56:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.019+0000 7f90cb5ff700 1 -- 172.21.15.2:0/3625621023 shutdown_connections 2023-11-16T16:56:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.019+0000 7f90cb5ff700 1 --2- 172.21.15.2:0/3625621023 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f90b00a1340 0x7f90b00a37f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.019+0000 7f90cb5ff700 1 --2- 172.21.15.2:0/3625621023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90c4100c50 0x7f90c4072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.019+0000 7f90cb5ff700 1 --2- 172.21.15.2:0/3625621023 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f90c4105560 0x7f90c41a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.019+0000 7f90cb5ff700 1 --2- 172.21.15.2:0/3625621023 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90c41015f0 0x7f90c4073330 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:56:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.019+0000 7f90cb5ff700 1 -- 172.21.15.2:0/3625621023 >> 172.21.15.2:0/3625621023 conn(0x7f90c4077550 msgr2=0x7f90c40fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:56:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.019+0000 7f90cb5ff700 1 -- 172.21.15.2:0/3625621023 shutdown_connections 2023-11-16T16:56:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2023-11-16T16:56:10.019+0000 7f90cb5ff700 1 -- 172.21.15.2:0/3625621023 wait complete. 2023-11-16T16:56:10.363 INFO:tasks.workunit.client.0.smithi002.stdout:ok 2023-11-16T16:56:10.368 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-16T16:56:10.368 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-11-16T16:56:10.455 INFO:tasks.workunit:Running workunit fs/misc/general_vxattrs.sh... 2023-11-16T16:56:10.455 DEBUG:teuthology.orchestra.run.smithi002:workunit test fs/misc/general_vxattrs.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh 2023-11-16T16:56:10.528 INFO:tasks.workunit.client.0.smithi002.stderr:+ mkdir -p dir 2023-11-16T16:56:10.533 INFO:tasks.workunit.client.0.smithi002.stderr:+ getfattr -n ceph.dir.pin dir 2023-11-16T16:56:10.533 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep 'ceph.dir.pin="-1"' 2023-11-16T16:56:10.536 INFO:tasks.workunit.client.0.smithi002.stdout:ceph.dir.pin="-1" 2023-11-16T16:56:10.536 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n ceph.dir.pin dir 2023-11-16T16:56:10.537 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep 'setfattr: dir: Invalid argument' 2023-11-16T16:56:10.538 INFO:tasks.workunit.client.0.smithi002.stdout:setfattr: dir: Invalid argument 2023-11-16T16:56:10.539 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n ceph.dir.pin -v 1 dir 2023-11-16T16:56:10.543 INFO:tasks.workunit.client.0.smithi002.stderr:+ getfattr -n ceph.dir.pin dir 2023-11-16T16:56:10.543 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep 'ceph.dir.pin="1"' 2023-11-16T16:56:10.546 INFO:tasks.workunit.client.0.smithi002.stdout:ceph.dir.pin="1" 2023-11-16T16:56:10.546 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -x ceph.dir.pin dir 2023-11-16T16:56:10.550 INFO:tasks.workunit.client.0.smithi002.stderr:+ getfattr -n ceph.dir.pin dir 2023-11-16T16:56:10.550 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep 'ceph.dir.pin="-1"' 2023-11-16T16:56:10.552 INFO:tasks.workunit.client.0.smithi002.stdout:ceph.dir.pin="-1" 2023-11-16T16:56:10.553 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n ceph.dir.pin.distributed dir 2023-11-16T16:56:10.554 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep 'setfattr: dir: Invalid argument' 2023-11-16T16:56:10.555 INFO:tasks.workunit.client.0.smithi002.stdout:setfattr: dir: Invalid argument 2023-11-16T16:56:10.556 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n ceph.dir.pin.distributed -v 1 dir 2023-11-16T16:56:10.559 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -x ceph.dir.pin.distributed dir 2023-11-16T16:56:10.563 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n ceph.dir.pin.random dir 2023-11-16T16:56:10.563 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep 'setfattr: dir: Invalid argument' 2023-11-16T16:56:10.565 INFO:tasks.workunit.client.0.smithi002.stdout:setfattr: dir: Invalid argument 2023-11-16T16:56:10.565 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n ceph.dir.pin.random -v 0.01 dir 2023-11-16T16:56:10.568 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -x ceph.dir.pin.random dir 2023-11-16T16:56:10.572 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n ceph.dir.subvolume dir 2023-11-16T16:56:10.572 INFO:tasks.workunit.client.0.smithi002.stderr:+ grep 'setfattr: dir: Invalid argument' 2023-11-16T16:56:10.574 INFO:tasks.workunit.client.0.smithi002.stdout:setfattr: dir: Invalid argument 2023-11-16T16:56:10.574 INFO:tasks.workunit.client.0.smithi002.stderr:+ setfattr -n ceph.dir.subvolume -v 1 dir 2023-11-16T16:56:10.585 INFO:tasks.workunit.client.0.smithi002.stderr:setfattr: dir: Invalid argument 2023-11-16T16:56:10.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T16:56:10.586 INFO:tasks.workunit:Stopping ['fs/misc'] on client.0... 2023-11-16T16:56:10.586 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-11-16T16:56:11.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:10 smithi002 ceph-mon[128712]: pgmap v191: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 87 KiB/s wr, 198 op/s 2023-11-16T16:56:11.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:10 smithi141 ceph-mon[131184]: pgmap v191: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 87 KiB/s wr, 198 op/s 2023-11-16T16:56:11.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:10 smithi155 ceph-mon[131607]: pgmap v191: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 87 KiB/s wr, 198 op/s 2023-11-16T16:56:12.081 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_5e4a743ad7fd1422ebd6c4dc950a21a303279207/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_5e4a743ad7fd1422ebd6c4dc950a21a303279207/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_5e4a743ad7fd1422ebd6c4dc950a21a303279207/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi002 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh' 2023-11-16T16:56:12.154 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f9d9071e83e849048fc413a6cc6d58cd Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_5e4a743ad7fd1422ebd6c4dc950a21a303279207/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_5e4a743ad7fd1422ebd6c4dc950a21a303279207/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_5e4a743ad7fd1422ebd6c4dc950a21a303279207/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi002 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh' 2023-11-16T16:56:12.157 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2023-11-16T16:56:12.195 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T16:56:12.200 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:11 smithi002 ceph-mon[128712]: osdmap e96: 12 total, 12 up, 12 in 2023-11-16T16:56:12.207 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2023-11-16T16:56:12.216 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2023-11-16T16:56:12.216 INFO:teuthology.task.exec:Executing custom commands... 2023-11-16T16:56:12.216 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T16:56:12.217 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2023-11-16T16:56:12.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:11 smithi141 ceph-mon[131184]: osdmap e96: 12 total, 12 up, 12 in 2023-11-16T16:56:12.334 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2023-11-16T16:56:12.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:11 smithi155 ceph-mon[131607]: osdmap e96: 12 total, 12 up, 12 in 2023-11-16T16:56:13.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:12 smithi002 ceph-mon[128712]: pgmap v193: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 13 KiB/s wr, 10 op/s 2023-11-16T16:56:13.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:12 smithi141 ceph-mon[131184]: pgmap v193: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 13 KiB/s wr, 10 op/s 2023-11-16T16:56:13.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:12 smithi155 ceph-mon[131607]: pgmap v193: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 13 KiB/s wr, 10 op/s 2023-11-16T16:56:15.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:14 smithi002 ceph-mon[128712]: pgmap v194: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 13 KiB/s wr, 10 op/s 2023-11-16T16:56:15.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:14 smithi141 ceph-mon[131184]: pgmap v194: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 13 KiB/s wr, 10 op/s 2023-11-16T16:56:15.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:14 smithi155 ceph-mon[131607]: pgmap v194: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 13 KiB/s wr, 10 op/s 2023-11-16T16:56:17.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:16 smithi002 ceph-mon[128712]: pgmap v195: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 13 KiB/s wr, 10 op/s 2023-11-16T16:56:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:16 smithi141 ceph-mon[131184]: pgmap v195: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 13 KiB/s wr, 10 op/s 2023-11-16T16:56:17.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:16 smithi155 ceph-mon[131607]: pgmap v195: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 13 KiB/s wr, 10 op/s 2023-11-16T16:56:19.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:18 smithi002 ceph-mon[128712]: pgmap v196: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-11-16T16:56:19.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:18 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:56:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:18 smithi141 ceph-mon[131184]: pgmap v196: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-11-16T16:56:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:18 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:56:19.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:18 smithi155 ceph-mon[131607]: pgmap v196: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-11-16T16:56:19.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:18 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:56:20.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:19 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:56:20.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:19 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:56:20.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:19 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:56:20.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:19 smithi002 ceph-mon[128712]: osdmap e97: 12 total, 12 up, 12 in 2023-11-16T16:56:20.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:19 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:56:20.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:19 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:56:20.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:19 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:56:20.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:19 smithi141 ceph-mon[131184]: osdmap e97: 12 total, 12 up, 12 in 2023-11-16T16:56:20.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:19 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:56:20.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:19 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:56:20.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:19 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:56:20.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:19 smithi155 ceph-mon[131607]: osdmap e97: 12 total, 12 up, 12 in 2023-11-16T16:56:21.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:20 smithi002 ceph-mon[128712]: pgmap v197: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-11-16T16:56:21.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:20 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:56:21.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:20 smithi002 ceph-mon[128712]: Service mds.all has no related daemon already placed. 2023-11-16T16:56:21.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:20 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:56:21.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:20 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:56:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:20 smithi141 ceph-mon[131184]: pgmap v197: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-11-16T16:56:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:20 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:56:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:20 smithi141 ceph-mon[131184]: Service mds.all has no related daemon already placed. 2023-11-16T16:56:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:20 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:56:21.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:20 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:56:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:20 smithi155 ceph-mon[131607]: pgmap v197: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-11-16T16:56:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:20 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:56:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:20 smithi155 ceph-mon[131607]: Service mds.all has no related daemon already placed. 2023-11-16T16:56:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:20 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:56:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:20 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:56:23.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:22 smithi002 ceph-mon[128712]: pgmap v199: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.3 KiB/s wr, 8 op/s 2023-11-16T16:56:23.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:22 smithi141 ceph-mon[131184]: pgmap v199: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.3 KiB/s wr, 8 op/s 2023-11-16T16:56:23.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:22 smithi155 ceph-mon[131607]: pgmap v199: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.3 KiB/s wr, 8 op/s 2023-11-16T16:56:25.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:24 smithi002 ceph-mon[128712]: pgmap v200: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.3 KiB/s wr, 8 op/s 2023-11-16T16:56:25.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:24 smithi141 ceph-mon[131184]: pgmap v200: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.3 KiB/s wr, 8 op/s 2023-11-16T16:56:25.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:24 smithi155 ceph-mon[131607]: pgmap v200: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.3 KiB/s wr, 8 op/s 2023-11-16T16:56:27.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:26 smithi141 ceph-mon[131184]: pgmap v201: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.3 KiB/s wr, 8 op/s 2023-11-16T16:56:27.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:26 smithi155 ceph-mon[131607]: pgmap v201: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.3 KiB/s wr, 8 op/s 2023-11-16T16:56:27.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:26 smithi002 ceph-mon[128712]: pgmap v201: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.3 KiB/s wr, 8 op/s 2023-11-16T16:56:29.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:28 smithi141 ceph-mon[131184]: pgmap v202: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.6 KiB/s wr, 7 op/s 2023-11-16T16:56:29.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:28 smithi155 ceph-mon[131607]: pgmap v202: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.6 KiB/s wr, 7 op/s 2023-11-16T16:56:29.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:28 smithi002 ceph-mon[128712]: pgmap v202: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.6 KiB/s wr, 7 op/s 2023-11-16T16:56:31.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:30 smithi141 ceph-mon[131184]: pgmap v203: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.6 KiB/s wr, 7 op/s 2023-11-16T16:56:31.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:30 smithi155 ceph-mon[131607]: pgmap v203: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.6 KiB/s wr, 7 op/s 2023-11-16T16:56:31.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:30 smithi002 ceph-mon[128712]: pgmap v203: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.6 KiB/s wr, 7 op/s 2023-11-16T16:56:33.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:32 smithi141 ceph-mon[131184]: pgmap v204: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.9 KiB/s wr, 9 op/s 2023-11-16T16:56:33.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:32 smithi155 ceph-mon[131607]: pgmap v204: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.9 KiB/s wr, 9 op/s 2023-11-16T16:56:33.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:32 smithi002 ceph-mon[128712]: pgmap v204: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.9 KiB/s wr, 9 op/s 2023-11-16T16:56:35.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:34 smithi141 ceph-mon[131184]: pgmap v205: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2023-11-16T16:56:35.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:34 smithi155 ceph-mon[131607]: pgmap v205: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2023-11-16T16:56:35.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:34 smithi002 ceph-mon[128712]: pgmap v205: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2023-11-16T16:56:37.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:36 smithi141 ceph-mon[131184]: pgmap v206: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2023-11-16T16:56:37.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:36 smithi155 ceph-mon[131607]: pgmap v206: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2023-11-16T16:56:37.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:36 smithi002 ceph-mon[128712]: pgmap v206: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2023-11-16T16:56:39.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:38 smithi141 ceph-mon[131184]: pgmap v207: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2023-11-16T16:56:39.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:38 smithi155 ceph-mon[131607]: pgmap v207: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2023-11-16T16:56:39.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:38 smithi002 ceph-mon[128712]: pgmap v207: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2023-11-16T16:56:41.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:40 smithi141 ceph-mon[131184]: pgmap v208: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2023-11-16T16:56:41.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:40 smithi155 ceph-mon[131607]: pgmap v208: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2023-11-16T16:56:41.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:40 smithi002 ceph-mon[128712]: pgmap v208: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2023-11-16T16:56:43.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:42 smithi141 ceph-mon[131184]: pgmap v209: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2023-11-16T16:56:43.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:42 smithi155 ceph-mon[131607]: pgmap v209: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2023-11-16T16:56:43.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:42 smithi002 ceph-mon[128712]: pgmap v209: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2023-11-16T16:56:45.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:44 smithi141 ceph-mon[131184]: pgmap v210: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 5 op/s 2023-11-16T16:56:45.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:44 smithi155 ceph-mon[131607]: pgmap v210: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 5 op/s 2023-11-16T16:56:45.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:44 smithi002 ceph-mon[128712]: pgmap v210: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 5 op/s 2023-11-16T16:56:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:46 smithi141 ceph-mon[131184]: pgmap v211: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 5 op/s 2023-11-16T16:56:47.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:46 smithi155 ceph-mon[131607]: pgmap v211: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 5 op/s 2023-11-16T16:56:47.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:46 smithi002 ceph-mon[128712]: pgmap v211: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1023 B/s wr, 5 op/s 2023-11-16T16:56:49.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:48 smithi141 ceph-mon[131184]: pgmap v212: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1023 B/s wr, 8 op/s 2023-11-16T16:56:49.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:48 smithi155 ceph-mon[131607]: pgmap v212: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1023 B/s wr, 8 op/s 2023-11-16T16:56:49.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:48 smithi002 ceph-mon[128712]: pgmap v212: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1023 B/s wr, 8 op/s 2023-11-16T16:56:51.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:50 smithi141 ceph-mon[131184]: pgmap v213: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T16:56:51.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:50 smithi155 ceph-mon[131607]: pgmap v213: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T16:56:51.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:50 smithi002 ceph-mon[128712]: pgmap v213: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T16:56:53.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:52 smithi141 ceph-mon[131184]: pgmap v214: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2023-11-16T16:56:53.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:52 smithi155 ceph-mon[131607]: pgmap v214: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2023-11-16T16:56:53.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:52 smithi002 ceph-mon[128712]: pgmap v214: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2023-11-16T16:56:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:54 smithi141 ceph-mon[131184]: pgmap v215: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:56:55.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:54 smithi155 ceph-mon[131607]: pgmap v215: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:56:55.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:54 smithi002 ceph-mon[128712]: pgmap v215: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:56:57.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:56 smithi141 ceph-mon[131184]: pgmap v216: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:56:57.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:56 smithi155 ceph-mon[131607]: pgmap v216: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:56:57.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:56 smithi002 ceph-mon[128712]: pgmap v216: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:56:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:56:58 smithi141 ceph-mon[131184]: pgmap v217: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:56:59.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:56:58 smithi155 ceph-mon[131607]: pgmap v217: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:56:59.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:56:58 smithi002 ceph-mon[128712]: pgmap v217: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:57:01.269 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:00 smithi141 ceph-mon[131184]: pgmap v218: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:01.372 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:00 smithi155 ceph-mon[131607]: pgmap v218: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:01.408 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:00 smithi002 ceph-mon[128712]: pgmap v218: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:03.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:02 smithi141 ceph-mon[131184]: pgmap v219: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-11-16T16:57:03.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:02 smithi155 ceph-mon[131607]: pgmap v219: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-11-16T16:57:03.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:02 smithi002 ceph-mon[128712]: pgmap v219: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-11-16T16:57:05.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:04 smithi141 ceph-mon[131184]: pgmap v220: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:57:05.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:04 smithi155 ceph-mon[131607]: pgmap v220: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:57:05.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:04 smithi002 ceph-mon[128712]: pgmap v220: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:57:07.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:06 smithi141 ceph-mon[131184]: pgmap v221: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:57:07.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:06 smithi155 ceph-mon[131607]: pgmap v221: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:57:07.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:06 smithi002 ceph-mon[128712]: pgmap v221: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:57:09.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:08 smithi141 ceph-mon[131184]: pgmap v222: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-11-16T16:57:09.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:08 smithi155 ceph-mon[131607]: pgmap v222: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-11-16T16:57:09.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:08 smithi002 ceph-mon[128712]: pgmap v222: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-11-16T16:57:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:10 smithi141 ceph-mon[131184]: pgmap v223: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:57:11.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:10 smithi155 ceph-mon[131607]: pgmap v223: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:57:11.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:10 smithi002 ceph-mon[128712]: pgmap v223: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:57:12.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:11 smithi141 ceph-mon[131184]: osdmap e98: 12 total, 12 up, 12 in 2023-11-16T16:57:12.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:12 smithi155 ceph-mon[131607]: osdmap e98: 12 total, 12 up, 12 in 2023-11-16T16:57:12.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:11 smithi002 ceph-mon[128712]: osdmap e98: 12 total, 12 up, 12 in 2023-11-16T16:57:13.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:13 smithi141 ceph-mon[131184]: pgmap v225: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:13.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:13 smithi155 ceph-mon[131607]: pgmap v225: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:13.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:13 smithi002 ceph-mon[128712]: pgmap v225: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:15 smithi141 ceph-mon[131184]: pgmap v226: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:15.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:15 smithi155 ceph-mon[131607]: pgmap v226: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:15.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:15 smithi002 ceph-mon[128712]: pgmap v226: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:17 smithi141 ceph-mon[131184]: pgmap v227: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:17.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:17 smithi155 ceph-mon[131607]: pgmap v227: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:17.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:17 smithi002 ceph-mon[128712]: pgmap v227: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:19.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:19 smithi141 ceph-mon[131184]: pgmap v228: 129 pgs: 129 active+clean; 607 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:19.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:19 smithi155 ceph-mon[131607]: pgmap v228: 129 pgs: 129 active+clean; 607 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:19.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:19 smithi002 ceph-mon[128712]: pgmap v228: 129 pgs: 129 active+clean; 607 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:20.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:20 smithi002 ceph-mon[128712]: pgmap v229: 129 pgs: 129 active+clean; 607 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:20.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:20 smithi002 ceph-mon[128712]: osdmap e99: 12 total, 12 up, 12 in 2023-11-16T16:57:20.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:20 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:57:21.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:20 smithi141 ceph-mon[131184]: pgmap v229: 129 pgs: 129 active+clean; 607 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:21.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:20 smithi141 ceph-mon[131184]: osdmap e99: 12 total, 12 up, 12 in 2023-11-16T16:57:21.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:20 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:57:21.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:20 smithi155 ceph-mon[131607]: pgmap v229: 129 pgs: 129 active+clean; 607 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:21.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:20 smithi155 ceph-mon[131607]: osdmap e99: 12 total, 12 up, 12 in 2023-11-16T16:57:21.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:20 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:57:22.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:21 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:57:22.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:21 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:57:22.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:21 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:57:22.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:21 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:57:22.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:21 smithi141 ceph-mon[131184]: Service mds.all has no related daemon already placed. 2023-11-16T16:57:22.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:21 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:57:22.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:21 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:57:22.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:21 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:57:22.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:21 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:57:22.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:21 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:57:22.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:21 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:57:22.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:21 smithi155 ceph-mon[131607]: Service mds.all has no related daemon already placed. 2023-11-16T16:57:22.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:21 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:57:22.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:21 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:57:22.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:21 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:57:22.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:21 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:57:22.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:21 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:57:22.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:21 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:57:22.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:21 smithi002 ceph-mon[128712]: Service mds.all has no related daemon already placed. 2023-11-16T16:57:22.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:21 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:57:22.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:21 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:57:23.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:22 smithi141 ceph-mon[131184]: pgmap v231: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:23.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:22 smithi155 ceph-mon[131607]: pgmap v231: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:23.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:22 smithi002 ceph-mon[128712]: pgmap v231: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:25.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:24 smithi141 ceph-mon[131184]: pgmap v232: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:25.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:24 smithi155 ceph-mon[131607]: pgmap v232: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:25.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:24 smithi002 ceph-mon[128712]: pgmap v232: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:27.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:26 smithi141 ceph-mon[131184]: pgmap v233: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:27.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:26 smithi155 ceph-mon[131607]: pgmap v233: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:27.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:26 smithi002 ceph-mon[128712]: pgmap v233: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:57:29.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:28 smithi141 ceph-mon[131184]: pgmap v234: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-11-16T16:57:29.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:28 smithi155 ceph-mon[131607]: pgmap v234: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-11-16T16:57:29.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:28 smithi002 ceph-mon[128712]: pgmap v234: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-11-16T16:57:31.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:30 smithi141 ceph-mon[131184]: pgmap v235: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-11-16T16:57:31.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:30 smithi155 ceph-mon[131607]: pgmap v235: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-11-16T16:57:31.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:30 smithi002 ceph-mon[128712]: pgmap v235: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-11-16T16:57:33.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:32 smithi141 ceph-mon[131184]: pgmap v236: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 87 B/s wr, 7 op/s 2023-11-16T16:57:33.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:32 smithi155 ceph-mon[131607]: pgmap v236: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 87 B/s wr, 7 op/s 2023-11-16T16:57:33.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:32 smithi002 ceph-mon[128712]: pgmap v236: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 87 B/s wr, 7 op/s 2023-11-16T16:57:35.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:34 smithi141 ceph-mon[131184]: pgmap v237: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:35.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:34 smithi155 ceph-mon[131607]: pgmap v237: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:35.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:34 smithi002 ceph-mon[128712]: pgmap v237: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:37.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:36 smithi141 ceph-mon[131184]: pgmap v238: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:37.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:36 smithi155 ceph-mon[131607]: pgmap v238: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:37.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:36 smithi002 ceph-mon[128712]: pgmap v238: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:38 smithi141 ceph-mon[131184]: pgmap v239: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:57:39.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:38 smithi155 ceph-mon[131607]: pgmap v239: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:57:39.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:38 smithi002 ceph-mon[128712]: pgmap v239: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:57:41.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:40 smithi141 ceph-mon[131184]: pgmap v240: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:41.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:40 smithi155 ceph-mon[131607]: pgmap v240: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:41.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:40 smithi002 ceph-mon[128712]: pgmap v240: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:43.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:42 smithi141 ceph-mon[131184]: pgmap v241: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:57:43.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:42 smithi155 ceph-mon[131607]: pgmap v241: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:57:43.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:42 smithi002 ceph-mon[128712]: pgmap v241: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:57:45.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:44 smithi141 ceph-mon[131184]: pgmap v242: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:45.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:44 smithi155 ceph-mon[131607]: pgmap v242: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:45.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:44 smithi002 ceph-mon[128712]: pgmap v242: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:47.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:46 smithi141 ceph-mon[131184]: pgmap v243: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:47.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:46 smithi155 ceph-mon[131607]: pgmap v243: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:47.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:46 smithi002 ceph-mon[128712]: pgmap v243: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:49.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:48 smithi141 ceph-mon[131184]: pgmap v244: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:57:49.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:48 smithi155 ceph-mon[131607]: pgmap v244: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:57:49.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:48 smithi002 ceph-mon[128712]: pgmap v244: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:57:51.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:50 smithi155 ceph-mon[131607]: pgmap v245: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:51.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:50 smithi002 ceph-mon[128712]: pgmap v245: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:51.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:50 smithi141 ceph-mon[131184]: pgmap v245: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:53.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:52 smithi155 ceph-mon[131607]: pgmap v246: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:57:53.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:52 smithi002 ceph-mon[128712]: pgmap v246: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:57:53.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:52 smithi141 ceph-mon[131184]: pgmap v246: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:57:55.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:54 smithi155 ceph-mon[131607]: pgmap v247: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:55.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:54 smithi002 ceph-mon[128712]: pgmap v247: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:54 smithi141 ceph-mon[131184]: pgmap v247: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:57.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:56 smithi155 ceph-mon[131607]: pgmap v248: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:57.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:56 smithi002 ceph-mon[128712]: pgmap v248: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:57.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:56 smithi141 ceph-mon[131184]: pgmap v248: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:57:59.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:57:58 smithi155 ceph-mon[131607]: pgmap v249: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:57:59.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:57:58 smithi002 ceph-mon[128712]: pgmap v249: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:57:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:57:58 smithi141 ceph-mon[131184]: pgmap v249: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:58:01.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:00 smithi155 ceph-mon[131607]: pgmap v250: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:01.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:00 smithi002 ceph-mon[128712]: pgmap v250: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:01.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:00 smithi141 ceph-mon[131184]: pgmap v250: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:03.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:02 smithi155 ceph-mon[131607]: pgmap v251: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-11-16T16:58:03.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:02 smithi002 ceph-mon[128712]: pgmap v251: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-11-16T16:58:03.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:02 smithi141 ceph-mon[131184]: pgmap v251: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-11-16T16:58:05.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:04 smithi155 ceph-mon[131607]: pgmap v252: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:58:05.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:04 smithi002 ceph-mon[128712]: pgmap v252: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:58:05.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:04 smithi141 ceph-mon[131184]: pgmap v252: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:58:07.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:06 smithi155 ceph-mon[131607]: pgmap v253: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:58:07.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:06 smithi002 ceph-mon[128712]: pgmap v253: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:58:07.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:06 smithi141 ceph-mon[131184]: pgmap v253: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:58:09.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:08 smithi155 ceph-mon[131607]: pgmap v254: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-11-16T16:58:09.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:08 smithi002 ceph-mon[128712]: pgmap v254: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-11-16T16:58:09.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:08 smithi141 ceph-mon[131184]: pgmap v254: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-11-16T16:58:11.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:10 smithi155 ceph-mon[131607]: pgmap v255: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:58:11.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:10 smithi002 ceph-mon[128712]: pgmap v255: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:58:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:10 smithi141 ceph-mon[131184]: pgmap v255: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:58:12.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:11 smithi155 ceph-mon[131607]: osdmap e100: 12 total, 12 up, 12 in 2023-11-16T16:58:12.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:11 smithi002 ceph-mon[128712]: osdmap e100: 12 total, 12 up, 12 in 2023-11-16T16:58:12.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:11 smithi141 ceph-mon[131184]: osdmap e100: 12 total, 12 up, 12 in 2023-11-16T16:58:13.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:12 smithi155 ceph-mon[131607]: pgmap v257: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:58:13.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:12 smithi002 ceph-mon[128712]: pgmap v257: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:58:13.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:12 smithi141 ceph-mon[131184]: pgmap v257: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:58:15.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:14 smithi155 ceph-mon[131607]: pgmap v258: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:58:15.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:14 smithi002 ceph-mon[128712]: pgmap v258: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:58:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:14 smithi141 ceph-mon[131184]: pgmap v258: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:58:17.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:16 smithi155 ceph-mon[131607]: pgmap v259: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:58:17.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:16 smithi002 ceph-mon[128712]: pgmap v259: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:58:17.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:16 smithi141 ceph-mon[131184]: pgmap v259: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:58:19.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:18 smithi155 ceph-mon[131607]: pgmap v260: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:58:19.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:18 smithi002 ceph-mon[128712]: pgmap v260: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:58:19.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:18 smithi141 ceph-mon[131184]: pgmap v260: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:58:21.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:20 smithi141 ceph-mon[131184]: pgmap v261: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:58:21.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:20 smithi141 ceph-mon[131184]: osdmap e101: 12 total, 12 up, 12 in 2023-11-16T16:58:21.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:20 smithi155 ceph-mon[131607]: pgmap v261: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:58:21.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:20 smithi155 ceph-mon[131607]: osdmap e101: 12 total, 12 up, 12 in 2023-11-16T16:58:21.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:20 smithi002 ceph-mon[128712]: pgmap v261: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:58:21.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:20 smithi002 ceph-mon[128712]: osdmap e101: 12 total, 12 up, 12 in 2023-11-16T16:58:22.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:21 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:58:22.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:21 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:58:22.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:21 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:58:23.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:22 smithi141 ceph-mon[131184]: pgmap v263: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-11-16T16:58:23.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:22 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:58:23.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:22 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:58:23.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:22 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:58:23.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:22 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:58:23.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:22 smithi141 ceph-mon[131184]: Service mds.all has no related daemon already placed. 2023-11-16T16:58:23.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:22 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:58:23.016 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:22 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:58:23.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:22 smithi155 ceph-mon[131607]: pgmap v263: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-11-16T16:58:23.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:22 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:58:23.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:22 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:58:23.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:22 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:58:23.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:22 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:58:23.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:22 smithi155 ceph-mon[131607]: Service mds.all has no related daemon already placed. 2023-11-16T16:58:23.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:22 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:58:23.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:22 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:58:23.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:22 smithi002 ceph-mon[128712]: pgmap v263: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-11-16T16:58:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:22 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:58:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:22 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:58:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:22 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:58:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:22 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:58:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:22 smithi002 ceph-mon[128712]: Service mds.all has no related daemon already placed. 2023-11-16T16:58:23.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:22 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:58:23.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:22 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:58:25.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:24 smithi141 ceph-mon[131184]: pgmap v264: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-11-16T16:58:25.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:24 smithi155 ceph-mon[131607]: pgmap v264: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-11-16T16:58:25.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:24 smithi002 ceph-mon[128712]: pgmap v264: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-11-16T16:58:27.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:26 smithi141 ceph-mon[131184]: pgmap v265: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-11-16T16:58:27.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:26 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:58:27.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:26 smithi155 ceph-mon[131607]: pgmap v265: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-11-16T16:58:27.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:26 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:58:27.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:26 smithi002 ceph-mon[128712]: pgmap v265: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-11-16T16:58:27.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:26 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:58:29.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:28 smithi141 ceph-mon[131184]: pgmap v266: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:58:29.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:28 smithi155 ceph-mon[131607]: pgmap v266: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:58:29.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:28 smithi002 ceph-mon[128712]: pgmap v266: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:58:31.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:30 smithi141 ceph-mon[131184]: pgmap v267: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:58:31.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:30 smithi155 ceph-mon[131607]: pgmap v267: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:58:31.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:30 smithi002 ceph-mon[128712]: pgmap v267: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:58:32.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:31 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:58:32.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:31 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:58:32.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:31 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:58:33.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:32 smithi141 ceph-mon[131184]: pgmap v268: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2023-11-16T16:58:33.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:32 smithi155 ceph-mon[131607]: pgmap v268: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2023-11-16T16:58:33.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:32 smithi002 ceph-mon[128712]: pgmap v268: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2023-11-16T16:58:35.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:34 smithi141 ceph-mon[131184]: pgmap v269: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:35.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:34 smithi155 ceph-mon[131607]: pgmap v269: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:35.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:34 smithi002 ceph-mon[128712]: pgmap v269: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:37.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:36 smithi141 ceph-mon[131184]: pgmap v270: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:37.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:36 smithi155 ceph-mon[131607]: pgmap v270: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:37.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:36 smithi002 ceph-mon[128712]: pgmap v270: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:39.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:38 smithi141 ceph-mon[131184]: pgmap v271: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:58:39.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:38 smithi155 ceph-mon[131607]: pgmap v271: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:58:39.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:38 smithi002 ceph-mon[128712]: pgmap v271: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:58:41.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:40 smithi141 ceph-mon[131184]: pgmap v272: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:41.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:40 smithi155 ceph-mon[131607]: pgmap v272: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:41.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:40 smithi002 ceph-mon[128712]: pgmap v272: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:43.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:42 smithi141 ceph-mon[131184]: pgmap v273: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:58:43.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:42 smithi155 ceph-mon[131607]: pgmap v273: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:58:43.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:42 smithi002 ceph-mon[128712]: pgmap v273: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:58:45.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:44 smithi141 ceph-mon[131184]: pgmap v274: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:45.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:44 smithi155 ceph-mon[131607]: pgmap v274: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:45.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:44 smithi002 ceph-mon[128712]: pgmap v274: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:47.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:46 smithi141 ceph-mon[131184]: pgmap v275: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:47.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:46 smithi155 ceph-mon[131607]: pgmap v275: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:47.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:46 smithi002 ceph-mon[128712]: pgmap v275: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:49.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:48 smithi141 ceph-mon[131184]: pgmap v276: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:58:49.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:48 smithi155 ceph-mon[131607]: pgmap v276: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:58:49.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:48 smithi002 ceph-mon[128712]: pgmap v276: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:58:51.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:50 smithi141 ceph-mon[131184]: pgmap v277: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:51.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:50 smithi155 ceph-mon[131607]: pgmap v277: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:51.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:50 smithi002 ceph-mon[128712]: pgmap v277: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:53.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:52 smithi141 ceph-mon[131184]: pgmap v278: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:58:53.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:52 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:58:53.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:52 smithi155 ceph-mon[131607]: pgmap v278: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:58:53.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:52 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:58:53.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:52 smithi002 ceph-mon[128712]: pgmap v278: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:58:53.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:52 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:58:55.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:54 smithi141 ceph-mon[131184]: pgmap v279: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:55.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:54 smithi155 ceph-mon[131607]: pgmap v279: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:55.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:54 smithi002 ceph-mon[128712]: pgmap v279: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:57.015 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:56 smithi141 ceph-mon[131184]: pgmap v280: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:57.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:56 smithi155 ceph-mon[131607]: pgmap v280: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:57.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:56 smithi002 ceph-mon[128712]: pgmap v280: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:58:59.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:58:58 smithi155 ceph-mon[131607]: pgmap v281: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:58:59.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:58:58 smithi002 ceph-mon[128712]: pgmap v281: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:58:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:58:58 smithi141 ceph-mon[131184]: pgmap v281: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:59:01.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:00 smithi155 ceph-mon[131607]: pgmap v282: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:59:01.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:00 smithi002 ceph-mon[128712]: pgmap v282: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:59:01.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:00 smithi141 ceph-mon[131184]: pgmap v282: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:59:03.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:02 smithi155 ceph-mon[131607]: pgmap v283: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-11-16T16:59:03.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:02 smithi002 ceph-mon[128712]: pgmap v283: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-11-16T16:59:03.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:02 smithi141 ceph-mon[131184]: pgmap v283: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-11-16T16:59:05.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:04 smithi155 ceph-mon[131607]: pgmap v284: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:59:05.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:04 smithi002 ceph-mon[128712]: pgmap v284: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:59:05.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:04 smithi141 ceph-mon[131184]: pgmap v284: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:59:06.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:05 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:59:06.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:05 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:59:06.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:05 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:59:07.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:06 smithi155 ceph-mon[131607]: pgmap v285: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:59:07.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:06 smithi002 ceph-mon[128712]: pgmap v285: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:59:07.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:06 smithi141 ceph-mon[131184]: pgmap v285: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:59:09.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:08 smithi155 ceph-mon[131607]: pgmap v286: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-11-16T16:59:09.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:08 smithi002 ceph-mon[128712]: pgmap v286: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-11-16T16:59:09.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:08 smithi141 ceph-mon[131184]: pgmap v286: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-11-16T16:59:11.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:10 smithi155 ceph-mon[131607]: pgmap v287: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:59:11.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:10 smithi002 ceph-mon[128712]: pgmap v287: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:59:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:10 smithi141 ceph-mon[131184]: pgmap v287: 129 pgs: 129 active+clean; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-11-16T16:59:12.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:11 smithi155 ceph-mon[131607]: osdmap e102: 12 total, 12 up, 12 in 2023-11-16T16:59:12.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:11 smithi002 ceph-mon[128712]: osdmap e102: 12 total, 12 up, 12 in 2023-11-16T16:59:12.267 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:11 smithi141 ceph-mon[131184]: osdmap e102: 12 total, 12 up, 12 in 2023-11-16T16:59:13.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:12 smithi155 ceph-mon[131607]: pgmap v289: 129 pgs: 24 active+clean+snaptrim, 33 active+clean, 72 active+clean+snaptrim_wait; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:59:13.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:12 smithi002 ceph-mon[128712]: pgmap v289: 129 pgs: 24 active+clean+snaptrim, 33 active+clean, 72 active+clean+snaptrim_wait; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:59:13.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:12 smithi141 ceph-mon[131184]: pgmap v289: 129 pgs: 24 active+clean+snaptrim, 33 active+clean, 72 active+clean+snaptrim_wait; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:59:15.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:14 smithi155 ceph-mon[131607]: pgmap v290: 129 pgs: 24 active+clean+snaptrim, 33 active+clean, 72 active+clean+snaptrim_wait; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:59:15.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:14 smithi002 ceph-mon[128712]: pgmap v290: 129 pgs: 24 active+clean+snaptrim, 33 active+clean, 72 active+clean+snaptrim_wait; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:59:15.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:14 smithi141 ceph-mon[131184]: pgmap v290: 129 pgs: 24 active+clean+snaptrim, 33 active+clean, 72 active+clean+snaptrim_wait; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:59:17.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:16 smithi155 ceph-mon[131607]: pgmap v291: 129 pgs: 24 active+clean+snaptrim, 33 active+clean, 72 active+clean+snaptrim_wait; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:59:17.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:16 smithi002 ceph-mon[128712]: pgmap v291: 129 pgs: 24 active+clean+snaptrim, 33 active+clean, 72 active+clean+snaptrim_wait; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:59:17.267 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:16 smithi141 ceph-mon[131184]: pgmap v291: 129 pgs: 24 active+clean+snaptrim, 33 active+clean, 72 active+clean+snaptrim_wait; 607 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:59:18.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:17 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:59:18.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:17 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:59:18.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:17 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:59:19.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:18 smithi155 ceph-mon[131607]: pgmap v292: 129 pgs: 23 active+clean+snaptrim, 36 active+clean, 70 active+clean+snaptrim_wait; 607 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:59:19.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:18 smithi002 ceph-mon[128712]: pgmap v292: 129 pgs: 23 active+clean+snaptrim, 36 active+clean, 70 active+clean+snaptrim_wait; 607 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:59:19.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:18 smithi141 ceph-mon[131184]: pgmap v292: 129 pgs: 23 active+clean+snaptrim, 36 active+clean, 70 active+clean+snaptrim_wait; 607 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:59:21.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:20 smithi002 ceph-mon[128712]: pgmap v293: 129 pgs: 23 active+clean+snaptrim, 36 active+clean, 70 active+clean+snaptrim_wait; 607 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:59:21.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:20 smithi002 ceph-mon[128712]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:59:21.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:20 smithi141 ceph-mon[131184]: pgmap v293: 129 pgs: 23 active+clean+snaptrim, 36 active+clean, 70 active+clean+snaptrim_wait; 607 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:59:21.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:20 smithi141 ceph-mon[131184]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:59:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:20 smithi155 ceph-mon[131607]: pgmap v293: 129 pgs: 23 active+clean+snaptrim, 36 active+clean, 70 active+clean+snaptrim_wait; 607 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T16:59:21.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:20 smithi155 ceph-mon[131607]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-11-16T16:59:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:22 smithi002 ceph-mon[128712]: pgmap v294: 129 pgs: 19 active+clean+snaptrim, 52 active+clean, 58 active+clean+snaptrim_wait; 607 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 193 B/s wr, 8 op/s 2023-11-16T16:59:23.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:22 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:59:23.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:22 smithi141 ceph-mon[131184]: pgmap v294: 129 pgs: 19 active+clean+snaptrim, 52 active+clean, 58 active+clean+snaptrim_wait; 607 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 193 B/s wr, 8 op/s 2023-11-16T16:59:23.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:22 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:59:23.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:22 smithi155 ceph-mon[131607]: pgmap v294: 129 pgs: 19 active+clean+snaptrim, 52 active+clean, 58 active+clean+snaptrim_wait; 607 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 193 B/s wr, 8 op/s 2023-11-16T16:59:23.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:22 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T16:59:24.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:23 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:59:24.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:23 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:59:24.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:23 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:59:24.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:23 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:59:24.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:23 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T16:59:24.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:23 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T16:59:25.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:24 smithi002 ceph-mon[128712]: pgmap v295: 129 pgs: 19 active+clean+snaptrim, 52 active+clean, 58 active+clean+snaptrim_wait; 607 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T16:59:25.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:24 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:59:25.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:24 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:59:25.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:24 smithi002 ceph-mon[128712]: Service mds.all has no related daemon already placed. 2023-11-16T16:59:25.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:24 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T16:59:25.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:24 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T16:59:25.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:24 smithi141 ceph-mon[131184]: pgmap v295: 129 pgs: 19 active+clean+snaptrim, 52 active+clean, 58 active+clean+snaptrim_wait; 607 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T16:59:25.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:24 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:59:25.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:24 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:59:25.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:24 smithi141 ceph-mon[131184]: Service mds.all has no related daemon already placed. 2023-11-16T16:59:25.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:24 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T16:59:25.266 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:24 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T16:59:25.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:24 smithi155 ceph-mon[131607]: pgmap v295: 129 pgs: 19 active+clean+snaptrim, 52 active+clean, 58 active+clean+snaptrim_wait; 607 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T16:59:25.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:24 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:59:25.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:24 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T16:59:25.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:24 smithi155 ceph-mon[131607]: Service mds.all has no related daemon already placed. 2023-11-16T16:59:25.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:24 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T16:59:25.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:24 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T16:59:27.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:26 smithi002 ceph-mon[128712]: pgmap v296: 129 pgs: 19 active+clean+snaptrim, 52 active+clean, 58 active+clean+snaptrim_wait; 607 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T16:59:27.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:26 smithi141 ceph-mon[131184]: pgmap v296: 129 pgs: 19 active+clean+snaptrim, 52 active+clean, 58 active+clean+snaptrim_wait; 607 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T16:59:27.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:26 smithi155 ceph-mon[131607]: pgmap v296: 129 pgs: 19 active+clean+snaptrim, 52 active+clean, 58 active+clean+snaptrim_wait; 607 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-11-16T16:59:29.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:28 smithi141 ceph-mon[131184]: pgmap v297: 129 pgs: 4 active+clean+snaptrim, 111 active+clean, 14 active+clean+snaptrim_wait; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2023-11-16T16:59:29.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:28 smithi155 ceph-mon[131607]: pgmap v297: 129 pgs: 4 active+clean+snaptrim, 111 active+clean, 14 active+clean+snaptrim_wait; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2023-11-16T16:59:29.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:28 smithi002 ceph-mon[128712]: pgmap v297: 129 pgs: 4 active+clean+snaptrim, 111 active+clean, 14 active+clean+snaptrim_wait; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2023-11-16T16:59:31.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:30 smithi141 ceph-mon[131184]: pgmap v298: 129 pgs: 4 active+clean+snaptrim, 111 active+clean, 14 active+clean+snaptrim_wait; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:59:31.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:30 smithi155 ceph-mon[131607]: pgmap v298: 129 pgs: 4 active+clean+snaptrim, 111 active+clean, 14 active+clean+snaptrim_wait; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:59:31.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:30 smithi002 ceph-mon[128712]: pgmap v298: 129 pgs: 4 active+clean+snaptrim, 111 active+clean, 14 active+clean+snaptrim_wait; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:59:33.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:32 smithi141 ceph-mon[131184]: pgmap v299: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:59:33.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:32 smithi155 ceph-mon[131607]: pgmap v299: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:59:33.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:32 smithi002 ceph-mon[128712]: pgmap v299: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:59:35.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:34 smithi141 ceph-mon[131184]: pgmap v300: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:59:35.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:34 smithi141 ceph-mon[131184]: osdmap e103: 12 total, 12 up, 12 in 2023-11-16T16:59:35.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:34 smithi155 ceph-mon[131607]: pgmap v300: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:59:35.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:34 smithi155 ceph-mon[131607]: osdmap e103: 12 total, 12 up, 12 in 2023-11-16T16:59:35.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:34 smithi002 ceph-mon[128712]: pgmap v300: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:59:35.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:34 smithi002 ceph-mon[128712]: osdmap e103: 12 total, 12 up, 12 in 2023-11-16T16:59:37.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:36 smithi141 ceph-mon[131184]: pgmap v302: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:59:37.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:36 smithi155 ceph-mon[131607]: pgmap v302: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:59:37.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:36 smithi002 ceph-mon[128712]: pgmap v302: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:59:39.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:38 smithi141 ceph-mon[131184]: pgmap v303: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:59:39.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:38 smithi155 ceph-mon[131607]: pgmap v303: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:59:39.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:38 smithi002 ceph-mon[128712]: pgmap v303: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:59:41.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:40 smithi141 ceph-mon[131184]: pgmap v304: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:59:41.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:40 smithi155 ceph-mon[131607]: pgmap v304: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:59:41.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:40 smithi002 ceph-mon[128712]: pgmap v304: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:59:43.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:42 smithi141 ceph-mon[131184]: pgmap v305: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:59:43.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:42 smithi155 ceph-mon[131607]: pgmap v305: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:59:43.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:42 smithi002 ceph-mon[128712]: pgmap v305: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:59:45.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:44 smithi141 ceph-mon[131184]: pgmap v306: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:59:45.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:44 smithi155 ceph-mon[131607]: pgmap v306: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:59:45.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:44 smithi002 ceph-mon[128712]: pgmap v306: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T16:59:47.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:46 smithi141 ceph-mon[131184]: pgmap v307: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-11-16T16:59:47.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:46 smithi155 ceph-mon[131607]: pgmap v307: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-11-16T16:59:47.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:46 smithi002 ceph-mon[128712]: pgmap v307: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-11-16T16:59:49.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:48 smithi141 ceph-mon[131184]: pgmap v308: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:59:49.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:48 smithi155 ceph-mon[131607]: pgmap v308: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:59:49.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:48 smithi002 ceph-mon[128712]: pgmap v308: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:59:51.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:50 smithi141 ceph-mon[131184]: pgmap v309: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:59:51.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:50 smithi155 ceph-mon[131607]: pgmap v309: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:59:51.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:50 smithi002 ceph-mon[128712]: pgmap v309: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:59:53.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:52 smithi141 ceph-mon[131184]: pgmap v310: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:59:53.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:52 smithi155 ceph-mon[131607]: pgmap v310: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:59:53.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:52 smithi002 ceph-mon[128712]: pgmap v310: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:59:55.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:54 smithi141 ceph-mon[131184]: pgmap v311: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:59:55.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:54 smithi155 ceph-mon[131607]: pgmap v311: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:59:55.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:54 smithi002 ceph-mon[128712]: pgmap v311: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:59:57.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:56 smithi141 ceph-mon[131184]: pgmap v312: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:59:57.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:56 smithi155 ceph-mon[131607]: pgmap v312: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:59:57.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:56 smithi002 ceph-mon[128712]: pgmap v312: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T16:59:57.439 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2023-11-16T16:59:57.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.651+0000 7f84d85ff700 1 -- 172.21.15.2:0/3118964566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84d0100c50 msgr2=0x7f84d0101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:57.652 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.651+0000 7f84d85ff700 1 --2- 172.21.15.2:0/3118964566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84d0100c50 0x7f84d0101020 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f84cc0099c0 tx=0x7f84cc009cd0 comp rx=0 tx=0).stop 2023-11-16T16:59:57.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.651+0000 7f84d85ff700 1 -- 172.21.15.2:0/3118964566 shutdown_connections 2023-11-16T16:59:57.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.651+0000 7f84d85ff700 1 --2- 172.21.15.2:0/3118964566 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f84d0105560 0x7f84d010d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:57.653 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.651+0000 7f84d85ff700 1 --2- 172.21.15.2:0/3118964566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84d0100c50 0x7f84d0101020 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:57.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.651+0000 7f84d85ff700 1 --2- 172.21.15.2:0/3118964566 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84d01015f0 0x7f84d0104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:57.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.651+0000 7f84d85ff700 1 -- 172.21.15.2:0/3118964566 >> 172.21.15.2:0/3118964566 conn(0x7f84d0077550 msgr2=0x7f84d0077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:57.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.651+0000 7f84d85ff700 1 -- 172.21.15.2:0/3118964566 shutdown_connections 2023-11-16T16:59:57.654 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.651+0000 7f84d85ff700 1 -- 172.21.15.2:0/3118964566 wait complete. 2023-11-16T16:59:57.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.652+0000 7f84d85ff700 1 Processor -- start 2023-11-16T16:59:57.657 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.652+0000 7f84d85ff700 1 -- start start 2023-11-16T16:59:57.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.653+0000 7f84d85ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84d0100c50 0x7f84d01985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:57.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.653+0000 7f84d85ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84d01015f0 0x7f84d0198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:57.658 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.653+0000 7f84d85ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f84d0105560 0x7f84d019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:57.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.653+0000 7f84d85ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d019d5c0 con 0x7f84d01015f0 2023-11-16T16:59:57.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.653+0000 7f84d85ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d019d730 con 0x7f84d0105560 2023-11-16T16:59:57.659 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.653+0000 7f84d85ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d019d8f0 con 0x7f84d0100c50 2023-11-16T16:59:57.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.653+0000 7f84d5b9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84d01015f0 0x7f84d0198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:57.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.653+0000 7f84d5b9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84d01015f0 0x7f84d0198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36344/0 (socket says 172.21.15.2:36344) 2023-11-16T16:59:57.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.653+0000 7f84d639b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84d0100c50 0x7f84d01985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:57.660 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.653+0000 7f84d5b9a700 1 -- 172.21.15.2:0/3833045051 learned_addr learned my addr 172.21.15.2:0/3833045051 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:59:57.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.653+0000 7f84d6b9c700 1 --2- 172.21.15.2:0/3833045051 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f84d0105560 0x7f84d019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:57.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.653+0000 7f84d5b9a700 1 -- 172.21.15.2:0/3833045051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84d0100c50 msgr2=0x7f84d01985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:57.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.653+0000 7f84d5b9a700 1 --2- 172.21.15.2:0/3833045051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84d0100c50 0x7f84d01985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:57.661 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.653+0000 7f84d5b9a700 1 -- 172.21.15.2:0/3833045051 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f84d0105560 msgr2=0x7f84d019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:57.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.653+0000 7f84d5b9a700 1 --2- 172.21.15.2:0/3833045051 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f84d0105560 0x7f84d019cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:57.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.653+0000 7f84d5b9a700 1 -- 172.21.15.2:0/3833045051 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84cc0095f0 con 0x7f84d01015f0 2023-11-16T16:59:57.662 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.654+0000 7f84d5b9a700 1 --2- 172.21.15.2:0/3833045051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84d01015f0 0x7f84d0198b20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f84c000d950 tx=0x7f84c000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:57.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.654+0000 7f84c77fe700 1 -- 172.21.15.2:0/3833045051 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c000f890 con 0x7f84d01015f0 2023-11-16T16:59:57.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.654+0000 7f84c77fe700 1 -- 172.21.15.2:0/3833045051 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84c0010460 con 0x7f84d01015f0 2023-11-16T16:59:57.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.654+0000 7f84d85ff700 1 -- 172.21.15.2:0/3833045051 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d01103b0 con 0x7f84d01015f0 2023-11-16T16:59:57.663 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.654+0000 7f84c77fe700 1 -- 172.21.15.2:0/3833045051 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c000e610 con 0x7f84d01015f0 2023-11-16T16:59:57.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.654+0000 7f84d85ff700 1 -- 172.21.15.2:0/3833045051 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d0110220 con 0x7f84d01015f0 2023-11-16T16:59:57.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.655+0000 7f84d85ff700 1 -- 172.21.15.2:0/3833045051 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d0110540 con 0x7f84d01015f0 2023-11-16T16:59:57.664 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.657+0000 7f84c77fe700 1 -- 172.21.15.2:0/3833045051 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f84c00105d0 con 0x7f84d01015f0 2023-11-16T16:59:57.665 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.658+0000 7f84c77fe700 1 --2- 172.21.15.2:0/3833045051 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f84bc0a12c0 0x7f84bc0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:57.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.658+0000 7f84c77fe700 1 -- 172.21.15.2:0/3833045051 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9482+0+0 (secure 0 0 0) 0x7f84c00ccbf0 con 0x7f84d01015f0 2023-11-16T16:59:57.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.659+0000 7f84d639b700 1 --2- 172.21.15.2:0/3833045051 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f84bc0a12c0 0x7f84bc0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:57.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.659+0000 7f84d639b700 1 --2- 172.21.15.2:0/3833045051 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f84bc0a12c0 0x7f84bc0a3770 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00b580 tx=0x7f84cc0058e0 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:57.666 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.663+0000 7f84c77fe700 1 -- 172.21.15.2:0/3833045051 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f84c0095be0 con 0x7f84d01015f0 2023-11-16T16:59:57.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.902+0000 7f84d85ff700 1 -- 172.21.15.2:0/3833045051 --> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f84d0073d30 con 0x7f84bc0a12c0 2023-11-16T16:59:57.911 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.911+0000 7f84c77fe700 1 -- 172.21.15.2:0/3833045051 <== mgr.15372 v2:172.21.15.2:6800/1352003968 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+335 (secure 0 0 0) 0x7f84d0073d30 con 0x7f84bc0a12c0 2023-11-16T16:59:57.912 INFO:teuthology.orchestra.run.smithi002.stdout:{"fs": "cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "1M", "retention": {"h": 3, "M": 6}, "start": "2023-11-16T00:00:00", "created": "2023-11-16T16:49:55", "first": "2023-11-16T16:50:00", "last": "2023-11-16T16:59:00", "last_pruned": "2023-11-16T16:59:00", "created_count": 10, "pruned_count": 4, "active": true} 2023-11-16T16:59:57.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.913+0000 7f84d85ff700 1 -- 172.21.15.2:0/3833045051 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f84bc0a12c0 msgr2=0x7f84bc0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:57.914 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.913+0000 7f84d85ff700 1 --2- 172.21.15.2:0/3833045051 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f84bc0a12c0 0x7f84bc0a3770 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00b580 tx=0x7f84cc0058e0 comp rx=0 tx=0).stop 2023-11-16T16:59:57.915 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.913+0000 7f84d85ff700 1 -- 172.21.15.2:0/3833045051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84d01015f0 msgr2=0x7f84d0198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:57.915 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.913+0000 7f84d85ff700 1 --2- 172.21.15.2:0/3833045051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84d01015f0 0x7f84d0198b20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f84c000d950 tx=0x7f84c000dc60 comp rx=0 tx=0).stop 2023-11-16T16:59:57.915 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.913+0000 7f84d85ff700 1 -- 172.21.15.2:0/3833045051 shutdown_connections 2023-11-16T16:59:57.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.913+0000 7f84d85ff700 1 --2- 172.21.15.2:0/3833045051 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f84bc0a12c0 0x7f84bc0a3770 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:57.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.913+0000 7f84d85ff700 1 --2- 172.21.15.2:0/3833045051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84d0100c50 0x7f84d01985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:57.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.913+0000 7f84d85ff700 1 --2- 172.21.15.2:0/3833045051 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f84d0105560 0x7f84d019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:57.916 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.913+0000 7f84d85ff700 1 --2- 172.21.15.2:0/3833045051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84d01015f0 0x7f84d0198b20 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:57.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.913+0000 7f84d85ff700 1 -- 172.21.15.2:0/3833045051 >> 172.21.15.2:0/3833045051 conn(0x7f84d0077550 msgr2=0x7f84d00feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:57.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.913+0000 7f84d85ff700 1 -- 172.21.15.2:0/3833045051 shutdown_connections 2023-11-16T16:59:57.917 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:57.913+0000 7f84d85ff700 1 -- 172.21.15.2:0/3833045051 wait complete. 2023-11-16T16:59:57.933 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2023-11-16T16:59:58.140 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.136+0000 7fed64dff700 1 -- 172.21.15.2:0/1162383846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed60100f00 msgr2=0x7fed60104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:58.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.136+0000 7fed64dff700 1 --2- 172.21.15.2:0/1162383846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed60100f00 0x7fed60104ec0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fed480099c0 tx=0x7fed48009cd0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.136+0000 7fed64dff700 1 -- 172.21.15.2:0/1162383846 shutdown_connections 2023-11-16T16:59:58.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.136+0000 7fed64dff700 1 --2- 172.21.15.2:0/1162383846 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fed601055a0 0x7fed6010d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.136+0000 7fed64dff700 1 --2- 172.21.15.2:0/1162383846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed60100560 0x7fed60100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.136+0000 7fed64dff700 1 --2- 172.21.15.2:0/1162383846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed60100f00 0x7fed60104ec0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.136+0000 7fed64dff700 1 -- 172.21.15.2:0/1162383846 >> 172.21.15.2:0/1162383846 conn(0x7fed600781b0 msgr2=0x7fed600785b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:58.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.136+0000 7fed64dff700 1 -- 172.21.15.2:0/1162383846 shutdown_connections 2023-11-16T16:59:58.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.136+0000 7fed64dff700 1 -- 172.21.15.2:0/1162383846 wait complete. 2023-11-16T16:59:58.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.137+0000 7fed64dff700 1 Processor -- start 2023-11-16T16:59:58.142 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.137+0000 7fed64dff700 1 -- start start 2023-11-16T16:59:58.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.138+0000 7fed64dff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fed60100560 0x7fed60198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:58.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.138+0000 7fed64dff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed60100f00 0x7fed60198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:58.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.138+0000 7fed64dff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed601055a0 0x7fed6019cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:58.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.138+0000 7fed64dff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6019d610 con 0x7fed60100f00 2023-11-16T16:59:58.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.138+0000 7fed64dff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6019d780 con 0x7fed60100560 2023-11-16T16:59:58.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.138+0000 7fed64dff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6019d940 con 0x7fed601055a0 2023-11-16T16:59:58.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.138+0000 7fed57fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed60100f00 0x7fed60198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:58.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.138+0000 7fed5e59c700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fed60100560 0x7fed60198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:58.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.138+0000 7fed57fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed60100f00 0x7fed60198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36366/0 (socket says 172.21.15.2:36366) 2023-11-16T16:59:58.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.138+0000 7fed57fff700 1 -- 172.21.15.2:0/2441800694 learned_addr learned my addr 172.21.15.2:0/2441800694 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:59:58.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.138+0000 7fed5ed9d700 1 --2- 172.21.15.2:0/2441800694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed601055a0 0x7fed6019cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:58.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.139+0000 7fed57fff700 1 -- 172.21.15.2:0/2441800694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed601055a0 msgr2=0x7fed6019cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:58.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.139+0000 7fed57fff700 1 --2- 172.21.15.2:0/2441800694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed601055a0 0x7fed6019cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.139+0000 7fed57fff700 1 -- 172.21.15.2:0/2441800694 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fed60100560 msgr2=0x7fed60198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:58.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.139+0000 7fed57fff700 1 --2- 172.21.15.2:0/2441800694 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fed60100560 0x7fed60198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.139+0000 7fed57fff700 1 -- 172.21.15.2:0/2441800694 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed50009520 con 0x7fed60100f00 2023-11-16T16:59:58.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.139+0000 7fed57fff700 1 --2- 172.21.15.2:0/2441800694 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed60100f00 0x7fed60198b70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fed480098e0 tx=0x7fed4800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:58.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.139+0000 7fed577fe700 1 -- 172.21.15.2:0/2441800694 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4801d070 con 0x7fed60100f00 2023-11-16T16:59:58.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.139+0000 7fed577fe700 1 -- 172.21.15.2:0/2441800694 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed48022470 con 0x7fed60100f00 2023-11-16T16:59:58.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.139+0000 7fed64dff700 1 -- 172.21.15.2:0/2441800694 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed480095f0 con 0x7fed60100f00 2023-11-16T16:59:58.146 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.139+0000 7fed577fe700 1 -- 172.21.15.2:0/2441800694 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed48017650 con 0x7fed60100f00 2023-11-16T16:59:58.147 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.139+0000 7fed64dff700 1 -- 172.21.15.2:0/2441800694 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed60110370 con 0x7fed60100f00 2023-11-16T16:59:58.148 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.140+0000 7fed64dff700 1 -- 172.21.15.2:0/2441800694 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed601101e0 con 0x7fed60100f00 2023-11-16T16:59:58.148 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.142+0000 7fed577fe700 1 -- 172.21.15.2:0/2441800694 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fed48022850 con 0x7fed60100f00 2023-11-16T16:59:58.148 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.144+0000 7fed577fe700 1 --2- 172.21.15.2:0/2441800694 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fed4c0a1340 0x7fed4c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:58.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.144+0000 7fed577fe700 1 -- 172.21.15.2:0/2441800694 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9482+0+0 (secure 0 0 0) 0x7fed480cdef0 con 0x7fed60100f00 2023-11-16T16:59:58.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.144+0000 7fed5e59c700 1 --2- 172.21.15.2:0/2441800694 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fed4c0a1340 0x7fed4c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:58.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.144+0000 7fed5e59c700 1 --2- 172.21.15.2:0/2441800694 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fed4c0a1340 0x7fed4c0a37f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fed50009df0 tx=0x7fed50009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:58.149 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.147+0000 7fed577fe700 1 -- 172.21.15.2:0/2441800694 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fed48096f70 con 0x7fed60100f00 2023-11-16T16:59:58.385 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.384+0000 7fed64dff700 1 -- 172.21.15.2:0/2441800694 --> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fed60110500 con 0x7fed4c0a1340 2023-11-16T16:59:58.393 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.392+0000 7fed577fe700 1 -- 172.21.15.2:0/2441800694 <== mgr.15372 v2:172.21.15.2:6800/1352003968 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7fed60110500 con 0x7fed4c0a1340 2023-11-16T16:59:58.393 INFO:teuthology.orchestra.run.smithi002.stdout:/ 1M 6M3h 2023-11-16T16:59:58.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.394+0000 7fed64dff700 1 -- 172.21.15.2:0/2441800694 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fed4c0a1340 msgr2=0x7fed4c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:58.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.394+0000 7fed64dff700 1 --2- 172.21.15.2:0/2441800694 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fed4c0a1340 0x7fed4c0a37f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fed50009df0 tx=0x7fed50009260 comp rx=0 tx=0).stop 2023-11-16T16:59:58.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.394+0000 7fed64dff700 1 -- 172.21.15.2:0/2441800694 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed60100f00 msgr2=0x7fed60198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:58.396 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.394+0000 7fed64dff700 1 --2- 172.21.15.2:0/2441800694 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed60100f00 0x7fed60198b70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fed480098e0 tx=0x7fed4800f690 comp rx=0 tx=0).stop 2023-11-16T16:59:58.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.394+0000 7fed64dff700 1 -- 172.21.15.2:0/2441800694 shutdown_connections 2023-11-16T16:59:58.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.394+0000 7fed64dff700 1 --2- 172.21.15.2:0/2441800694 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fed4c0a1340 0x7fed4c0a37f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.394+0000 7fed64dff700 1 --2- 172.21.15.2:0/2441800694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed601055a0 0x7fed6019cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.394+0000 7fed64dff700 1 --2- 172.21.15.2:0/2441800694 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fed60100560 0x7fed60198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.394+0000 7fed64dff700 1 --2- 172.21.15.2:0/2441800694 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed60100f00 0x7fed60198b70 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.397 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.394+0000 7fed64dff700 1 -- 172.21.15.2:0/2441800694 >> 172.21.15.2:0/2441800694 conn(0x7fed600781b0 msgr2=0x7fed600fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:58.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.394+0000 7fed64dff700 1 -- 172.21.15.2:0/2441800694 shutdown_connections 2023-11-16T16:59:58.398 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.394+0000 7fed64dff700 1 -- 172.21.15.2:0/2441800694 wait complete. 2023-11-16T16:59:58.415 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2023-11-16T16:59:58.426 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T16:59:58.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.632+0000 7f73c1dff700 1 -- 172.21.15.2:0/4207589786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f73bc100550 msgr2=0x7f73bc100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:58.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.632+0000 7f73c1dff700 1 --2- 172.21.15.2:0/4207589786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f73bc100550 0x7f73bc100920 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009990 tx=0x7f73ac009ca0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.632+0000 7f73c1dff700 1 -- 172.21.15.2:0/4207589786 shutdown_connections 2023-11-16T16:59:58.634 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.632+0000 7f73c1dff700 1 --2- 172.21.15.2:0/4207589786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f73bc105540 0x7f73bc10d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.632+0000 7f73c1dff700 1 --2- 172.21.15.2:0/4207589786 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f73bc100550 0x7f73bc100920 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.632+0000 7f73c1dff700 1 --2- 172.21.15.2:0/4207589786 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73bc100ef0 0x7f73bc104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.632+0000 7f73c1dff700 1 -- 172.21.15.2:0/4207589786 >> 172.21.15.2:0/4207589786 conn(0x7f73bc077020 msgr2=0x7f73bc077420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:58.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.632+0000 7f73c1dff700 1 -- 172.21.15.2:0/4207589786 shutdown_connections 2023-11-16T16:59:58.635 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.632+0000 7f73c1dff700 1 -- 172.21.15.2:0/4207589786 wait complete. 2023-11-16T16:59:58.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.633+0000 7f73c1dff700 1 Processor -- start 2023-11-16T16:59:58.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.634+0000 7f73c1dff700 1 -- start start 2023-11-16T16:59:58.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.634+0000 7f73c1dff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73bc100550 0x7f73bc198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:58.638 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.634+0000 7f73c1dff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f73bc100ef0 0x7f73bc198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:58.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.634+0000 7f73c1dff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f73bc105540 0x7f73bc19cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:58.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.634+0000 7f73c1dff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc19d620 con 0x7f73bc100550 2023-11-16T16:59:58.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.634+0000 7f73c1dff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc19d790 con 0x7f73bc100ef0 2023-11-16T16:59:58.639 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.634+0000 7f73c1dff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc19d950 con 0x7f73bc105540 2023-11-16T16:59:58.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.634+0000 7f73bb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73bc100550 0x7f73bc198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:58.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.635+0000 7f73bb7fe700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73bc100550 0x7f73bc198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36382/0 (socket says 172.21.15.2:36382) 2023-11-16T16:59:58.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.635+0000 7f73bb7fe700 1 -- 172.21.15.2:0/2271336414 learned_addr learned my addr 172.21.15.2:0/2271336414 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:59:58.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.635+0000 7f73bbfff700 1 --2- 172.21.15.2:0/2271336414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f73bc105540 0x7f73bc19cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:58.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.635+0000 7f73baffd700 1 --2- 172.21.15.2:0/2271336414 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f73bc100ef0 0x7f73bc198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:58.640 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.635+0000 7f73bb7fe700 1 -- 172.21.15.2:0/2271336414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f73bc105540 msgr2=0x7f73bc19cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:58.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.635+0000 7f73bb7fe700 1 --2- 172.21.15.2:0/2271336414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f73bc105540 0x7f73bc19cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.635+0000 7f73bb7fe700 1 -- 172.21.15.2:0/2271336414 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f73bc100ef0 msgr2=0x7f73bc198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:58.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.635+0000 7f73bb7fe700 1 --2- 172.21.15.2:0/2271336414 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f73bc100ef0 0x7f73bc198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.635+0000 7f73bb7fe700 1 -- 172.21.15.2:0/2271336414 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b0009520 con 0x7f73bc100550 2023-11-16T16:59:58.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.635+0000 7f73baffd700 1 --2- 172.21.15.2:0/2271336414 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f73bc100ef0 0x7f73bc198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T16:59:58.641 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.635+0000 7f73bb7fe700 1 --2- 172.21.15.2:0/2271336414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73bc100550 0x7f73bc198640 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f73ac005910 tx=0x7f73ac004710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:58.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.636+0000 7f73b8ff9700 1 -- 172.21.15.2:0/2271336414 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac01d070 con 0x7f73bc100550 2023-11-16T16:59:58.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.636+0000 7f73c1dff700 1 -- 172.21.15.2:0/2271336414 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73ac0095f0 con 0x7f73bc100550 2023-11-16T16:59:58.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.636+0000 7f73b8ff9700 1 -- 172.21.15.2:0/2271336414 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73ac022470 con 0x7f73bc100550 2023-11-16T16:59:58.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.636+0000 7f73b8ff9700 1 -- 172.21.15.2:0/2271336414 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac017650 con 0x7f73bc100550 2023-11-16T16:59:58.642 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.636+0000 7f73c1dff700 1 -- 172.21.15.2:0/2271336414 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73bc110350 con 0x7f73bc100550 2023-11-16T16:59:58.643 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.637+0000 7f73c1dff700 1 -- 172.21.15.2:0/2271336414 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73bc1101c0 con 0x7f73bc100550 2023-11-16T16:59:58.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.639+0000 7f73b8ff9700 1 -- 172.21.15.2:0/2271336414 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f73ac0225e0 con 0x7f73bc100550 2023-11-16T16:59:58.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.640+0000 7f73b8ff9700 1 --2- 172.21.15.2:0/2271336414 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f73a40a12c0 0x7f73a40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:58.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.640+0000 7f73b8ff9700 1 -- 172.21.15.2:0/2271336414 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9482+0+0 (secure 0 0 0) 0x7f73ac0ce3a0 con 0x7f73bc100550 2023-11-16T16:59:58.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.640+0000 7f73baffd700 1 --2- 172.21.15.2:0/2271336414 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f73a40a12c0 0x7f73a40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:58.644 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.641+0000 7f73baffd700 1 --2- 172.21.15.2:0/2271336414 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f73a40a12c0 0x7f73a40a3770 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f73bc199c50 tx=0x7f73b0009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:58.645 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.643+0000 7f73b8ff9700 1 -- 172.21.15.2:0/2271336414 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f73ac05c730 con 0x7f73bc100550 2023-11-16T16:59:58.856 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.855+0000 7f73c1dff700 1 -- 172.21.15.2:0/2271336414 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73bc1104e0 con 0x7f73bc100550 2023-11-16T16:59:58.858 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.856+0000 7f73b8ff9700 1 -- 172.21.15.2:0/2271336414 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+25125 (secure 0 0 0) 0x7f73ac097460 con 0x7f73bc100550 2023-11-16T16:59:58.859 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T16:59:58.859 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":103,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T16:59:34.695878+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:24.458452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:25.786005+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:29.313181+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2834229980":"2023-11-17T16:49:54.626188+0000","172.21.15.2:0/3820370584":"2023-11-17T16:49:54.626188+0000","172.21.15.2:6801/1793067160":"2023-11-17T16:49:54.626188+0000","172.21.15.2:6800/1793067160":"2023-11-17T16:49:54.626188+0000","172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3784433150":"2023-11-17T16:49:54.626188+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/1824419243":"2023-11-17T16:49:54.626188+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T16:59:58.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.859+0000 7f73c1dff700 1 -- 172.21.15.2:0/2271336414 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f73a40a12c0 msgr2=0x7f73a40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:58.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.859+0000 7f73c1dff700 1 --2- 172.21.15.2:0/2271336414 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f73a40a12c0 0x7f73a40a3770 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f73bc199c50 tx=0x7f73b0009260 comp rx=0 tx=0).stop 2023-11-16T16:59:58.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.859+0000 7f73c1dff700 1 -- 172.21.15.2:0/2271336414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73bc100550 msgr2=0x7f73bc198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:58.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.859+0000 7f73c1dff700 1 --2- 172.21.15.2:0/2271336414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73bc100550 0x7f73bc198640 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f73ac005910 tx=0x7f73ac004710 comp rx=0 tx=0).stop 2023-11-16T16:59:58.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.859+0000 7f73c1dff700 1 -- 172.21.15.2:0/2271336414 shutdown_connections 2023-11-16T16:59:58.861 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.859+0000 7f73c1dff700 1 --2- 172.21.15.2:0/2271336414 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f73a40a12c0 0x7f73a40a3770 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.859+0000 7f73c1dff700 1 --2- 172.21.15.2:0/2271336414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f73bc105540 0x7f73bc19cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.859+0000 7f73c1dff700 1 --2- 172.21.15.2:0/2271336414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73bc100550 0x7f73bc198640 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.859+0000 7f73c1dff700 1 --2- 172.21.15.2:0/2271336414 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f73bc100ef0 0x7f73bc198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:58.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.859+0000 7f73c1dff700 1 -- 172.21.15.2:0/2271336414 >> 172.21.15.2:0/2271336414 conn(0x7f73bc077020 msgr2=0x7f73bc0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:58.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.859+0000 7f73c1dff700 1 -- 172.21.15.2:0/2271336414 shutdown_connections 2023-11-16T16:59:58.862 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:58.859+0000 7f73c1dff700 1 -- 172.21.15.2:0/2271336414 wait complete. 2023-11-16T16:59:58.879 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:04.306584+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T16:47:24.458452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T16:47:25.786005+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 102, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T16:47:29.313181+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 102, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T16:59:58.883 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T16:59:59.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.091+0000 7f223efff700 1 -- 172.21.15.2:0/3819273846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2238105620 msgr2=0x7f223810d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:59.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.091+0000 7f223efff700 1 --2- 172.21.15.2:0/3819273846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2238105620 0x7f223810d1a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f223000b0d0 tx=0x7f223000b3e0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.091+0000 7f223efff700 1 -- 172.21.15.2:0/3819273846 shutdown_connections 2023-11-16T16:59:59.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.091+0000 7f223efff700 1 --2- 172.21.15.2:0/3819273846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2238105620 0x7f223810d1a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.091+0000 7f223efff700 1 --2- 172.21.15.2:0/3819273846 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2238100cc0 0x7f2238101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.091+0000 7f223efff700 1 --2- 172.21.15.2:0/3819273846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2238101660 0x7f2238104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.093 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.091+0000 7f223efff700 1 -- 172.21.15.2:0/3819273846 >> 172.21.15.2:0/3819273846 conn(0x7f2238077560 msgr2=0x7f2238077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:59.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.091+0000 7f223efff700 1 -- 172.21.15.2:0/3819273846 shutdown_connections 2023-11-16T16:59:59.094 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.091+0000 7f223efff700 1 -- 172.21.15.2:0/3819273846 wait complete. 2023-11-16T16:59:59.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.092+0000 7f223efff700 1 Processor -- start 2023-11-16T16:59:59.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.092+0000 7f223efff700 1 -- start start 2023-11-16T16:59:59.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.093+0000 7f223efff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2238100cc0 0x7f2238196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:59.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.093+0000 7f223efff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2238101660 0x7f2238196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:59.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.093+0000 7f223efff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2238105620 0x7f223819afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:59.096 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.093+0000 7f223efff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223819b620 con 0x7f2238100cc0 2023-11-16T16:59:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.093+0000 7f223efff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223819b790 con 0x7f2238105620 2023-11-16T16:59:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.093+0000 7f223efff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223819b950 con 0x7f2238101660 2023-11-16T16:59:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.093+0000 7f223cd9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2238100cc0 0x7f2238196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.093+0000 7f223cd9b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2238100cc0 0x7f2238196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36400/0 (socket says 172.21.15.2:36400) 2023-11-16T16:59:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.093+0000 7f223cd9b700 1 -- 172.21.15.2:0/2008911749 learned_addr learned my addr 172.21.15.2:0/2008911749 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:59:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.093+0000 7f223d59c700 1 --2- 172.21.15.2:0/2008911749 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2238105620 0x7f223819afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.093+0000 7f2237fff700 1 --2- 172.21.15.2:0/2008911749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2238101660 0x7f2238196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.093+0000 7f223cd9b700 1 -- 172.21.15.2:0/2008911749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2238101660 msgr2=0x7f2238196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.093+0000 7f223cd9b700 1 --2- 172.21.15.2:0/2008911749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2238101660 0x7f2238196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.093+0000 7f223cd9b700 1 -- 172.21.15.2:0/2008911749 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2238105620 msgr2=0x7f223819afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.093+0000 7f223cd9b700 1 --2- 172.21.15.2:0/2008911749 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2238105620 0x7f223819afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.093+0000 7f223cd9b700 1 -- 172.21.15.2:0/2008911749 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2230009d00 con 0x7f2238100cc0 2023-11-16T16:59:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.094+0000 7f223cd9b700 1 --2- 172.21.15.2:0/2008911749 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2238100cc0 0x7f2238196640 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f222c00e9a0 tx=0x7f222c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.094+0000 7f2235ffb700 1 -- 172.21.15.2:0/2008911749 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c009790 con 0x7f2238100cc0 2023-11-16T16:59:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.094+0000 7f223efff700 1 -- 172.21.15.2:0/2008911749 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22381103e0 con 0x7f2238100cc0 2023-11-16T16:59:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.094+0000 7f2235ffb700 1 -- 172.21.15.2:0/2008911749 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f222c010430 con 0x7f2238100cc0 2023-11-16T16:59:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.094+0000 7f223efff700 1 -- 172.21.15.2:0/2008911749 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2238110250 con 0x7f2238100cc0 2023-11-16T16:59:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.094+0000 7f2235ffb700 1 -- 172.21.15.2:0/2008911749 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c018620 con 0x7f2238100cc0 2023-11-16T16:59:59.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.095+0000 7f223efff700 1 -- 172.21.15.2:0/2008911749 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2238110570 con 0x7f2238100cc0 2023-11-16T16:59:59.099 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.097+0000 7f2235ffb700 1 -- 172.21.15.2:0/2008911749 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f222c0105a0 con 0x7f2238100cc0 2023-11-16T16:59:59.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.099+0000 7f2235ffb700 1 --2- 172.21.15.2:0/2008911749 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f22280a12c0 0x7f22280a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:59.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.100+0000 7f2235ffb700 1 -- 172.21.15.2:0/2008911749 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9482+0+0 (secure 0 0 0) 0x7f222c014070 con 0x7f2238100cc0 2023-11-16T16:59:59.101 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.100+0000 7f2237fff700 1 --2- 172.21.15.2:0/2008911749 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f22280a12c0 0x7f22280a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:59.102 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.100+0000 7f2237fff700 1 --2- 172.21.15.2:0/2008911749 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f22280a12c0 0x7f22280a3770 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2238197c50 tx=0x7f2224009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:59.105 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.103+0000 7f2235ffb700 1 -- 172.21.15.2:0/2008911749 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f222c051730 con 0x7f2238100cc0 2023-11-16T16:59:59.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:58 smithi002 ceph-mon[128712]: pgmap v313: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:59:59.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:58 smithi002 ceph-mon[128712]: from='client.15882 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:59:59.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:58 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2271336414' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:59:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:58 smithi141 ceph-mon[131184]: pgmap v313: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:59:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:58 smithi141 ceph-mon[131184]: from='client.15882 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:59:59.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:58 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2271336414' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:59:59.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.311+0000 7f223efff700 1 -- 172.21.15.2:0/2008911749 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2238197a30 con 0x7f2238100cc0 2023-11-16T16:59:59.313 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.312+0000 7f2235ffb700 1 -- 172.21.15.2:0/2008911749 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) v1 ==== 93+0+10 (secure 0 0 0) 0x7f222c096280 con 0x7f2238100cc0 2023-11-16T16:59:59.314 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T16:59:59.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.314+0000 7f223efff700 1 -- 172.21.15.2:0/2008911749 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f22280a12c0 msgr2=0x7f22280a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:59.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.314+0000 7f223efff700 1 --2- 172.21.15.2:0/2008911749 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f22280a12c0 0x7f22280a3770 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2238197c50 tx=0x7f2224009260 comp rx=0 tx=0).stop 2023-11-16T16:59:59.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.314+0000 7f223efff700 1 -- 172.21.15.2:0/2008911749 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2238100cc0 msgr2=0x7f2238196640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:59.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.314+0000 7f223efff700 1 --2- 172.21.15.2:0/2008911749 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2238100cc0 0x7f2238196640 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f222c00e9a0 tx=0x7f222c00ecb0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.314+0000 7f223efff700 1 -- 172.21.15.2:0/2008911749 shutdown_connections 2023-11-16T16:59:59.316 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.314+0000 7f223efff700 1 --2- 172.21.15.2:0/2008911749 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f22280a12c0 0x7f22280a3770 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.314+0000 7f223efff700 1 --2- 172.21.15.2:0/2008911749 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2238105620 0x7f223819afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.314+0000 7f223efff700 1 --2- 172.21.15.2:0/2008911749 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2238100cc0 0x7f2238196640 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.314+0000 7f223efff700 1 --2- 172.21.15.2:0/2008911749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2238101660 0x7f2238196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.314+0000 7f223efff700 1 -- 172.21.15.2:0/2008911749 >> 172.21.15.2:0/2008911749 conn(0x7f2238077560 msgr2=0x7f2238103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:59.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.314+0000 7f223efff700 1 -- 172.21.15.2:0/2008911749 shutdown_connections 2023-11-16T16:59:59.317 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.314+0000 7f223efff700 1 -- 172.21.15.2:0/2008911749 wait complete. 2023-11-16T16:59:59.334 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T16:59:59.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:58 smithi155 ceph-mon[131607]: pgmap v313: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T16:59:59.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:58 smithi155 ceph-mon[131607]: from='client.15882 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T16:59:59.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:58 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2271336414' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T16:59:59.542 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.541+0000 7f29afbff700 1 -- 172.21.15.2:0/3050649163 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f29a8101660 msgr2=0x7f29a8104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:59.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.541+0000 7f29afbff700 1 --2- 172.21.15.2:0/3050649163 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f29a8101660 0x7f29a8104ef0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f29980099c0 tx=0x7f2998009cd0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.541+0000 7f29afbff700 1 -- 172.21.15.2:0/3050649163 shutdown_connections 2023-11-16T16:59:59.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.541+0000 7f29afbff700 1 --2- 172.21.15.2:0/3050649163 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29a8100cc0 0x7f29a8101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.541+0000 7f29afbff700 1 --2- 172.21.15.2:0/3050649163 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29a8105620 0x7f29a810d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.543 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.541+0000 7f29afbff700 1 --2- 172.21.15.2:0/3050649163 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f29a8101660 0x7f29a8104ef0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.544 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.541+0000 7f29afbff700 1 -- 172.21.15.2:0/3050649163 >> 172.21.15.2:0/3050649163 conn(0x7f29a8077560 msgr2=0x7f29a8077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:59.544 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.541+0000 7f29afbff700 1 -- 172.21.15.2:0/3050649163 shutdown_connections 2023-11-16T16:59:59.544 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.541+0000 7f29afbff700 1 -- 172.21.15.2:0/3050649163 wait complete. 2023-11-16T16:59:59.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.542+0000 7f29afbff700 1 Processor -- start 2023-11-16T16:59:59.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.542+0000 7f29afbff700 1 -- start start 2023-11-16T16:59:59.547 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.543+0000 7f29afbff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29a8100cc0 0x7f29a8072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:59.548 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.543+0000 7f29afbff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f29a8101660 0x7f29a8073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:59.548 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.543+0000 7f29afbff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29a8105620 0x7f29a81a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:59.548 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.543+0000 7f29afbff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a8073b60 con 0x7f29a8100cc0 2023-11-16T16:59:59.548 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.543+0000 7f29afbff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a81a4b80 con 0x7f29a8101660 2023-11-16T16:59:59.548 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.543+0000 7f29afbff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a81a4cc0 con 0x7f29a8105620 2023-11-16T16:59:59.548 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.543+0000 7f29ad99b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29a8100cc0 0x7f29a8072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:59.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.543+0000 7f29ad99b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29a8100cc0 0x7f29a8072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36414/0 (socket says 172.21.15.2:36414) 2023-11-16T16:59:59.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.543+0000 7f29ad99b700 1 -- 172.21.15.2:0/3104847907 learned_addr learned my addr 172.21.15.2:0/3104847907 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:59:59.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.543+0000 7f29ae19c700 1 --2- 172.21.15.2:0/3104847907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29a8105620 0x7f29a81a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:59.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.543+0000 7f29ad19a700 1 --2- 172.21.15.2:0/3104847907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f29a8101660 0x7f29a8073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:59.549 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.544+0000 7f29ad99b700 1 -- 172.21.15.2:0/3104847907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29a8105620 msgr2=0x7f29a81a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:59.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.544+0000 7f29ad99b700 1 --2- 172.21.15.2:0/3104847907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29a8105620 0x7f29a81a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.544+0000 7f29ad99b700 1 -- 172.21.15.2:0/3104847907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f29a8101660 msgr2=0x7f29a8073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:59.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.544+0000 7f29ad99b700 1 --2- 172.21.15.2:0/3104847907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f29a8101660 0x7f29a8073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.544+0000 7f29ad99b700 1 -- 172.21.15.2:0/3104847907 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29980095f0 con 0x7f29a8100cc0 2023-11-16T16:59:59.550 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.544+0000 7f29ad99b700 1 --2- 172.21.15.2:0/3104847907 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29a8100cc0 0x7f29a8072df0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f29a400e9a0 tx=0x7f29a400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:59.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.544+0000 7f299effd700 1 -- 172.21.15.2:0/3104847907 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a4009790 con 0x7f29a8100cc0 2023-11-16T16:59:59.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.544+0000 7f299effd700 1 -- 172.21.15.2:0/3104847907 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f29a4010430 con 0x7f29a8100cc0 2023-11-16T16:59:59.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.544+0000 7f299effd700 1 -- 172.21.15.2:0/3104847907 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a4018680 con 0x7f29a8100cc0 2023-11-16T16:59:59.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.544+0000 7f29afbff700 1 -- 172.21.15.2:0/3104847907 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a8110320 con 0x7f29a8100cc0 2023-11-16T16:59:59.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.544+0000 7f29afbff700 1 -- 172.21.15.2:0/3104847907 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a8110190 con 0x7f29a8100cc0 2023-11-16T16:59:59.551 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.546+0000 7f29afbff700 1 -- 172.21.15.2:0/3104847907 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a81104b0 con 0x7f29a8100cc0 2023-11-16T16:59:59.552 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.547+0000 7f299effd700 1 -- 172.21.15.2:0/3104847907 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f29a40105a0 con 0x7f29a8100cc0 2023-11-16T16:59:59.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.549+0000 7f299effd700 1 --2- 172.21.15.2:0/3104847907 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f29940a12c0 0x7f29940a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:59.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.549+0000 7f299effd700 1 -- 172.21.15.2:0/3104847907 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9482+0+0 (secure 0 0 0) 0x7f29a4014070 con 0x7f29a8100cc0 2023-11-16T16:59:59.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.549+0000 7f29ad19a700 1 --2- 172.21.15.2:0/3104847907 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f29940a12c0 0x7f29940a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:59.553 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.550+0000 7f29ad19a700 1 --2- 172.21.15.2:0/3104847907 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f29940a12c0 0x7f29940a3770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f29980098e0 tx=0x7f2998000bc0 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:59.554 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.553+0000 7f299effd700 1 -- 172.21.15.2:0/3104847907 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f29a4051730 con 0x7f29a8100cc0 2023-11-16T16:59:59.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.759+0000 7f29afbff700 1 -- 172.21.15.2:0/3104847907 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f29a8074320 con 0x7f29a8100cc0 2023-11-16T16:59:59.761 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.760+0000 7f299effd700 1 -- 172.21.15.2:0/3104847907 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) v1 ==== 104+0+11 (secure 0 0 0) 0x7f29a8074320 con 0x7f29a8100cc0 2023-11-16T16:59:59.762 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T16:59:59.764 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.762+0000 7f29afbff700 1 -- 172.21.15.2:0/3104847907 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f29940a12c0 msgr2=0x7f29940a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:59.764 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.762+0000 7f29afbff700 1 --2- 172.21.15.2:0/3104847907 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f29940a12c0 0x7f29940a3770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f29980098e0 tx=0x7f2998000bc0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.764 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.762+0000 7f29afbff700 1 -- 172.21.15.2:0/3104847907 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29a8100cc0 msgr2=0x7f29a8072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:59.764 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.762+0000 7f29afbff700 1 --2- 172.21.15.2:0/3104847907 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29a8100cc0 0x7f29a8072df0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f29a400e9a0 tx=0x7f29a400ecb0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.762+0000 7f29afbff700 1 -- 172.21.15.2:0/3104847907 shutdown_connections 2023-11-16T16:59:59.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.762+0000 7f29afbff700 1 --2- 172.21.15.2:0/3104847907 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f29940a12c0 0x7f29940a3770 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.762+0000 7f29afbff700 1 --2- 172.21.15.2:0/3104847907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29a8105620 0x7f29a81a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.762+0000 7f29afbff700 1 --2- 172.21.15.2:0/3104847907 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29a8100cc0 0x7f29a8072df0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.765 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.762+0000 7f29afbff700 1 --2- 172.21.15.2:0/3104847907 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f29a8101660 0x7f29a8073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.762+0000 7f29afbff700 1 -- 172.21.15.2:0/3104847907 >> 172.21.15.2:0/3104847907 conn(0x7f29a8077560 msgr2=0x7f29a80fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:59.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.762+0000 7f29afbff700 1 -- 172.21.15.2:0/3104847907 shutdown_connections 2023-11-16T16:59:59.766 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.762+0000 7f29afbff700 1 -- 172.21.15.2:0/3104847907 wait complete. 2023-11-16T16:59:59.783 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T16:59:59.994 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.992+0000 7f2a0cdff700 1 -- 172.21.15.2:0/3506408125 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a0810a8b0 msgr2=0x7f2a0810ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:59.994 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.992+0000 7f2a0cdff700 1 --2- 172.21.15.2:0/3506408125 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a0810a8b0 0x7f2a0810ccd0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2a0000b0a0 tx=0x7f2a0000b3b0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.994 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.992+0000 7f2a0cdff700 1 -- 172.21.15.2:0/3506408125 shutdown_connections 2023-11-16T16:59:59.995 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.992+0000 7f2a0cdff700 1 --2- 172.21.15.2:0/3506408125 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a08101390 0x7f2a08101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.995 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.992+0000 7f2a0cdff700 1 --2- 172.21.15.2:0/3506408125 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a0810a8b0 0x7f2a0810ccd0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.995 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.992+0000 7f2a0cdff700 1 --2- 172.21.15.2:0/3506408125 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a08101ca0 0x7f2a0810a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.995 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.992+0000 7f2a0cdff700 1 -- 172.21.15.2:0/3506408125 >> 172.21.15.2:0/3506408125 conn(0x7f2a080770b0 msgr2=0x7f2a080774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T16:59:59.995 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.992+0000 7f2a0cdff700 1 -- 172.21.15.2:0/3506408125 shutdown_connections 2023-11-16T16:59:59.995 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.992+0000 7f2a0cdff700 1 -- 172.21.15.2:0/3506408125 wait complete. 2023-11-16T16:59:59.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.993+0000 7f2a0cdff700 1 Processor -- start 2023-11-16T16:59:59.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.993+0000 7f2a0cdff700 1 -- start start 2023-11-16T16:59:59.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.994+0000 7f2a0cdff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a08101390 0x7f2a08196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:59.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.994+0000 7f2a0cdff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a08101ca0 0x7f2a08196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:59.997 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.994+0000 7f2a0cdff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a0810a8b0 0x7f2a0819af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T16:59:59.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.994+0000 7f2a0cdff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0819b550 con 0x7f2a08101390 2023-11-16T16:59:59.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.994+0000 7f2a0cdff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0819b6c0 con 0x7f2a0810a8b0 2023-11-16T16:59:59.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.994+0000 7f2a0cdff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0819b880 con 0x7f2a08101ca0 2023-11-16T16:59:59.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.994+0000 7f2a0659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a08101390 0x7f2a08196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:59.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.994+0000 7f2a0659c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a08101390 0x7f2a08196570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36424/0 (socket says 172.21.15.2:36424) 2023-11-16T16:59:59.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.994+0000 7f2a0659c700 1 -- 172.21.15.2:0/3803389771 learned_addr learned my addr 172.21.15.2:0/3803389771 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T16:59:59.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.994+0000 7f29fffff700 1 --2- 172.21.15.2:0/3803389771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a08101ca0 0x7f2a08196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:59.998 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.994+0000 7f2a06d9d700 1 --2- 172.21.15.2:0/3803389771 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a0810a8b0 0x7f2a0819af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T16:59:59.999 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.995+0000 7f2a0659c700 1 -- 172.21.15.2:0/3803389771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a08101ca0 msgr2=0x7f2a08196ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:59.999 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.995+0000 7f2a0659c700 1 --2- 172.21.15.2:0/3803389771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a08101ca0 0x7f2a08196ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.999 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.995+0000 7f2a0659c700 1 -- 172.21.15.2:0/3803389771 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a0810a8b0 msgr2=0x7f2a0819af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T16:59:59.999 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.995+0000 7f2a0659c700 1 --2- 172.21.15.2:0/3803389771 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a0810a8b0 0x7f2a0819af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T16:59:59.999 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.995+0000 7f2a0659c700 1 -- 172.21.15.2:0/3803389771 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a00009d00 con 0x7f2a08101390 2023-11-16T16:59:59.999 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.995+0000 7f2a0659c700 1 --2- 172.21.15.2:0/3803389771 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a08101390 0x7f2a08196570 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f29f800e9a0 tx=0x7f29f800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T16:59:59.999 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.995+0000 7f29ff7fe700 1 -- 172.21.15.2:0/3803389771 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f8009790 con 0x7f2a08101390 2023-11-16T17:00:00.000 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.995+0000 7f29ff7fe700 1 -- 172.21.15.2:0/3803389771 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f29f8010430 con 0x7f2a08101390 2023-11-16T17:00:00.000 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.995+0000 7f29ff7fe700 1 -- 172.21.15.2:0/3803389771 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f8018680 con 0x7f2a08101390 2023-11-16T17:00:00.000 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.995+0000 7f2a0cdff700 1 -- 172.21.15.2:0/3803389771 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a08110350 con 0x7f2a08101390 2023-11-16T17:00:00.000 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.995+0000 7f2a0cdff700 1 -- 172.21.15.2:0/3803389771 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a081101c0 con 0x7f2a08101390 2023-11-16T17:00:00.000 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.996+0000 7f2a0cdff700 1 -- 172.21.15.2:0/3803389771 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a081104e0 con 0x7f2a08101390 2023-11-16T17:00:00.001 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T16:59:59.998+0000 7f29ff7fe700 1 -- 172.21.15.2:0/3803389771 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f29f80105a0 con 0x7f2a08101390 2023-11-16T17:00:00.001 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.000+0000 7f29ff7fe700 1 --2- 172.21.15.2:0/3803389771 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f29f40a1340 0x7f29f40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:00.001 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.000+0000 7f29ff7fe700 1 -- 172.21.15.2:0/3803389771 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9482+0+0 (secure 0 0 0) 0x7f29f8014070 con 0x7f2a08101390 2023-11-16T17:00:00.001 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.000+0000 7f29fffff700 1 --2- 172.21.15.2:0/3803389771 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f29f40a1340 0x7f29f40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:00.002 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.001+0000 7f29fffff700 1 --2- 172.21.15.2:0/3803389771 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f29f40a1340 0x7f29f40a37f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2a08197b80 tx=0x7f29f0009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:00.003 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.002+0000 7f29ff7fe700 1 -- 172.21.15.2:0/3803389771 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f29f80964b0 con 0x7f2a08101390 2023-11-16T17:00:00.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:59 smithi002 ceph-mon[128712]: from='client.15888 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T17:00:00.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:59 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2008911749' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T17:00:00.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 16:59:59 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3104847907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T17:00:00.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.211+0000 7f2a0cdff700 1 -- 172.21.15.2:0/3803389771 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2a0819bad0 con 0x7f2a08101390 2023-11-16T17:00:00.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.211+0000 7f29ff7fe700 1 -- 172.21.15.2:0/3803389771 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) v1 ==== 100+0+11 (secure 0 0 0) 0x7f29f8026070 con 0x7f2a08101390 2023-11-16T17:00:00.213 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2023-11-16T17:00:00.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.213+0000 7f2a0cdff700 1 -- 172.21.15.2:0/3803389771 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f29f40a1340 msgr2=0x7f29f40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:00.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.213+0000 7f2a0cdff700 1 --2- 172.21.15.2:0/3803389771 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f29f40a1340 0x7f29f40a37f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2a08197b80 tx=0x7f29f0009260 comp rx=0 tx=0).stop 2023-11-16T17:00:00.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.213+0000 7f2a0cdff700 1 -- 172.21.15.2:0/3803389771 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a08101390 msgr2=0x7f2a08196570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:00.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.213+0000 7f2a0cdff700 1 --2- 172.21.15.2:0/3803389771 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a08101390 0x7f2a08196570 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f29f800e9a0 tx=0x7f29f800ecb0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.214+0000 7f2a0cdff700 1 -- 172.21.15.2:0/3803389771 shutdown_connections 2023-11-16T17:00:00.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.214+0000 7f2a0cdff700 1 --2- 172.21.15.2:0/3803389771 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f29f40a1340 0x7f29f40a37f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.214+0000 7f2a0cdff700 1 --2- 172.21.15.2:0/3803389771 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f2a0810a8b0 0x7f2a0819af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.214+0000 7f2a0cdff700 1 --2- 172.21.15.2:0/3803389771 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2a08101390 0x7f2a08196570 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.214+0000 7f2a0cdff700 1 --2- 172.21.15.2:0/3803389771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a08101ca0 0x7f2a08196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.214+0000 7f2a0cdff700 1 -- 172.21.15.2:0/3803389771 >> 172.21.15.2:0/3803389771 conn(0x7f2a080770b0 msgr2=0x7f2a080fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:00.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.214+0000 7f2a0cdff700 1 -- 172.21.15.2:0/3803389771 shutdown_connections 2023-11-16T17:00:00.217 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.214+0000 7f2a0cdff700 1 -- 172.21.15.2:0/3803389771 wait complete. 2023-11-16T17:00:00.234 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T17:00:00.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:59 smithi141 ceph-mon[131184]: from='client.15888 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T17:00:00.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:59 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2008911749' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T17:00:00.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 16:59:59 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3104847907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T17:00:00.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:59 smithi155 ceph-mon[131607]: from='client.15888 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-16T17:00:00.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:59 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2008911749' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T17:00:00.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 16:59:59 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3104847907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T17:00:00.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.445+0000 7fbea11ff700 1 -- 172.21.15.2:0/1248509754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe9c100a50 msgr2=0x7fbe9c100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:00.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.445+0000 7fbea11ff700 1 --2- 172.21.15.2:0/1248509754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe9c100a50 0x7fbe9c100ec0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fbe90009990 tx=0x7fbe90009ca0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.447 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.445+0000 7fbea11ff700 1 -- 172.21.15.2:0/1248509754 shutdown_connections 2023-11-16T17:00:00.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.445+0000 7fbea11ff700 1 --2- 172.21.15.2:0/1248509754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe9c101400 0x7fbe9c10d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.445+0000 7fbea11ff700 1 --2- 172.21.15.2:0/1248509754 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe9c1063f0 0x7fbe9c1067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.445+0000 7fbea11ff700 1 --2- 172.21.15.2:0/1248509754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe9c100a50 0x7fbe9c100ec0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.445+0000 7fbea11ff700 1 -- 172.21.15.2:0/1248509754 >> 172.21.15.2:0/1248509754 conn(0x7fbe9c077590 msgr2=0x7fbe9c077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:00.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.445+0000 7fbea11ff700 1 -- 172.21.15.2:0/1248509754 shutdown_connections 2023-11-16T17:00:00.448 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.446+0000 7fbea11ff700 1 -- 172.21.15.2:0/1248509754 wait complete. 2023-11-16T17:00:00.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.446+0000 7fbea11ff700 1 Processor -- start 2023-11-16T17:00:00.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.447+0000 7fbea11ff700 1 -- start start 2023-11-16T17:00:00.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.447+0000 7fbea11ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe9c100a50 0x7fbe9c196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:00.451 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.447+0000 7fbea11ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe9c101400 0x7fbe9c196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:00.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.447+0000 7fbea11ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe9c1063f0 0x7fbe9c19adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:00.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.447+0000 7fbea11ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c19b3f0 con 0x7fbe9c100a50 2023-11-16T17:00:00.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.447+0000 7fbea11ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c19b560 con 0x7fbe9c1063f0 2023-11-16T17:00:00.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.447+0000 7fbea11ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c19b720 con 0x7fbe9c101400 2023-11-16T17:00:00.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.447+0000 7fbe9ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe9c100a50 0x7fbe9c196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:00.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.447+0000 7fbe9ad9d700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe9c100a50 0x7fbe9c196410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36434/0 (socket says 172.21.15.2:36434) 2023-11-16T17:00:00.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.447+0000 7fbe9ad9d700 1 -- 172.21.15.2:0/765228245 learned_addr learned my addr 172.21.15.2:0/765228245 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T17:00:00.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.447+0000 7fbe9a59c700 1 --2- 172.21.15.2:0/765228245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe9c101400 0x7fbe9c196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:00.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.447+0000 7fbe9b59e700 1 --2- 172.21.15.2:0/765228245 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe9c1063f0 0x7fbe9c19adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:00.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.448+0000 7fbe9ad9d700 1 -- 172.21.15.2:0/765228245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe9c101400 msgr2=0x7fbe9c196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:00.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.448+0000 7fbe9ad9d700 1 --2- 172.21.15.2:0/765228245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe9c101400 0x7fbe9c196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.448+0000 7fbe9ad9d700 1 -- 172.21.15.2:0/765228245 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe9c1063f0 msgr2=0x7fbe9c19adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:00.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.448+0000 7fbe9ad9d700 1 --2- 172.21.15.2:0/765228245 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe9c1063f0 0x7fbe9c19adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.448+0000 7fbe9ad9d700 1 -- 172.21.15.2:0/765228245 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe900095f0 con 0x7fbe9c100a50 2023-11-16T17:00:00.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.448+0000 7fbe9b59e700 1 --2- 172.21.15.2:0/765228245 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe9c1063f0 0x7fbe9c19adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T17:00:00.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.448+0000 7fbe9ad9d700 1 --2- 172.21.15.2:0/765228245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe9c100a50 0x7fbe9c196410 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00e9a0 tx=0x7fbe8c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:00.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.448+0000 7fbe83fff700 1 -- 172.21.15.2:0/765228245 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c009790 con 0x7fbe9c100a50 2023-11-16T17:00:00.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.448+0000 7fbe83fff700 1 -- 172.21.15.2:0/765228245 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe8c00cbd0 con 0x7fbe9c100a50 2023-11-16T17:00:00.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.448+0000 7fbe83fff700 1 -- 172.21.15.2:0/765228245 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c018680 con 0x7fbe9c100a50 2023-11-16T17:00:00.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.448+0000 7fbea11ff700 1 -- 172.21.15.2:0/765228245 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9c110410 con 0x7fbe9c100a50 2023-11-16T17:00:00.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.448+0000 7fbea11ff700 1 -- 172.21.15.2:0/765228245 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9c110280 con 0x7fbe9c100a50 2023-11-16T17:00:00.455 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.449+0000 7fbea11ff700 1 -- 172.21.15.2:0/765228245 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9c1105a0 con 0x7fbe9c100a50 2023-11-16T17:00:00.455 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.451+0000 7fbe83fff700 1 -- 172.21.15.2:0/765228245 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fbe8c00cd40 con 0x7fbe9c100a50 2023-11-16T17:00:00.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.453+0000 7fbe83fff700 1 --2- 172.21.15.2:0/765228245 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fbe840a12c0 0x7fbe840a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:00.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.453+0000 7fbe83fff700 1 -- 172.21.15.2:0/765228245 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9482+0+0 (secure 0 0 0) 0x7fbe8c014070 con 0x7fbe9c100a50 2023-11-16T17:00:00.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.453+0000 7fbe9a59c700 1 --2- 172.21.15.2:0/765228245 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fbe840a12c0 0x7fbe840a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:00.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.454+0000 7fbe9a59c700 1 --2- 172.21.15.2:0/765228245 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fbe840a12c0 0x7fbe840a3770 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c197a20 tx=0x7fbe90005fd0 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:00.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.457+0000 7fbe83fff700 1 -- 172.21.15.2:0/765228245 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbe8c051730 con 0x7fbe9c100a50 2023-11-16T17:00:00.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.666+0000 7fbea11ff700 1 -- 172.21.15.2:0/765228245 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fbe9c1a3480 con 0x7fbe9c100a50 2023-11-16T17:00:00.668 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.667+0000 7fbe83fff700 1 -- 172.21.15.2:0/765228245 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v103) v1 ==== 103+0+11 (secure 0 0 0) 0x7fbe8c096320 con 0x7fbe9c100a50 2023-11-16T17:00:00.668 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T17:00:00.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.669+0000 7fbea11ff700 1 -- 172.21.15.2:0/765228245 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fbe840a12c0 msgr2=0x7fbe840a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:00.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.669+0000 7fbea11ff700 1 --2- 172.21.15.2:0/765228245 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fbe840a12c0 0x7fbe840a3770 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c197a20 tx=0x7fbe90005fd0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.669+0000 7fbea11ff700 1 -- 172.21.15.2:0/765228245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe9c100a50 msgr2=0x7fbe9c196410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:00.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.669+0000 7fbea11ff700 1 --2- 172.21.15.2:0/765228245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe9c100a50 0x7fbe9c196410 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00e9a0 tx=0x7fbe8c00ecb0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.671 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.669+0000 7fbea11ff700 1 -- 172.21.15.2:0/765228245 shutdown_connections 2023-11-16T17:00:00.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.669+0000 7fbea11ff700 1 --2- 172.21.15.2:0/765228245 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fbe840a12c0 0x7fbe840a3770 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.669+0000 7fbea11ff700 1 --2- 172.21.15.2:0/765228245 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fbe9c1063f0 0x7fbe9c19adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.669+0000 7fbea11ff700 1 --2- 172.21.15.2:0/765228245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe9c100a50 0x7fbe9c196410 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.669+0000 7fbea11ff700 1 --2- 172.21.15.2:0/765228245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe9c101400 0x7fbe9c196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.669+0000 7fbea11ff700 1 -- 172.21.15.2:0/765228245 >> 172.21.15.2:0/765228245 conn(0x7fbe9c077590 msgr2=0x7fbe9c0ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:00.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.669+0000 7fbea11ff700 1 -- 172.21.15.2:0/765228245 shutdown_connections 2023-11-16T17:00:00.672 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.669+0000 7fbea11ff700 1 -- 172.21.15.2:0/765228245 wait complete. 2023-11-16T17:00:00.690 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2023-11-16T17:00:00.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.893+0000 7f1cc37ff700 1 -- 172.21.15.2:0/3992079033 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1cbc100550 msgr2=0x7f1cbc100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:00.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.893+0000 7f1cc37ff700 1 --2- 172.21.15.2:0/3992079033 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1cbc100550 0x7f1cbc100920 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1cb80099c0 tx=0x7f1cb8009cd0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.893+0000 7f1cc37ff700 1 -- 172.21.15.2:0/3992079033 shutdown_connections 2023-11-16T17:00:00.895 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.893+0000 7f1cc37ff700 1 --2- 172.21.15.2:0/3992079033 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1cbc105540 0x7f1cbc10d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.893+0000 7f1cc37ff700 1 --2- 172.21.15.2:0/3992079033 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1cbc100550 0x7f1cbc100920 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.893+0000 7f1cc37ff700 1 --2- 172.21.15.2:0/3992079033 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1cbc100ef0 0x7f1cbc104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.893+0000 7f1cc37ff700 1 -- 172.21.15.2:0/3992079033 >> 172.21.15.2:0/3992079033 conn(0x7f1cbc077020 msgr2=0x7f1cbc077420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:00.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.894+0000 7f1cc37ff700 1 -- 172.21.15.2:0/3992079033 shutdown_connections 2023-11-16T17:00:00.896 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.894+0000 7f1cc37ff700 1 -- 172.21.15.2:0/3992079033 wait complete. 2023-11-16T17:00:00.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.895+0000 7f1cc37ff700 1 Processor -- start 2023-11-16T17:00:00.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.895+0000 7f1cc37ff700 1 -- start start 2023-11-16T17:00:00.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.895+0000 7f1cc37ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1cbc100550 0x7f1cbc072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:00.898 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.895+0000 7f1cc37ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1cbc100ef0 0x7f1cbc073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:00.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.895+0000 7f1cc37ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1cbc105540 0x7f1cbc1a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:00.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.895+0000 7f1cc37ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc073b60 con 0x7f1cbc100ef0 2023-11-16T17:00:00.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.895+0000 7f1cc37ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc1a4ad0 con 0x7f1cbc105540 2023-11-16T17:00:00.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.895+0000 7f1cc37ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc1a4c10 con 0x7f1cbc100550 2023-11-16T17:00:00.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.895+0000 7f1cc0d9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1cbc100ef0 0x7f1cbc073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:00.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.895+0000 7f1cc0d9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1cbc100ef0 0x7f1cbc073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36460/0 (socket says 172.21.15.2:36460) 2023-11-16T17:00:00.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.895+0000 7f1cc0d9a700 1 -- 172.21.15.2:0/2962512053 learned_addr learned my addr 172.21.15.2:0/2962512053 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T17:00:00.899 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.895+0000 7f1cc159b700 1 --2- 172.21.15.2:0/2962512053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1cbc100550 0x7f1cbc072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:00.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.896+0000 7f1cc1d9c700 1 --2- 172.21.15.2:0/2962512053 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1cbc105540 0x7f1cbc1a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:00.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.896+0000 7f1cc159b700 1 -- 172.21.15.2:0/2962512053 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1cbc105540 msgr2=0x7f1cbc1a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:00.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.896+0000 7f1cc159b700 1 --2- 172.21.15.2:0/2962512053 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1cbc105540 0x7f1cbc1a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.896+0000 7f1cc159b700 1 -- 172.21.15.2:0/2962512053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1cbc100ef0 msgr2=0x7f1cbc073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:00.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.896+0000 7f1cc159b700 1 --2- 172.21.15.2:0/2962512053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1cbc100ef0 0x7f1cbc073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:00.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.896+0000 7f1cc159b700 1 -- 172.21.15.2:0/2962512053 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb80095f0 con 0x7f1cbc100550 2023-11-16T17:00:00.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.896+0000 7f1cc1d9c700 1 --2- 172.21.15.2:0/2962512053 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1cbc105540 0x7f1cbc1a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-16T17:00:00.900 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.896+0000 7f1cc159b700 1 --2- 172.21.15.2:0/2962512053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1cbc100550 0x7f1cbc072df0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1cb80098e0 tx=0x7f1cb800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:00.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.896+0000 7f1cb27fc700 1 -- 172.21.15.2:0/2962512053 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb801d070 con 0x7f1cbc100550 2023-11-16T17:00:00.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.896+0000 7f1cc37ff700 1 -- 172.21.15.2:0/2962512053 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cbc110350 con 0x7f1cbc100550 2023-11-16T17:00:00.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.896+0000 7f1cb27fc700 1 -- 172.21.15.2:0/2962512053 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1cb8005510 con 0x7f1cbc100550 2023-11-16T17:00:00.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.896+0000 7f1cb27fc700 1 -- 172.21.15.2:0/2962512053 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8004b30 con 0x7f1cbc100550 2023-11-16T17:00:00.901 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.897+0000 7f1cc37ff700 1 -- 172.21.15.2:0/2962512053 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cbc1101c0 con 0x7f1cbc100550 2023-11-16T17:00:00.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.897+0000 7f1cc37ff700 1 -- 172.21.15.2:0/2962512053 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cbc1104e0 con 0x7f1cbc100550 2023-11-16T17:00:00.902 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.900+0000 7f1cb27fc700 1 -- 172.21.15.2:0/2962512053 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f1cb800fb80 con 0x7f1cbc100550 2023-11-16T17:00:00.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.901+0000 7f1cb27fc700 1 --2- 172.21.15.2:0/2962512053 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f1ca80a12c0 0x7f1ca80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:00.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.901+0000 7f1cb27fc700 1 -- 172.21.15.2:0/2962512053 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9482+0+0 (secure 0 0 0) 0x7f1cb80ce090 con 0x7f1cbc100550 2023-11-16T17:00:00.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.901+0000 7f1cc0d9a700 1 --2- 172.21.15.2:0/2962512053 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f1ca80a12c0 0x7f1ca80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:00.903 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.902+0000 7f1cc0d9a700 1 --2- 172.21.15.2:0/2962512053 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f1ca80a12c0 0x7f1ca80a3770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc0745e0 tx=0x7f1cac009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:00.905 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:00.903+0000 7f1cb27fc700 1 -- 172.21.15.2:0/2962512053 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1cb8097110 con 0x7f1cbc100550 2023-11-16T17:00:01.189 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:01.188+0000 7f1cc37ff700 1 -- 172.21.15.2:0/2962512053 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f1cbc067a60 con 0x7f1cbc100550 2023-11-16T17:00:01.198 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:01.197+0000 7f1cb27fc700 1 -- 172.21.15.2:0/2962512053 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) v1 ==== 81+0+733765 (secure 0 0 0) 0x7f1cb809a730 con 0x7f1cbc100550 2023-11-16T17:00:01.210 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T17:00:01.210 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T17:00:01.211 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 25, 2023-11-16T17:00:01.211 INFO:teuthology.orchestra.run.smithi002.stdout: "active_gid": 15372, 2023-11-16T17:00:01.211 INFO:teuthology.orchestra.run.smithi002.stdout: "active_name": "x", 2023-11-16T17:00:01.211 INFO:teuthology.orchestra.run.smithi002.stdout: "active_addrs": { 2023-11-16T17:00:01.211 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-16T17:00:01.211 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.211 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-16T17:00:01.212 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6800", 2023-11-16T17:00:01.212 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1352003968 2023-11-16T17:00:01.212 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.212 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.212 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2023-11-16T17:00:01.212 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6801", 2023-11-16T17:00:01.212 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1352003968 2023-11-16T17:00:01.212 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.213 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T17:00:01.213 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.213 INFO:teuthology.orchestra.run.smithi002.stdout: "active_addr": "172.21.15.2:6801/1352003968", 2023-11-16T17:00:01.213 INFO:teuthology.orchestra.run.smithi002.stdout: "active_change": "2023-11-16T16:49:54.626440+0000", 2023-11-16T17:00:01.213 INFO:teuthology.orchestra.run.smithi002.stdout: "active_mgr_features": 4540138322906710015, 2023-11-16T17:00:01.213 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2023-11-16T17:00:01.213 INFO:teuthology.orchestra.run.smithi002.stdout: "standbys": [ 2023-11-16T17:00:01.214 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.214 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 25363, 2023-11-16T17:00:01.214 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "z", 2023-11-16T17:00:01.214 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_features": 4540138322906710015, 2023-11-16T17:00:01.214 INFO:teuthology.orchestra.run.smithi002.stdout: "available_modules": [ 2023-11-16T17:00:01.214 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.214 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alerts", 2023-11-16T17:00:01.215 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.215 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.215 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.215 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T17:00:01.215 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T17:00:01.215 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.215 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.215 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.216 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T17:00:01.216 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.216 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.216 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.216 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How frequently to reexamine health status", 2023-11-16T17:00:01.216 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.216 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.217 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.217 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.217 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.217 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.217 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.217 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.217 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.217 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.218 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.218 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.218 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.218 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.218 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.218 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.219 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.219 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.219 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.219 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.219 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.219 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.219 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.220 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.220 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.220 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.220 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.220 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.220 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.220 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.221 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.221 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.221 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.221 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.221 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.221 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.221 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.221 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.222 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.222 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.222 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.222 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.222 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.222 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.222 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.222 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.223 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.223 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.223 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.223 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.223 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.223 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.223 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.224 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.224 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.224 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.224 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.224 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.224 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.224 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.224 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.225 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.225 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.225 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.225 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.225 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.225 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.225 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.226 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.226 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.226 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.226 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.226 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.226 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.226 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_destination": { 2023-11-16T17:00:01.226 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_destination", 2023-11-16T17:00:01.227 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.227 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.227 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.227 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.227 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.227 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.227 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.228 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Email address to send alerts to", 2023-11-16T17:00:01.228 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.228 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.228 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.228 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.228 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_from_name": { 2023-11-16T17:00:01.228 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_from_name", 2023-11-16T17:00:01.229 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.229 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.229 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.229 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "Ceph", 2023-11-16T17:00:01.229 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.229 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.229 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.230 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Email From: name", 2023-11-16T17:00:01.230 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.230 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.230 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.230 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.230 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_host": { 2023-11-16T17:00:01.230 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_host", 2023-11-16T17:00:01.231 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.231 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.231 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.231 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.231 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.231 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.231 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.232 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP server", 2023-11-16T17:00:01.232 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.232 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.232 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.232 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.232 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_password": { 2023-11-16T17:00:01.232 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_password", 2023-11-16T17:00:01.233 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.233 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.233 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.233 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.233 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.233 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.233 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.233 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Password to authenticate with", 2023-11-16T17:00:01.234 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.234 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.234 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.234 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.234 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_port": { 2023-11-16T17:00:01.234 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_port", 2023-11-16T17:00:01.234 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.235 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.235 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.235 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "465", 2023-11-16T17:00:01.235 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.235 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.235 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.235 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP port", 2023-11-16T17:00:01.235 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.236 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.236 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.236 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.236 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_sender": { 2023-11-16T17:00:01.236 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_sender", 2023-11-16T17:00:01.236 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.236 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.236 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.237 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.237 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.237 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.237 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.237 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP envelope sender", 2023-11-16T17:00:01.237 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.237 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.237 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.238 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.238 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_ssl": { 2023-11-16T17:00:01.238 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_ssl", 2023-11-16T17:00:01.238 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.238 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.238 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.239 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.239 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.239 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.239 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.239 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use SSL to connect to SMTP server", 2023-11-16T17:00:01.239 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.239 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.239 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.240 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.240 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_user": { 2023-11-16T17:00:01.240 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_user", 2023-11-16T17:00:01.240 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.240 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.240 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.240 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.241 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.241 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.241 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.241 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "User to authenticate as", 2023-11-16T17:00:01.241 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.241 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.241 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.241 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.242 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.242 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.242 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.242 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "balancer", 2023-11-16T17:00:01.242 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.242 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.242 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.242 INFO:teuthology.orchestra.run.smithi002.stdout: "active": { 2023-11-16T17:00:01.243 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "active", 2023-11-16T17:00:01.243 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.243 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.243 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.243 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.243 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.243 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.244 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.244 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "automatically balance PGs across cluster", 2023-11-16T17:00:01.244 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.244 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.244 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.244 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.244 INFO:teuthology.orchestra.run.smithi002.stdout: "begin_time": { 2023-11-16T17:00:01.244 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "begin_time", 2023-11-16T17:00:01.245 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.245 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.245 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.245 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0000", 2023-11-16T17:00:01.245 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.245 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.245 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.245 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "beginning time of day to automatically balance", 2023-11-16T17:00:01.246 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-16T17:00:01.246 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.246 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.246 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.246 INFO:teuthology.orchestra.run.smithi002.stdout: "begin_weekday": { 2023-11-16T17:00:01.246 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "begin_weekday", 2023-11-16T17:00:01.246 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T17:00:01.246 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.247 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.247 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T17:00:01.247 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0", 2023-11-16T17:00:01.247 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "6", 2023-11-16T17:00:01.247 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.247 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-11-16T17:00:01.247 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-16T17:00:01.247 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.248 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.248 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.248 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_max_iterations": { 2023-11-16T17:00:01.248 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_max_iterations", 2023-11-16T17:00:01.248 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T17:00:01.248 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.248 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.248 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "25", 2023-11-16T17:00:01.249 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T17:00:01.249 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "250", 2023-11-16T17:00:01.249 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.249 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-11-16T17:00:01.249 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.249 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.249 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.249 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.250 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_metrics": { 2023-11-16T17:00:01.250 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_metrics", 2023-11-16T17:00:01.250 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.250 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.250 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.250 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "pgs,objects,bytes", 2023-11-16T17:00:01.250 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.250 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.251 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.251 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-11-16T17:00:01.251 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-11-16T17:00:01.251 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.251 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.251 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.251 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_step": { 2023-11-16T17:00:01.251 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_step", 2023-11-16T17:00:01.251 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:01.252 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.252 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.252 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0.5", 2023-11-16T17:00:01.252 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0.001", 2023-11-16T17:00:01.252 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "0.999", 2023-11-16T17:00:01.252 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.252 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "aggressiveness of optimization", 2023-11-16T17:00:01.252 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-11-16T17:00:01.253 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.253 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.253 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.253 INFO:teuthology.orchestra.run.smithi002.stdout: "end_time": { 2023-11-16T17:00:01.253 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_time", 2023-11-16T17:00:01.253 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.253 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.253 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.254 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "2359", 2023-11-16T17:00:01.254 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.254 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.254 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.254 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "ending time of day to automatically balance", 2023-11-16T17:00:01.254 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-16T17:00:01.254 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.254 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.255 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.255 INFO:teuthology.orchestra.run.smithi002.stdout: "end_weekday": { 2023-11-16T17:00:01.255 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_weekday", 2023-11-16T17:00:01.255 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T17:00:01.255 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.255 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.255 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T17:00:01.255 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0", 2023-11-16T17:00:01.256 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "6", 2023-11-16T17:00:01.256 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.256 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-11-16T17:00:01.256 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-16T17:00:01.256 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.256 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.256 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.256 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.257 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.257 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.257 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.257 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.257 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.257 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.257 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.257 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.258 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.258 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.258 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.258 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.258 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.258 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.258 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.258 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.259 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.259 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.259 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.259 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.259 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.259 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.259 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.259 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.260 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.260 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.260 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.260 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.260 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.260 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.260 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.260 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.261 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.261 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.261 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.261 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.261 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.261 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.261 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.261 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.262 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.262 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.262 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.262 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.262 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.262 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.262 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.262 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.263 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.263 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.263 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.263 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.263 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.263 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.263 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.263 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.263 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.264 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.264 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.264 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.264 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.264 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.264 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.264 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.264 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.265 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.265 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.265 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.265 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.265 INFO:teuthology.orchestra.run.smithi002.stdout: "min_score": { 2023-11-16T17:00:01.265 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "min_score", 2023-11-16T17:00:01.265 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:01.265 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.266 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.266 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T17:00:01.266 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.266 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.266 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.266 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-11-16T17:00:01.266 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.266 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.267 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.267 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.267 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": { 2023-11-16T17:00:01.267 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mode", 2023-11-16T17:00:01.267 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.267 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.267 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.267 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "upmap", 2023-11-16T17:00:01.268 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.268 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.268 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.268 INFO:teuthology.orchestra.run.smithi002.stdout: "crush-compat", 2023-11-16T17:00:01.268 INFO:teuthology.orchestra.run.smithi002.stdout: "none", 2023-11-16T17:00:01.268 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap" 2023-11-16T17:00:01.268 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.268 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Balancer mode", 2023-11-16T17:00:01.268 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.269 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.269 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.269 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.269 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_ids": { 2023-11-16T17:00:01.269 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pool_ids", 2023-11-16T17:00:01.269 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.269 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.270 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.270 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.270 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.270 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.270 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.270 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-11-16T17:00:01.270 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.270 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.271 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.271 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.271 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T17:00:01.271 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T17:00:01.271 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.271 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.271 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.271 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T17:00:01.271 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.272 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.272 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.272 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-11-16T17:00:01.272 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.272 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.272 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.272 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.272 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap_max_deviation": { 2023-11-16T17:00:01.273 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "upmap_max_deviation", 2023-11-16T17:00:01.273 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.273 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.273 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.273 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T17:00:01.273 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T17:00:01.273 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.273 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.273 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "deviation below which no optimization is attempted", 2023-11-16T17:00:01.274 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-11-16T17:00:01.274 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.274 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.274 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.274 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap_max_optimizations": { 2023-11-16T17:00:01.274 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "upmap_max_optimizations", 2023-11-16T17:00:01.274 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T17:00:01.274 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.275 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.275 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:01.275 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.275 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.275 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.275 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-11-16T17:00:01.275 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.275 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.275 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.276 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.276 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.276 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.276 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.276 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephadm", 2023-11-16T17:00:01.276 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.276 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.276 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.277 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_down_multiplier": { 2023-11-16T17:00:01.277 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_down_multiplier", 2023-11-16T17:00:01.277 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:01.277 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.277 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.277 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3.0", 2023-11-16T17:00:01.277 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.277 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.277 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.278 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-11-16T17:00:01.278 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.278 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.278 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.278 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.278 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_refresh_rate": { 2023-11-16T17:00:01.278 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_refresh_rate", 2023-11-16T17:00:01.278 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.279 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.279 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.279 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "20", 2023-11-16T17:00:01.279 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.279 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.279 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.279 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-11-16T17:00:01.279 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.280 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.280 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.280 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.280 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_starting_port": { 2023-11-16T17:00:01.280 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_starting_port", 2023-11-16T17:00:01.280 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.280 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.280 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.281 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "4721", 2023-11-16T17:00:01.281 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.281 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.281 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.281 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-11-16T17:00:01.281 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.281 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.281 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.282 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.282 INFO:teuthology.orchestra.run.smithi002.stdout: "alertmanager_web_password": { 2023-11-16T17:00:01.282 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alertmanager_web_password", 2023-11-16T17:00:01.282 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.282 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.282 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.282 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:01.282 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.283 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.283 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.283 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Alertmanager web password", 2023-11-16T17:00:01.283 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.283 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.283 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.283 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.283 INFO:teuthology.orchestra.run.smithi002.stdout: "alertmanager_web_user": { 2023-11-16T17:00:01.283 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alertmanager_web_user", 2023-11-16T17:00:01.284 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.284 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.284 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.284 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:01.284 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.284 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.284 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.284 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Alertmanager web user", 2023-11-16T17:00:01.284 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.285 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.285 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.285 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.285 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_ptrace": { 2023-11-16T17:00:01.285 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_ptrace", 2023-11-16T17:00:01.285 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.285 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.286 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.286 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.286 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.286 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.286 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.286 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-11-16T17:00:01.286 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-11-16T17:00:01.286 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.287 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.287 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.287 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_interval": { 2023-11-16T17:00:01.287 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_interval", 2023-11-16T17:00:01.287 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.287 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.287 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.287 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T17:00:01.288 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.288 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.288 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.288 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to autotune daemon memory", 2023-11-16T17:00:01.288 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.288 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.288 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.288 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.289 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_memory_target_ratio": { 2023-11-16T17:00:01.289 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_memory_target_ratio", 2023-11-16T17:00:01.289 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:01.289 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.289 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.289 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0.7", 2023-11-16T17:00:01.289 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.289 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.290 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.290 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-11-16T17:00:01.290 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.290 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.290 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.290 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.290 INFO:teuthology.orchestra.run.smithi002.stdout: "cgroups_split": { 2023-11-16T17:00:01.291 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cgroups_split", 2023-11-16T17:00:01.291 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.291 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.291 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.291 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.291 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.291 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.291 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.292 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-11-16T17:00:01.292 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.292 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.292 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.292 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.292 INFO:teuthology.orchestra.run.smithi002.stdout: "config_checks_enabled": { 2023-11-16T17:00:01.292 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_checks_enabled", 2023-11-16T17:00:01.292 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.293 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.293 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.293 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.293 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.293 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.293 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.293 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-11-16T17:00:01.293 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.294 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.294 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.294 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.294 INFO:teuthology.orchestra.run.smithi002.stdout: "config_dashboard": { 2023-11-16T17:00:01.294 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_dashboard", 2023-11-16T17:00:01.294 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.294 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.294 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.295 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.295 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.295 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.295 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.295 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-11-16T17:00:01.295 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.295 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.295 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.296 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.296 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_alertmanager": { 2023-11-16T17:00:01.296 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_alertmanager", 2023-11-16T17:00:01.296 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.296 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.296 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.296 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-11-16T17:00:01.296 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.296 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.297 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.297 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:01.297 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.297 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.297 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.297 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.297 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_base": { 2023-11-16T17:00:01.297 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_base", 2023-11-16T17:00:01.298 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.298 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.298 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.298 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/ceph", 2023-11-16T17:00:01.298 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.298 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.298 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.298 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Container image name, without the tag", 2023-11-16T17:00:01.299 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.299 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.299 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.299 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.299 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_elasticsearch": { 2023-11-16T17:00:01.299 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_elasticsearch", 2023-11-16T17:00:01.299 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.299 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.299 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.300 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-11-16T17:00:01.300 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.300 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.300 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.300 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "elasticsearch container image", 2023-11-16T17:00:01.300 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.300 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.300 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.300 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.301 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_grafana": { 2023-11-16T17:00:01.301 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_grafana", 2023-11-16T17:00:01.301 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.301 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.301 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.301 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-11-16T17:00:01.301 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.301 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.302 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.302 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:01.302 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.302 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.302 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.302 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.302 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_haproxy": { 2023-11-16T17:00:01.302 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_haproxy", 2023-11-16T17:00:01.303 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.303 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.303 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.303 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-11-16T17:00:01.303 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.303 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.303 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.303 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "HAproxy container image", 2023-11-16T17:00:01.303 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.304 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.304 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.304 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.304 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_agent": { 2023-11-16T17:00:01.304 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_agent", 2023-11-16T17:00:01.304 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.304 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.304 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.304 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-11-16T17:00:01.305 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.305 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.305 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.305 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger agent container image", 2023-11-16T17:00:01.305 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.305 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.305 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.305 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.305 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_collector": { 2023-11-16T17:00:01.306 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_collector", 2023-11-16T17:00:01.306 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.306 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.306 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.306 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-11-16T17:00:01.306 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.306 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.306 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.307 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger collector container image", 2023-11-16T17:00:01.307 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.307 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.307 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.307 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.307 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_query": { 2023-11-16T17:00:01.307 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_query", 2023-11-16T17:00:01.307 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.307 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.308 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.308 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-11-16T17:00:01.308 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.308 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.308 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.308 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger query container image", 2023-11-16T17:00:01.308 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.308 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.308 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.309 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.309 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_keepalived": { 2023-11-16T17:00:01.309 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_keepalived", 2023-11-16T17:00:01.309 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.309 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.309 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.309 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-11-16T17:00:01.309 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.310 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.310 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.310 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Keepalived container image", 2023-11-16T17:00:01.310 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.310 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.310 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.310 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.310 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_loki": { 2023-11-16T17:00:01.311 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_loki", 2023-11-16T17:00:01.311 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.311 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.311 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.311 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-11-16T17:00:01.311 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.311 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.311 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.311 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Loki container image", 2023-11-16T17:00:01.312 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.312 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.312 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.312 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.312 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_node_exporter": { 2023-11-16T17:00:01.312 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_node_exporter", 2023-11-16T17:00:01.312 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.312 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.312 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.313 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-11-16T17:00:01.313 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.313 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.313 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.313 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:01.313 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.313 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.313 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.313 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.314 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_prometheus": { 2023-11-16T17:00:01.314 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_prometheus", 2023-11-16T17:00:01.314 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.314 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.314 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.314 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-11-16T17:00:01.314 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.314 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.315 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.315 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:01.315 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.315 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.315 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.315 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.315 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_promtail": { 2023-11-16T17:00:01.315 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_promtail", 2023-11-16T17:00:01.315 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.316 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.316 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.316 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-11-16T17:00:01.316 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.316 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.316 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.316 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Promtail container image", 2023-11-16T17:00:01.316 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.316 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.317 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.317 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.317 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_snmp_gateway": { 2023-11-16T17:00:01.317 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_snmp_gateway", 2023-11-16T17:00:01.317 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.317 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.317 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.317 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-11-16T17:00:01.317 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.318 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.318 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.318 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SNMP Gateway container image", 2023-11-16T17:00:01.318 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.318 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.318 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.318 INFO:teuthology.orchestra.run.smithi002.stdout: "container_init": { 2023-11-16T17:00:01.319 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_init", 2023-11-16T17:00:01.319 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.319 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.319 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.319 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.319 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.319 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.319 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.320 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Run podman/docker with `--init`", 2023-11-16T17:00:01.320 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.320 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.320 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.320 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.320 INFO:teuthology.orchestra.run.smithi002.stdout: "daemon_cache_timeout": { 2023-11-16T17:00:01.320 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "daemon_cache_timeout", 2023-11-16T17:00:01.320 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.321 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.321 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.321 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T17:00:01.321 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.321 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.321 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.321 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-11-16T17:00:01.321 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.321 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.322 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.322 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.322 INFO:teuthology.orchestra.run.smithi002.stdout: "default_cephadm_command_timeout": { 2023-11-16T17:00:01.322 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default_cephadm_command_timeout", 2023-11-16T17:00:01.322 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.322 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.322 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.322 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "900", 2023-11-16T17:00:01.323 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.323 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.323 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.323 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-11-16T17:00:01.323 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.323 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.323 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.323 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.323 INFO:teuthology.orchestra.run.smithi002.stdout: "default_registry": { 2023-11-16T17:00:01.324 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default_registry", 2023-11-16T17:00:01.324 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.324 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.324 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.324 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io", 2023-11-16T17:00:01.324 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.324 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.324 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.324 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-11-16T17:00:01.325 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.325 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.325 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.325 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.325 INFO:teuthology.orchestra.run.smithi002.stdout: "device_cache_timeout": { 2023-11-16T17:00:01.325 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_cache_timeout", 2023-11-16T17:00:01.325 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.325 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.325 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.326 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1800", 2023-11-16T17:00:01.326 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.326 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.326 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.326 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache device inventory", 2023-11-16T17:00:01.326 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.326 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.326 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.327 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.327 INFO:teuthology.orchestra.run.smithi002.stdout: "device_enhanced_scan": { 2023-11-16T17:00:01.327 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_enhanced_scan", 2023-11-16T17:00:01.327 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.327 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.327 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.327 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.327 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.327 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.328 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.328 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use libstoragemgmt during device scans", 2023-11-16T17:00:01.328 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.328 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.328 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.328 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.328 INFO:teuthology.orchestra.run.smithi002.stdout: "facts_cache_timeout": { 2023-11-16T17:00:01.328 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "facts_cache_timeout", 2023-11-16T17:00:01.329 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.329 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.329 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.329 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T17:00:01.329 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.329 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.329 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.329 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache host facts data", 2023-11-16T17:00:01.330 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.330 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.330 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.330 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.330 INFO:teuthology.orchestra.run.smithi002.stdout: "host_check_interval": { 2023-11-16T17:00:01.330 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "host_check_interval", 2023-11-16T17:00:01.330 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.331 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.331 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.331 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T17:00:01.331 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.331 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.331 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.331 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to perform a host check", 2023-11-16T17:00:01.331 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.332 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.332 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.332 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.332 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.332 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.332 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.332 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.332 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.332 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.333 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.333 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.333 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.333 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.333 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.333 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.333 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.333 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.334 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.334 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.334 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.334 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.334 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.334 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.334 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.335 INFO:teuthology.orchestra.run.smithi002.stdout: "log_refresh_metadata": { 2023-11-16T17:00:01.335 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_refresh_metadata", 2023-11-16T17:00:01.335 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.335 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.335 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.335 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.335 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.335 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.335 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.336 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-11-16T17:00:01.336 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.336 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.336 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.336 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.336 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.336 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.336 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.337 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.337 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.337 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.337 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.337 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.337 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.337 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-11-16T17:00:01.337 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.337 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.338 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.338 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.338 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.338 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.338 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.338 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.338 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.338 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.339 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.339 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.339 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.339 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.339 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.339 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.339 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.340 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.340 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.340 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.340 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.340 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.340 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.340 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.340 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.341 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.341 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.341 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.341 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.341 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.341 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.341 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.341 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.342 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.342 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.342 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.342 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.342 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.342 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.342 INFO:teuthology.orchestra.run.smithi002.stdout: "manage_etc_ceph_ceph_conf": { 2023-11-16T17:00:01.342 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-11-16T17:00:01.342 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.343 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.343 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.343 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.343 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.343 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.343 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.343 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-11-16T17:00:01.344 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.344 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.344 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.344 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.344 INFO:teuthology.orchestra.run.smithi002.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-11-16T17:00:01.344 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-11-16T17:00:01.344 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.344 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.345 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.345 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "*", 2023-11-16T17:00:01.345 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.345 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.345 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.345 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-11-16T17:00:01.345 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.345 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.346 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.346 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.346 INFO:teuthology.orchestra.run.smithi002.stdout: "max_count_per_host": { 2023-11-16T17:00:01.346 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_count_per_host", 2023-11-16T17:00:01.346 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.346 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.346 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.346 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:01.347 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.347 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.347 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.347 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "max number of daemons per service per host", 2023-11-16T17:00:01.347 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.347 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.347 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.347 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.347 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd_draining_count": { 2023-11-16T17:00:01.348 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_osd_draining_count", 2023-11-16T17:00:01.348 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.348 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.348 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.348 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:01.348 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.348 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.348 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.349 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-11-16T17:00:01.349 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.349 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.349 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.349 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.349 INFO:teuthology.orchestra.run.smithi002.stdout: "migration_current": { 2023-11-16T17:00:01.349 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "migration_current", 2023-11-16T17:00:01.349 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.350 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.350 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.350 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.350 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.350 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.350 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.350 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "internal - do not modify", 2023-11-16T17:00:01.351 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.351 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.351 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.351 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.351 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": { 2023-11-16T17:00:01.351 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mode", 2023-11-16T17:00:01.351 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.351 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.351 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.352 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "root", 2023-11-16T17:00:01.352 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.352 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.352 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.352 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm-package", 2023-11-16T17:00:01.352 INFO:teuthology.orchestra.run.smithi002.stdout: "root" 2023-11-16T17:00:01.352 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.353 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "mode for remote execution of cephadm", 2023-11-16T17:00:01.353 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.353 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.353 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.353 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.353 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_alerts_path": { 2023-11-16T17:00:01.353 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_alerts_path", 2023-11-16T17:00:01.353 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.353 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.354 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.354 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-11-16T17:00:01.354 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.354 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.354 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.354 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-11-16T17:00:01.354 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.355 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.355 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.355 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.355 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_web_password": { 2023-11-16T17:00:01.355 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_web_password", 2023-11-16T17:00:01.355 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.355 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.355 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.356 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:01.356 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.356 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.356 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.356 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus web password", 2023-11-16T17:00:01.356 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.356 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.356 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.357 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.357 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_web_user": { 2023-11-16T17:00:01.357 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_web_user", 2023-11-16T17:00:01.357 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.357 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.357 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.357 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:01.357 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.358 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.358 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.358 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus web user", 2023-11-16T17:00:01.358 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.358 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.358 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.358 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.359 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_insecure": { 2023-11-16T17:00:01.359 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_insecure", 2023-11-16T17:00:01.359 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.359 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.359 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.359 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.359 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.359 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.360 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.360 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-11-16T17:00:01.360 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.360 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.360 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.360 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.360 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_password": { 2023-11-16T17:00:01.360 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_password", 2023-11-16T17:00:01.361 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.361 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.361 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.361 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.361 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.361 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.361 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.361 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-11-16T17:00:01.362 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.362 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.362 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.362 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.362 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_url": { 2023-11-16T17:00:01.362 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_url", 2023-11-16T17:00:01.362 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.363 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.363 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.363 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.363 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.363 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.363 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.363 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-11-16T17:00:01.363 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.364 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.364 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.364 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.364 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_username": { 2023-11-16T17:00:01.364 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_username", 2023-11-16T17:00:01.364 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.364 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.364 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.364 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.365 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.365 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.365 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.365 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-11-16T17:00:01.365 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.365 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.365 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.366 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.366 INFO:teuthology.orchestra.run.smithi002.stdout: "secure_monitoring_stack": { 2023-11-16T17:00:01.366 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "secure_monitoring_stack", 2023-11-16T17:00:01.366 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.366 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.366 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.366 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.366 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.367 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.367 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.367 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-11-16T17:00:01.367 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.367 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.367 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.367 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.367 INFO:teuthology.orchestra.run.smithi002.stdout: "service_discovery_port": { 2023-11-16T17:00:01.368 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "service_discovery_port", 2023-11-16T17:00:01.368 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.368 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.368 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.368 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8765", 2023-11-16T17:00:01.368 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.368 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.368 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.369 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "cephadm service discovery port", 2023-11-16T17:00:01.369 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.369 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.369 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.369 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.369 INFO:teuthology.orchestra.run.smithi002.stdout: "ssh_config_file": { 2023-11-16T17:00:01.369 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssh_config_file", 2023-11-16T17:00:01.370 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.370 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.370 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.370 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.370 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.370 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.370 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.370 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-11-16T17:00:01.371 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.371 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.371 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.371 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.371 INFO:teuthology.orchestra.run.smithi002.stdout: "use_agent": { 2023-11-16T17:00:01.371 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "use_agent", 2023-11-16T17:00:01.371 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.371 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.371 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.372 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.372 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.372 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.372 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.372 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-11-16T17:00:01.372 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.372 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.372 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.373 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.373 INFO:teuthology.orchestra.run.smithi002.stdout: "use_repo_digest": { 2023-11-16T17:00:01.373 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "use_repo_digest", 2023-11-16T17:00:01.373 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.373 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.373 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.373 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.373 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.373 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.374 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.374 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-11-16T17:00:01.374 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.374 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.374 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.374 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.374 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_failed_host_check": { 2023-11-16T17:00:01.374 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_failed_host_check", 2023-11-16T17:00:01.375 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.375 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.375 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.375 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.375 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.375 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.375 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.375 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if the host check fails", 2023-11-16T17:00:01.376 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.376 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.376 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.376 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.376 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_stray_daemons": { 2023-11-16T17:00:01.376 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_stray_daemons", 2023-11-16T17:00:01.376 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.376 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.377 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.377 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.377 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.377 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.377 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.377 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-11-16T17:00:01.377 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.377 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.378 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.378 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.378 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_stray_hosts": { 2023-11-16T17:00:01.378 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_stray_hosts", 2023-11-16T17:00:01.378 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.378 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.378 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.378 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.379 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.379 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.379 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.379 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-11-16T17:00:01.379 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.379 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.379 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.379 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.380 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.380 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.380 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.380 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crash", 2023-11-16T17:00:01.380 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.380 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.380 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.380 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.381 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.381 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.381 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.381 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.381 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.381 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.381 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.381 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.382 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.382 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.382 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.382 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.382 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.382 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.382 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.383 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.383 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.383 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.383 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.383 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.383 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.383 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.383 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.384 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.384 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.384 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.384 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.384 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.384 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.384 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.384 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.384 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.385 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.385 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.385 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.385 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.385 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.385 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.385 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.386 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.386 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.386 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.386 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.386 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.386 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.386 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.386 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.386 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.387 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.387 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.387 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.387 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.387 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.387 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.387 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.387 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.388 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.388 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.388 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.388 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.388 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.388 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.388 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.388 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.389 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.389 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.389 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.389 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.389 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.389 INFO:teuthology.orchestra.run.smithi002.stdout: "retain_interval": { 2023-11-16T17:00:01.389 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retain_interval", 2023-11-16T17:00:01.389 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.390 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.390 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.390 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "31536000", 2023-11-16T17:00:01.390 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.390 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.390 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.390 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long to retain crashes before pruning them", 2023-11-16T17:00:01.390 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.391 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.391 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.391 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.391 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_recent_interval": { 2023-11-16T17:00:01.391 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_recent_interval", 2023-11-16T17:00:01.391 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.391 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.391 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.392 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1209600", 2023-11-16T17:00:01.392 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.392 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.392 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.392 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "time interval in which to warn about recent crashes", 2023-11-16T17:00:01.392 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.392 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.392 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.393 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.393 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.393 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.393 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.393 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "dashboard", 2023-11-16T17:00:01.393 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.393 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.393 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.394 INFO:teuthology.orchestra.run.smithi002.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-11-16T17:00:01.394 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-11-16T17:00:01.394 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.394 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.394 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.394 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:01.394 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.394 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.395 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.395 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.395 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.395 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.395 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.395 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.395 INFO:teuthology.orchestra.run.smithi002.stdout: "ALERTMANAGER_API_HOST": { 2023-11-16T17:00:01.395 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ALERTMANAGER_API_HOST", 2023-11-16T17:00:01.395 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.396 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.396 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.396 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.396 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.396 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.396 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.396 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.396 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.396 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.397 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.397 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.397 INFO:teuthology.orchestra.run.smithi002.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-11-16T17:00:01.397 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-11-16T17:00:01.397 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.397 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.397 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.397 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.398 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.398 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.398 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.398 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.398 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.398 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.398 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.398 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.398 INFO:teuthology.orchestra.run.smithi002.stdout: "AUDIT_API_ENABLED": { 2023-11-16T17:00:01.399 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "AUDIT_API_ENABLED", 2023-11-16T17:00:01.399 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.399 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.399 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.399 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.399 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.399 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.399 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.400 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.400 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.400 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.400 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.400 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.400 INFO:teuthology.orchestra.run.smithi002.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-11-16T17:00:01.400 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-11-16T17:00:01.400 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.401 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.401 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.401 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.401 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.401 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.401 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.401 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.401 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.401 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.402 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.402 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.402 INFO:teuthology.orchestra.run.smithi002.stdout: "ENABLE_BROWSABLE_API": { 2023-11-16T17:00:01.402 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ENABLE_BROWSABLE_API", 2023-11-16T17:00:01.402 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.402 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.402 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.403 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.403 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.403 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.403 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.403 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.403 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.403 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.403 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.404 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.404 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-11-16T17:00:01.404 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-11-16T17:00:01.404 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.404 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.404 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.404 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.404 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.405 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.405 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.405 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.405 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.405 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.405 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.405 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.405 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-11-16T17:00:01.405 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-11-16T17:00:01.406 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.406 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.406 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.406 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.406 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.406 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.406 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.407 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.407 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.407 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.407 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.407 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.407 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-11-16T17:00:01.407 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-11-16T17:00:01.407 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.407 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.408 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.408 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.408 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.408 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.408 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.408 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.408 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.409 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.409 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.409 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.409 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-11-16T17:00:01.409 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-11-16T17:00:01.409 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.409 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.409 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.410 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.410 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.410 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.410 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.410 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.410 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.410 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.411 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.411 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.411 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_NFS": { 2023-11-16T17:00:01.411 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-11-16T17:00:01.411 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.411 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.411 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.411 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.412 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.412 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.412 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.412 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.412 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.412 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.412 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.412 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.413 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RBD": { 2023-11-16T17:00:01.413 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-11-16T17:00:01.413 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.413 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.413 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.413 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.413 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.413 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.414 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.414 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.414 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.414 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.414 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.414 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.414 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RGW": { 2023-11-16T17:00:01.415 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-11-16T17:00:01.415 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.415 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.415 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.415 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.415 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.415 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.415 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.415 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.416 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.416 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.416 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.416 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.416 INFO:teuthology.orchestra.run.smithi002.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-11-16T17:00:01.416 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-11-16T17:00:01.416 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.416 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.417 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.417 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.417 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.417 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.417 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.417 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.417 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.417 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.418 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.418 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.418 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_PASSWORD": { 2023-11-16T17:00:01.418 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_PASSWORD", 2023-11-16T17:00:01.418 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.418 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.418 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.418 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:01.419 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.419 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.419 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.419 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.419 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.419 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.419 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.420 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.420 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-11-16T17:00:01.420 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-11-16T17:00:01.420 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.420 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.420 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.420 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.420 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.420 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.421 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.421 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.421 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.421 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.421 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.421 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.421 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_URL": { 2023-11-16T17:00:01.421 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_URL", 2023-11-16T17:00:01.422 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.422 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.422 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.422 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.422 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.422 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.422 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.422 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.423 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.423 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.423 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.423 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.423 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_USERNAME": { 2023-11-16T17:00:01.423 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_USERNAME", 2023-11-16T17:00:01.423 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.423 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.424 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.424 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:01.424 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.424 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.424 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.424 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.424 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.424 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.425 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.425 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.425 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-11-16T17:00:01.425 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-11-16T17:00:01.425 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.425 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.425 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.425 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.425 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.426 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.426 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.426 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.426 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.426 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.426 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.426 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.426 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-11-16T17:00:01.427 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-11-16T17:00:01.427 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.427 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.427 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.427 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.427 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.427 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.427 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.428 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.428 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.428 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.428 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.428 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.428 INFO:teuthology.orchestra.run.smithi002.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-11-16T17:00:01.428 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-11-16T17:00:01.428 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.429 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.429 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.429 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.429 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.429 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.429 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.429 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.429 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.430 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.430 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.430 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.430 INFO:teuthology.orchestra.run.smithi002.stdout: "ISSUE_TRACKER_API_KEY": { 2023-11-16T17:00:01.430 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-11-16T17:00:01.430 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.430 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.430 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.431 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.431 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.431 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.431 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.431 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.431 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.431 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.431 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.432 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.432 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_HOST": { 2023-11-16T17:00:01.432 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_HOST", 2023-11-16T17:00:01.432 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.432 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.432 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.432 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.432 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.432 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.433 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.433 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.433 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.433 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.433 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.433 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.433 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-11-16T17:00:01.433 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-11-16T17:00:01.434 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.434 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.434 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.434 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.434 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.434 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.434 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.434 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.435 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.435 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.435 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.435 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.435 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-11-16T17:00:01.435 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-11-16T17:00:01.435 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.435 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.435 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.436 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.436 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.436 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.436 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.436 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.436 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.436 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.436 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.436 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.437 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-11-16T17:00:01.437 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-11-16T17:00:01.437 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.437 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.437 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.437 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.437 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.437 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.437 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.438 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.438 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.438 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.438 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.438 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.438 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-11-16T17:00:01.438 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-11-16T17:00:01.438 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.438 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.439 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.439 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.439 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.439 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.439 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.439 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.439 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.440 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.440 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.440 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.440 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-11-16T17:00:01.440 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-11-16T17:00:01.440 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.440 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.440 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.440 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.441 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.441 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.441 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.441 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.441 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.441 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.441 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.441 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.442 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-11-16T17:00:01.442 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-11-16T17:00:01.442 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.442 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.442 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.442 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.442 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.442 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.443 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.443 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.443 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.443 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.443 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.443 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.443 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-11-16T17:00:01.443 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-11-16T17:00:01.444 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.444 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.444 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.444 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.444 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.444 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.444 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.444 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.445 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.445 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.445 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.445 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.445 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-11-16T17:00:01.445 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-11-16T17:00:01.445 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.445 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.446 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.446 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.446 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.446 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.446 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.446 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.446 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.446 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.447 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.447 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.447 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_ENABLED": { 2023-11-16T17:00:01.447 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_ENABLED", 2023-11-16T17:00:01.447 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.447 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.447 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.447 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.448 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.448 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.448 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.448 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.448 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.448 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.448 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.448 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.448 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-11-16T17:00:01.449 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-11-16T17:00:01.449 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.449 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.449 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.449 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-11-16T17:00:01.449 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.449 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.450 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.450 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.450 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.450 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.450 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.450 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.450 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-11-16T17:00:01.450 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-11-16T17:00:01.451 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.451 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.451 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.451 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:01.451 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.451 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.451 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.451 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.452 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.452 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.452 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.452 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.452 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-11-16T17:00:01.452 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-11-16T17:00:01.452 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.452 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.453 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.453 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8", 2023-11-16T17:00:01.453 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.453 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.453 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.453 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.453 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.453 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.453 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.454 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.454 INFO:teuthology.orchestra.run.smithi002.stdout: "REST_REQUESTS_TIMEOUT": { 2023-11-16T17:00:01.454 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-11-16T17:00:01.454 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.454 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.454 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.454 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "45", 2023-11-16T17:00:01.454 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.455 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.455 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.455 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.455 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.455 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.455 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.455 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.455 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_ACCESS_KEY": { 2023-11-16T17:00:01.456 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_ACCESS_KEY", 2023-11-16T17:00:01.456 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.456 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.456 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.456 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.456 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.456 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.456 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.456 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.457 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.457 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.457 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.457 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.457 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-11-16T17:00:01.457 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-11-16T17:00:01.457 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.457 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.458 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.458 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:01.458 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.458 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.458 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.458 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.458 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.459 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.459 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.459 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.459 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_SECRET_KEY": { 2023-11-16T17:00:01.459 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_SECRET_KEY", 2023-11-16T17:00:01.459 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.459 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.459 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.460 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.460 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.460 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.460 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.460 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.460 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.460 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.460 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.461 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.461 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_SSL_VERIFY": { 2023-11-16T17:00:01.461 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_SSL_VERIFY", 2023-11-16T17:00:01.461 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.461 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.461 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.461 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.461 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.461 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.462 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.462 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.462 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.462 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.462 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.462 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.462 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-11-16T17:00:01.462 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-11-16T17:00:01.463 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.463 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.463 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.463 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T17:00:01.463 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.463 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.463 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.463 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.464 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.464 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.464 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.464 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.464 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-11-16T17:00:01.464 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-11-16T17:00:01.464 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.464 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.464 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.465 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:01.465 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.465 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.465 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.465 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.465 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.465 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.465 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.465 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.466 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-11-16T17:00:01.466 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-11-16T17:00:01.466 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.466 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.466 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.466 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T17:00:01.466 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.466 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.467 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.467 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.467 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.467 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.467 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.467 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.467 INFO:teuthology.orchestra.run.smithi002.stdout: "cross_origin_url": { 2023-11-16T17:00:01.467 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cross_origin_url", 2023-11-16T17:00:01.468 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.468 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.468 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.468 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.468 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.468 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.468 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.468 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.468 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.469 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.469 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.469 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.469 INFO:teuthology.orchestra.run.smithi002.stdout: "crt_file": { 2023-11-16T17:00:01.469 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crt_file", 2023-11-16T17:00:01.469 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.469 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.469 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.470 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.470 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.470 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.470 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.470 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.470 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.470 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.470 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.471 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.471 INFO:teuthology.orchestra.run.smithi002.stdout: "debug": { 2023-11-16T17:00:01.471 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "debug", 2023-11-16T17:00:01.471 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.471 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.471 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.471 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.471 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.471 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.472 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.472 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable/disable debug options", 2023-11-16T17:00:01.472 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.472 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.472 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.472 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.472 INFO:teuthology.orchestra.run.smithi002.stdout: "jwt_token_ttl": { 2023-11-16T17:00:01.472 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "jwt_token_ttl", 2023-11-16T17:00:01.472 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.473 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.473 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.473 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "28800", 2023-11-16T17:00:01.473 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.473 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.473 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.473 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.473 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.474 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.474 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.474 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.474 INFO:teuthology.orchestra.run.smithi002.stdout: "key_file": { 2023-11-16T17:00:01.474 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "key_file", 2023-11-16T17:00:01.474 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.474 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.474 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.475 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.475 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.475 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.475 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.475 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.475 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.475 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.475 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.475 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.476 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.476 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.476 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.476 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.476 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.476 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.476 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.476 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.476 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.477 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.477 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.477 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.477 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.477 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.477 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.477 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.477 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.478 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.478 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.478 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.478 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.478 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.478 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.478 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.478 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.478 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.479 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.479 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.479 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.479 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.479 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.479 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.479 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.480 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.480 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.480 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.480 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.480 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.480 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.480 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.480 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.480 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.481 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.481 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.481 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.481 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.481 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.481 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.481 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.481 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.481 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.482 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.482 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.482 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.482 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.482 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.482 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.482 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.482 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.483 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.483 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.483 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.483 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.483 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.483 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.483 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.483 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.483 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.484 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.484 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.484 INFO:teuthology.orchestra.run.smithi002.stdout: "motd": { 2023-11-16T17:00:01.484 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "motd", 2023-11-16T17:00:01.484 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.484 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.484 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.484 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.484 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.485 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.485 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.485 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "The message of the day", 2023-11-16T17:00:01.485 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.485 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.485 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.485 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.485 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect_resolve_ip_addr": { 2023-11-16T17:00:01.485 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "redirect_resolve_ip_addr", 2023-11-16T17:00:01.486 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.486 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.486 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.486 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.486 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.486 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.486 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.486 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.486 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.487 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.487 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.487 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.487 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2023-11-16T17:00:01.487 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2023-11-16T17:00:01.487 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.487 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.487 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.487 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-11-16T17:00:01.488 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.488 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.488 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.488 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.488 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.488 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.488 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.488 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.488 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2023-11-16T17:00:01.489 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2023-11-16T17:00:01.489 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.489 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.489 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.489 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8080", 2023-11-16T17:00:01.489 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.489 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.489 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.490 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.490 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.490 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.490 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.490 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.490 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl": { 2023-11-16T17:00:01.490 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl", 2023-11-16T17:00:01.490 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.491 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.491 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.491 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.491 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.491 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.491 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.491 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.491 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.491 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.492 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.492 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.492 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl_server_port": { 2023-11-16T17:00:01.492 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl_server_port", 2023-11-16T17:00:01.492 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.492 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.492 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.492 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8443", 2023-11-16T17:00:01.493 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.493 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.493 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.493 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.493 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.493 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.493 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.493 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.494 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_behaviour": { 2023-11-16T17:00:01.494 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_behaviour", 2023-11-16T17:00:01.494 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.494 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.494 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.494 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "redirect", 2023-11-16T17:00:01.494 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.494 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.495 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.495 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.495 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect" 2023-11-16T17:00:01.495 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.495 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.495 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.495 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.495 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.495 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.496 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_error_status_code": { 2023-11-16T17:00:01.496 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_error_status_code", 2023-11-16T17:00:01.496 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.496 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.496 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.496 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", 2023-11-16T17:00:01.496 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-11-16T17:00:01.496 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "599", 2023-11-16T17:00:01.497 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.497 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.497 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.497 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.497 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.497 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.497 INFO:teuthology.orchestra.run.smithi002.stdout: "url_prefix": { 2023-11-16T17:00:01.497 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "url_prefix", 2023-11-16T17:00:01.497 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.498 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.498 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.498 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.498 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.498 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.498 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.498 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.498 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.499 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.499 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.499 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.499 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.499 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.499 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.499 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", 2023-11-16T17:00:01.499 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.500 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.500 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.500 INFO:teuthology.orchestra.run.smithi002.stdout: "enable_monitoring": { 2023-11-16T17:00:01.500 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enable_monitoring", 2023-11-16T17:00:01.500 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.500 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.500 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.500 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.501 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.501 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.501 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.501 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "monitor device health metrics", 2023-11-16T17:00:01.501 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.501 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.501 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.501 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.501 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.502 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.502 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.502 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.502 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.502 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.502 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.502 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.503 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.503 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.503 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.503 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.503 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.503 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.503 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.503 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.504 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.504 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.504 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.504 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.504 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.504 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.504 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.504 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.505 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.505 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.505 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.505 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.505 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.505 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.505 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.505 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.505 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.506 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.506 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.506 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.506 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.506 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.506 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.506 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.506 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.507 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.507 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.507 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.507 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.507 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.507 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.507 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.507 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.508 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.508 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.508 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.508 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.508 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.508 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.508 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.508 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.509 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.509 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.509 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.509 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.509 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.509 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.509 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.509 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.510 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.510 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.510 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.510 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.510 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.510 INFO:teuthology.orchestra.run.smithi002.stdout: "mark_out_threshold": { 2023-11-16T17:00:01.510 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mark_out_threshold", 2023-11-16T17:00:01.510 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.511 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.511 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.511 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "2419200", 2023-11-16T17:00:01.511 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.511 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.511 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.511 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-11-16T17:00:01.511 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.512 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.512 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.512 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.512 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": { 2023-11-16T17:00:01.512 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pool_name", 2023-11-16T17:00:01.512 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.512 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.512 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.512 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "device_health_metrics", 2023-11-16T17:00:01.513 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.513 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.513 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.513 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "name of pool in which to store device health metrics", 2023-11-16T17:00:01.513 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.513 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.513 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.513 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.513 INFO:teuthology.orchestra.run.smithi002.stdout: "retention_period": { 2023-11-16T17:00:01.514 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retention_period", 2023-11-16T17:00:01.514 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.514 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.514 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.514 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15552000", 2023-11-16T17:00:01.514 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.514 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.514 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.515 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long to retain device health metrics", 2023-11-16T17:00:01.515 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.515 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.515 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.515 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.515 INFO:teuthology.orchestra.run.smithi002.stdout: "scrape_frequency": { 2023-11-16T17:00:01.515 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "scrape_frequency", 2023-11-16T17:00:01.515 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.515 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.516 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.516 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "86400", 2023-11-16T17:00:01.516 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.516 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.516 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.516 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to scrape device health metrics", 2023-11-16T17:00:01.516 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.516 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.517 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.517 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.517 INFO:teuthology.orchestra.run.smithi002.stdout: "self_heal": { 2023-11-16T17:00:01.517 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "self_heal", 2023-11-16T17:00:01.517 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.517 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.517 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.517 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.517 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.518 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.518 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.518 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-11-16T17:00:01.518 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.518 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.518 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.518 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.518 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T17:00:01.519 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T17:00:01.519 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.519 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.519 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.519 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T17:00:01.519 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.519 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.519 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.519 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to wake up and check device health", 2023-11-16T17:00:01.520 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.520 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.520 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.520 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.520 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_threshold": { 2023-11-16T17:00:01.520 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_threshold", 2023-11-16T17:00:01.520 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.520 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.521 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.521 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "7257600", 2023-11-16T17:00:01.521 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.521 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.521 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.521 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-11-16T17:00:01.521 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.521 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.522 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.522 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.522 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.522 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.522 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.522 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "diskprediction_local", 2023-11-16T17:00:01.522 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.523 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.523 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.523 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.523 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.523 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.523 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.523 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.523 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.524 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.524 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.524 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.524 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.524 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.524 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.524 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.524 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.525 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.525 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.525 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.525 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.525 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.525 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.525 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.525 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.526 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.526 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.526 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.526 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.526 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.526 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.526 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.527 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.527 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.527 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.527 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.527 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.527 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.527 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.527 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.527 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.528 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.528 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.528 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.528 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.528 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.528 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.528 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.528 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.529 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.529 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.529 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.529 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.529 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.529 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.529 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.529 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.530 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.530 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.530 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.530 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.530 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.530 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.530 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.531 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.531 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.531 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.531 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.531 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.531 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.531 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.531 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.531 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.532 INFO:teuthology.orchestra.run.smithi002.stdout: "predict_interval": { 2023-11-16T17:00:01.532 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "predict_interval", 2023-11-16T17:00:01.532 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.532 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.532 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.532 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "86400", 2023-11-16T17:00:01.532 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.532 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.532 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.533 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.533 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.533 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.533 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.533 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.533 INFO:teuthology.orchestra.run.smithi002.stdout: "predictor_model": { 2023-11-16T17:00:01.533 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "predictor_model", 2023-11-16T17:00:01.533 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.533 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.534 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.534 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "prophetstor", 2023-11-16T17:00:01.534 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.534 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.534 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.534 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.534 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.534 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.535 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.535 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.535 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T17:00:01.535 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T17:00:01.535 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.535 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.535 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.535 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T17:00:01.535 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.536 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.536 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.536 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.536 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.536 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.536 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.536 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.536 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.536 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.537 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.537 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "influx", 2023-11-16T17:00:01.537 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": false, 2023-11-16T17:00:01.537 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "influxdb python module not found", 2023-11-16T17:00:01.537 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.537 INFO:teuthology.orchestra.run.smithi002.stdout: "batch_size": { 2023-11-16T17:00:01.537 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "batch_size", 2023-11-16T17:00:01.537 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.537 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.538 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.538 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5000", 2023-11-16T17:00:01.538 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.538 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.538 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.538 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-11-16T17:00:01.538 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.538 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.538 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.539 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.539 INFO:teuthology.orchestra.run.smithi002.stdout: "database": { 2023-11-16T17:00:01.539 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "database", 2023-11-16T17:00:01.539 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.539 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.539 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.539 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "ceph", 2023-11-16T17:00:01.539 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.540 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.540 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.540 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-11-16T17:00:01.540 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.540 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.540 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.540 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.540 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": { 2023-11-16T17:00:01.541 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "hostname", 2023-11-16T17:00:01.541 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.541 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.541 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.541 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.541 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.541 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.541 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.541 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB server hostname", 2023-11-16T17:00:01.542 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.542 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.542 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.542 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.542 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T17:00:01.542 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T17:00:01.542 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.542 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.542 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.543 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "30", 2023-11-16T17:00:01.543 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "5", 2023-11-16T17:00:01.543 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.543 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.543 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-11-16T17:00:01.543 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.543 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.543 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.543 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.544 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.544 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.544 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.544 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.544 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.544 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.544 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.544 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.544 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.545 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.545 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.545 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.545 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.545 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.545 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.545 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.545 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.546 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.546 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.546 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.546 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.546 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.546 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.546 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.546 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.547 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.547 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.547 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.547 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.547 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.547 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.547 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.547 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.547 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.548 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.548 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.548 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.548 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.548 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.548 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.548 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.548 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.548 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.549 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.549 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.549 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.549 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.549 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.549 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.549 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.549 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.550 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.550 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.550 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.550 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.550 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.550 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.550 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.550 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.551 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.551 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.551 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.551 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.551 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.551 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.551 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.551 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.551 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.552 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.552 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.552 INFO:teuthology.orchestra.run.smithi002.stdout: "password": { 2023-11-16T17:00:01.552 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "password", 2023-11-16T17:00:01.552 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.552 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.552 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.552 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.552 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.553 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.553 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.553 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "password of InfluxDB server user", 2023-11-16T17:00:01.553 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.553 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.553 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.553 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.553 INFO:teuthology.orchestra.run.smithi002.stdout: "port": { 2023-11-16T17:00:01.553 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "port", 2023-11-16T17:00:01.554 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.554 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.554 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.554 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8086", 2023-11-16T17:00:01.554 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.554 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.554 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.554 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB server port", 2023-11-16T17:00:01.555 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.555 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.555 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.555 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.555 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl": { 2023-11-16T17:00:01.555 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl", 2023-11-16T17:00:01.555 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.555 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.555 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.556 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "false", 2023-11-16T17:00:01.556 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.556 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.556 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.556 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-11-16T17:00:01.556 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.556 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.556 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.556 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.557 INFO:teuthology.orchestra.run.smithi002.stdout: "threads": { 2023-11-16T17:00:01.557 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "threads", 2023-11-16T17:00:01.557 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.557 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.557 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.557 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T17:00:01.557 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T17:00:01.557 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "32", 2023-11-16T17:00:01.557 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.558 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-11-16T17:00:01.558 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.558 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.558 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.558 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.558 INFO:teuthology.orchestra.run.smithi002.stdout: "username": { 2023-11-16T17:00:01.558 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "username", 2023-11-16T17:00:01.558 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.559 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.559 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.559 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.559 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.559 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.559 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.559 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "username of InfluxDB server user", 2023-11-16T17:00:01.560 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.560 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.560 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.560 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.560 INFO:teuthology.orchestra.run.smithi002.stdout: "verify_ssl": { 2023-11-16T17:00:01.560 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "verify_ssl", 2023-11-16T17:00:01.561 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.561 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.561 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.561 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "true", 2023-11-16T17:00:01.561 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.562 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.562 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.562 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-11-16T17:00:01.562 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.562 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.562 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.563 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.563 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.563 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.563 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.563 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "insights", 2023-11-16T17:00:01.563 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.563 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.564 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.564 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.564 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.564 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.564 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.564 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.564 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.564 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.564 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.565 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.565 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.565 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.565 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.565 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.565 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.565 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.565 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.566 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.566 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.566 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.566 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.566 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.566 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.566 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.566 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.566 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.567 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.567 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.567 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.567 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.567 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.567 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.567 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.567 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.567 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.568 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.568 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.568 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.568 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.568 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.568 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.568 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.568 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.569 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.569 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.569 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.569 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.569 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.569 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.569 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.569 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.570 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.570 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.570 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.570 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.570 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.570 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.570 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.570 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.571 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.571 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.571 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.571 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.571 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.571 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.571 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.571 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.571 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.572 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.572 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.572 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.572 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.572 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.572 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.572 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "iostat", 2023-11-16T17:00:01.572 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.572 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.573 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.573 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.573 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.573 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.573 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.573 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.573 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.573 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.573 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.574 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.574 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.574 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.574 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.574 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.574 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.574 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.574 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.575 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.575 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.575 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.575 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.575 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.575 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.575 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.575 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.575 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.576 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.576 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.576 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.576 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.576 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.576 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.576 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.576 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.576 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.577 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.577 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.577 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.577 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.577 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.577 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.577 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.577 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.577 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.578 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.578 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.578 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.578 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.578 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.578 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.578 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.578 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.578 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.579 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.579 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.579 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.579 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.579 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.579 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.579 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.579 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.580 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.580 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.580 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.580 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.580 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.580 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.580 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.580 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.580 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.581 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.581 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.581 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.581 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.581 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "k8sevents", 2023-11-16T17:00:01.581 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.581 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.581 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.581 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_event_retention_days": { 2023-11-16T17:00:01.582 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ceph_event_retention_days", 2023-11-16T17:00:01.582 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.582 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.582 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.582 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "7", 2023-11-16T17:00:01.582 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.582 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.582 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.583 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Days to hold ceph event information within local cache", 2023-11-16T17:00:01.583 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.583 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.583 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.583 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.583 INFO:teuthology.orchestra.run.smithi002.stdout: "config_check_secs": { 2023-11-16T17:00:01.583 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_check_secs", 2023-11-16T17:00:01.583 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.583 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.584 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.584 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:01.584 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "10", 2023-11-16T17:00:01.584 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.584 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.584 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-11-16T17:00:01.584 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.584 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.584 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.585 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.585 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.585 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.585 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.585 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.585 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.585 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.585 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.585 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.586 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.586 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.586 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.586 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.586 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.586 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.586 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.586 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.587 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.587 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.587 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.587 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.587 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.587 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.587 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.587 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.587 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.588 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.588 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.588 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.588 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.588 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.588 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.588 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.588 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.588 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.589 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.589 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.589 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.589 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.589 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.589 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.589 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.589 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.589 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.590 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.590 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.590 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.590 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.590 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.590 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.590 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.591 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.591 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.591 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.591 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.591 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.591 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.591 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.591 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.591 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.592 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.592 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.592 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.592 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.592 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.592 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.592 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.592 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.592 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.593 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.593 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.593 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.593 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.593 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.593 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "localpool", 2023-11-16T17:00:01.593 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.593 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.593 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.594 INFO:teuthology.orchestra.run.smithi002.stdout: "failure_domain": { 2023-11-16T17:00:01.594 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "failure_domain", 2023-11-16T17:00:01.594 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.594 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.594 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.594 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "host", 2023-11-16T17:00:01.594 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.594 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.595 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.595 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "failure domain for any created local pool", 2023-11-16T17:00:01.595 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-11-16T17:00:01.595 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.595 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.595 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.595 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.595 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.595 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.596 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.596 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.596 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.596 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.596 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.596 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.596 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.596 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.596 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.597 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.597 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.597 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.597 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.597 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.597 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.597 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.597 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.597 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.598 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.598 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.598 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.598 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.598 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.598 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.598 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.598 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.599 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.599 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.599 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.599 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.599 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.599 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.599 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.599 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.599 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.600 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.600 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.600 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.600 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.600 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.600 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.600 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.600 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.601 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.601 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.601 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.601 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.601 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.601 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.601 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.601 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.602 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.602 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.602 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.602 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.602 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.602 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.602 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.602 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.602 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.603 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.603 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.603 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.603 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.603 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.603 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.603 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.603 INFO:teuthology.orchestra.run.smithi002.stdout: "min_size": { 2023-11-16T17:00:01.603 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "min_size", 2023-11-16T17:00:01.604 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.604 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.604 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.604 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.604 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.604 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.604 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.604 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default min_size for any created local pool", 2023-11-16T17:00:01.604 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-11-16T17:00:01.605 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.605 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.605 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.605 INFO:teuthology.orchestra.run.smithi002.stdout: "num_rep": { 2023-11-16T17:00:01.605 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "num_rep", 2023-11-16T17:00:01.605 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.605 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.605 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.606 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3", 2023-11-16T17:00:01.606 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.606 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.606 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.606 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default replica count for any created local pool", 2023-11-16T17:00:01.606 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.606 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.606 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.607 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.607 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num": { 2023-11-16T17:00:01.607 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pg_num", 2023-11-16T17:00:01.607 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.607 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.607 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.607 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "128", 2023-11-16T17:00:01.607 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.607 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.608 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.608 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default pg_num for any created local pool", 2023-11-16T17:00:01.608 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.608 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.608 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.608 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.608 INFO:teuthology.orchestra.run.smithi002.stdout: "prefix": { 2023-11-16T17:00:01.608 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prefix", 2023-11-16T17:00:01.608 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.609 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.609 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.609 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.609 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.609 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.609 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.609 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "name prefix for any created local pool", 2023-11-16T17:00:01.609 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.610 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.610 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.610 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.610 INFO:teuthology.orchestra.run.smithi002.stdout: "subtree": { 2023-11-16T17:00:01.610 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subtree", 2023-11-16T17:00:01.610 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.610 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.610 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.610 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "rack", 2023-11-16T17:00:01.611 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.611 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.611 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.611 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "CRUSH level for which to create a local pool", 2023-11-16T17:00:01.611 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-11-16T17:00:01.611 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.611 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.611 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.611 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.611 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.612 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.612 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mds_autoscaler", 2023-11-16T17:00:01.612 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.612 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.612 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.612 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.612 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.612 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.612 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.613 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.613 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.613 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.613 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.613 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.613 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.613 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.613 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.613 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.613 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.614 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.614 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.614 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.614 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.614 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.614 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.614 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.614 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.614 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.615 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.615 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.615 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.615 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.615 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.615 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.615 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.615 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.615 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.616 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.616 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.616 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.616 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.616 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.616 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.616 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.616 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.616 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.616 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.617 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.617 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.617 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.617 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.617 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.617 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.617 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.617 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.617 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.618 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.618 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.618 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.618 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.618 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.618 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.618 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.618 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.618 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.619 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.619 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.619 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.619 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.619 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.619 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.619 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.619 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.619 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.620 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.620 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.620 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.620 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.620 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirroring", 2023-11-16T17:00:01.620 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.620 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.620 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.620 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.620 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.621 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.621 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.621 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.621 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.621 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.621 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.621 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.621 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.621 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.622 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.622 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.622 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.622 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.622 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.622 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.622 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.622 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.622 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.622 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.623 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.623 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.623 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.623 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.623 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.623 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.623 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.623 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.623 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.624 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.624 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.624 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.624 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.624 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.624 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.624 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.624 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.624 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.624 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.625 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.625 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.625 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.625 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.625 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.625 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.625 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.625 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.625 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.626 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.626 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.626 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.626 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.626 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.626 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.626 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.626 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.626 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.627 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.627 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.627 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.627 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.627 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.627 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.627 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.627 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.628 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.628 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.628 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.628 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.628 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.628 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.628 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.628 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "nfs", 2023-11-16T17:00:01.628 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.629 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.629 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.629 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.629 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.629 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.629 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.629 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.629 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.630 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.630 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.630 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.630 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.630 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.630 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.630 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.630 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.630 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.631 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.631 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.631 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.631 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.631 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.631 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.631 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.631 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.631 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.632 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.632 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.632 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.632 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.632 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.632 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.632 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.632 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.632 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.633 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.633 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.633 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.633 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.633 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.633 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.633 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.633 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.633 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.634 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.634 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.634 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.634 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.634 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.634 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.634 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.634 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.634 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.635 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.635 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.635 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.635 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.635 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.635 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.635 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.635 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.635 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.635 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.636 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.636 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.636 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.636 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.636 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.636 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.636 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.636 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.636 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.637 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.637 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.637 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.637 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "orchestrator", 2023-11-16T17:00:01.637 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.637 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.637 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.637 INFO:teuthology.orchestra.run.smithi002.stdout: "fail_fs": { 2023-11-16T17:00:01.637 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "fail_fs", 2023-11-16T17:00:01.637 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.638 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.638 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.638 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.638 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.638 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.638 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.638 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-11-16T17:00:01.638 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.638 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.639 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.639 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.639 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.639 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.639 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.639 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.639 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.639 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.639 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.640 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.640 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.640 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.640 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.640 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.640 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.640 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.640 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.640 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.641 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.641 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.641 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.641 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.641 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.641 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.641 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.641 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.641 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.642 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.642 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.642 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.642 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.642 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.642 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.642 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.642 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.642 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.643 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.643 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.643 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.643 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.643 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.643 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.643 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.643 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.643 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.643 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.644 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.644 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.644 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.644 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.644 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.644 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.644 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.644 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.644 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.645 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.645 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.645 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.645 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.645 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.645 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.645 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.645 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.645 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.646 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.646 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.646 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.646 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.646 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.646 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.646 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.646 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.646 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator": { 2023-11-16T17:00:01.647 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "orchestrator", 2023-11-16T17:00:01.647 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.647 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.647 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.647 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.647 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.647 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.647 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.647 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm", 2023-11-16T17:00:01.648 INFO:teuthology.orchestra.run.smithi002.stdout: "rook", 2023-11-16T17:00:01.648 INFO:teuthology.orchestra.run.smithi002.stdout: "test_orchestrator" 2023-11-16T17:00:01.648 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.648 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Orchestrator backend", 2023-11-16T17:00:01.648 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.648 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.648 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.648 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.648 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.649 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.649 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.649 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "osd_perf_query", 2023-11-16T17:00:01.649 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.649 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.649 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.649 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.649 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.650 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.650 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.650 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.650 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.650 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.650 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.650 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.650 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.650 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.651 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.651 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.651 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.651 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.651 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.651 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.651 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.651 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.651 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.652 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.652 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.652 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.652 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.652 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.652 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.652 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.652 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.652 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.652 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.653 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.653 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.653 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.653 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.653 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.653 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.653 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.653 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.653 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.653 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.654 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.654 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.654 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.654 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.654 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.654 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.654 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.654 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.654 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.655 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.655 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.655 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.655 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.655 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.655 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.655 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.655 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.655 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.656 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.656 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.656 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.656 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.656 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.656 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.656 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.656 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.656 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.656 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.657 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.657 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.657 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.657 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.657 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.657 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "osd_support", 2023-11-16T17:00:01.657 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.657 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.657 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.658 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.658 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.658 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.658 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.658 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.658 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.658 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.658 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.658 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.659 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.659 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.659 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.659 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.659 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.659 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.659 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.659 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.660 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.660 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.660 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.660 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.660 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.660 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.660 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.660 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.660 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.660 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.661 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.661 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.661 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.661 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.661 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.661 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.661 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.661 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.661 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.662 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.662 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.662 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.662 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.662 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.662 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.662 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.662 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.662 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.662 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.663 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.663 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.663 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.663 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.663 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.663 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.663 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.663 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.663 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.664 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.664 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.664 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.664 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.664 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.664 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.664 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.664 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.664 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.664 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.665 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.665 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.665 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.665 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.665 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.665 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.665 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.665 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.665 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pg_autoscaler", 2023-11-16T17:00:01.666 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.666 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.666 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.666 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.666 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.666 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.666 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.666 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.666 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.667 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.667 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.667 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.667 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.667 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.667 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.667 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.667 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.667 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.667 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.668 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.668 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.668 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.668 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.668 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.668 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.668 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.668 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.668 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.669 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.669 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.669 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.669 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.669 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.669 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.669 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.669 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.669 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.670 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.670 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.670 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.670 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.670 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.670 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.670 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.670 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.670 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.671 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.671 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.671 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.671 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.671 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.671 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.671 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.671 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.671 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.672 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.672 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.672 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.672 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.672 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.672 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.672 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.672 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.672 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.673 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.673 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.673 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.673 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.673 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.673 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.673 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.673 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.673 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.673 INFO:teuthology.orchestra.run.smithi002.stdout: "noautoscale": { 2023-11-16T17:00:01.674 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "noautoscale", 2023-11-16T17:00:01.674 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.674 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.674 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.674 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.674 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.674 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.674 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.675 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "global autoscale flag", 2023-11-16T17:00:01.675 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-11-16T17:00:01.675 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.675 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.675 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.675 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T17:00:01.675 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T17:00:01.675 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.675 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.676 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.676 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T17:00:01.676 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.676 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.676 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.676 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.676 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.676 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.676 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.677 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.677 INFO:teuthology.orchestra.run.smithi002.stdout: "threshold": { 2023-11-16T17:00:01.677 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "threshold", 2023-11-16T17:00:01.677 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:01.677 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.677 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.677 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3.0", 2023-11-16T17:00:01.677 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1.0", 2023-11-16T17:00:01.677 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.678 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.678 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "scaling threshold", 2023-11-16T17:00:01.678 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-11-16T17:00:01.678 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.678 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.678 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.678 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.678 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.679 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.679 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "progress", 2023-11-16T17:00:01.679 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.679 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.679 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.679 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_pg_recovery_event": { 2023-11-16T17:00:01.679 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_pg_recovery_event", 2023-11-16T17:00:01.679 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.679 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.680 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.680 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.680 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.680 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.680 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.680 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow the module to show pg recovery progress", 2023-11-16T17:00:01.680 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.680 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.680 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.681 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.681 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": { 2023-11-16T17:00:01.681 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enabled", 2023-11-16T17:00:01.681 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.681 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.681 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.681 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.681 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.681 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.682 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.682 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.682 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.682 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.682 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.682 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.682 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.682 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.682 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.683 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.683 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.683 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.683 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.683 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.683 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.683 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.683 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.683 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.683 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.684 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.684 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.684 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.684 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.684 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.684 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.684 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.684 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.684 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.685 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.685 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.685 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.685 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.685 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.685 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.685 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.685 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.685 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.685 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.686 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.686 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.686 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.686 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.686 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.686 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.686 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.686 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.687 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.687 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.687 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.687 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.687 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.687 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.687 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.687 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.687 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.687 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.688 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.688 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.688 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.688 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.688 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.688 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.688 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.688 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.688 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.688 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.689 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.689 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.689 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.689 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.689 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.689 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.689 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.689 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.690 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.690 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.690 INFO:teuthology.orchestra.run.smithi002.stdout: "max_completed_events": { 2023-11-16T17:00:01.690 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_completed_events", 2023-11-16T17:00:01.690 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.690 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.690 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.690 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "50", 2023-11-16T17:00:01.690 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.691 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.691 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.691 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "number of past completed events to remember", 2023-11-16T17:00:01.691 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.691 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.691 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.691 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.691 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T17:00:01.691 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T17:00:01.692 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.692 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.692 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.692 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T17:00:01.692 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.692 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.692 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.692 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long the module is going to sleep", 2023-11-16T17:00:01.692 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.693 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.693 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.693 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.693 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.693 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.693 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.693 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus", 2023-11-16T17:00:01.693 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.693 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.694 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.694 INFO:teuthology.orchestra.run.smithi002.stdout: "cache": { 2023-11-16T17:00:01.694 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cache", 2023-11-16T17:00:01.694 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.694 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.694 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.694 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.694 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.694 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.695 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.695 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.695 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.695 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.695 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.695 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.695 INFO:teuthology.orchestra.run.smithi002.stdout: "exclude_perf_counters": { 2023-11-16T17:00:01.695 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "exclude_perf_counters", 2023-11-16T17:00:01.695 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.696 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.696 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.696 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.696 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.696 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.696 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.696 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-11-16T17:00:01.696 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-11-16T17:00:01.696 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.697 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.697 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.697 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.697 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.697 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.697 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.697 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.697 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.697 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.698 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.698 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.698 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.698 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.698 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.698 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.698 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.698 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.698 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.699 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.699 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.699 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.699 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.699 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.699 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.699 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.699 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.699 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.700 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.700 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.700 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.700 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.700 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.700 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.700 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.700 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.700 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.700 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.701 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.701 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.701 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.701 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.701 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.701 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.701 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.701 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.701 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.702 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.702 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.702 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.702 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.702 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.702 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.702 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.702 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.702 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.702 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.703 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.703 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.703 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.703 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.703 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.703 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.703 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.703 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.703 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.704 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.704 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.704 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.704 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.704 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.704 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.704 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.704 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_stats_pools": { 2023-11-16T17:00:01.704 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_stats_pools", 2023-11-16T17:00:01.704 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.705 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.705 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.705 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.705 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.705 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.705 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.705 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.705 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.705 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.706 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.706 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.706 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_stats_pools_refresh_interval": { 2023-11-16T17:00:01.706 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-11-16T17:00:01.706 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.706 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.706 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.706 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "300", 2023-11-16T17:00:01.706 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.706 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.707 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.707 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.707 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.707 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.707 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.707 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.707 INFO:teuthology.orchestra.run.smithi002.stdout: "scrape_interval": { 2023-11-16T17:00:01.707 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "scrape_interval", 2023-11-16T17:00:01.707 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:01.708 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.708 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.708 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15.0", 2023-11-16T17:00:01.708 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.708 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.708 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.708 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.708 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.708 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.708 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.709 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.709 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2023-11-16T17:00:01.709 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2023-11-16T17:00:01.709 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.709 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.709 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.709 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-11-16T17:00:01.709 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.709 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.710 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.710 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-11-16T17:00:01.710 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.710 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.710 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.710 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.710 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2023-11-16T17:00:01.710 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2023-11-16T17:00:01.710 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.711 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.711 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.711 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "9283", 2023-11-16T17:00:01.711 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.711 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.711 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.711 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-11-16T17:00:01.711 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.711 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.711 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.712 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.712 INFO:teuthology.orchestra.run.smithi002.stdout: "stale_cache_strategy": { 2023-11-16T17:00:01.712 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "stale_cache_strategy", 2023-11-16T17:00:01.712 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.712 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.712 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.712 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "log", 2023-11-16T17:00:01.712 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.712 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.713 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.713 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.713 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.713 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.713 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.713 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.713 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_behaviour": { 2023-11-16T17:00:01.713 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_behaviour", 2023-11-16T17:00:01.713 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.713 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "default", 2023-11-16T17:00:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: "default", 2023-11-16T17:00:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: "error" 2023-11-16T17:00:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.715 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.715 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.715 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.715 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.715 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_error_status_code": { 2023-11-16T17:00:01.715 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_error_status_code", 2023-11-16T17:00:01.715 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.715 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.715 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.715 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", 2023-11-16T17:00:01.716 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-11-16T17:00:01.716 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "599", 2023-11-16T17:00:01.716 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.716 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.716 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.716 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.716 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.716 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.716 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.717 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.717 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.717 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_support", 2023-11-16T17:00:01.717 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.717 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.717 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.717 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.717 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.717 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.717 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.718 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.718 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.718 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.718 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.718 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.718 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.718 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.718 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.718 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.719 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.719 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.719 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.719 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.719 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.719 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.719 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.719 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.719 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.720 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.720 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.720 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.720 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.720 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.720 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.720 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.720 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.720 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.721 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.721 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.721 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.721 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.721 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.721 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.721 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.721 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.721 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.721 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.722 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.722 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.722 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.722 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.722 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.722 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.722 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.722 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.723 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.723 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.723 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.723 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.723 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.723 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.723 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.723 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.723 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.724 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.724 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.724 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.724 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.724 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.724 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.724 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.724 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.724 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.724 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.725 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.725 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.725 INFO:teuthology.orchestra.run.smithi002.stdout: "max_concurrent_snap_create": { 2023-11-16T17:00:01.725 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_concurrent_snap_create", 2023-11-16T17:00:01.725 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.725 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.725 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.725 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:01.725 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.726 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.726 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.726 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.726 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.726 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.726 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.726 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.726 INFO:teuthology.orchestra.run.smithi002.stdout: "mirror_snapshot_schedule": { 2023-11-16T17:00:01.726 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirror_snapshot_schedule", 2023-11-16T17:00:01.727 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.727 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.727 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.727 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.727 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.727 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.727 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.727 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.727 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.728 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.728 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.728 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.728 INFO:teuthology.orchestra.run.smithi002.stdout: "trash_purge_schedule": { 2023-11-16T17:00:01.728 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "trash_purge_schedule", 2023-11-16T17:00:01.728 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.728 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.728 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.728 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.729 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.729 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.729 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.729 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.729 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.729 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.729 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.729 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.730 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.730 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.730 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.730 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "restful", 2023-11-16T17:00:01.730 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.730 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.730 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.730 INFO:teuthology.orchestra.run.smithi002.stdout: "enable_auth": { 2023-11-16T17:00:01.731 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enable_auth", 2023-11-16T17:00:01.731 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.731 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.731 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.731 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.731 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.731 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.731 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.731 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.732 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.732 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.732 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.732 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.732 INFO:teuthology.orchestra.run.smithi002.stdout: "key_file": { 2023-11-16T17:00:01.732 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "key_file", 2023-11-16T17:00:01.732 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.732 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.732 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.733 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.733 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.733 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.733 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.733 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.733 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.733 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.733 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.733 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.734 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.734 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.734 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.734 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.734 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.734 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.734 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.734 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.734 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.735 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.735 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.735 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.735 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.735 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.735 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.735 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.735 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.736 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.736 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.736 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.736 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.736 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.736 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.736 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.736 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.736 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.737 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.737 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.737 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.737 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.737 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.737 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.737 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.737 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.737 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.738 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.738 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.738 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.738 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.738 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.738 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.739 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.739 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.739 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.739 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.739 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.739 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.739 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.739 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.739 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.740 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.740 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.740 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.740 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.740 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.740 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.740 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.740 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.741 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.741 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.741 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.741 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.741 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.741 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.741 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.741 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.741 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.742 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.742 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.742 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.742 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2023-11-16T17:00:01.742 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2023-11-16T17:00:01.742 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.742 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.742 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.742 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.743 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.743 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.743 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.743 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.743 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.743 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.743 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.743 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.744 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2023-11-16T17:00:01.744 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2023-11-16T17:00:01.744 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.744 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.744 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.744 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.744 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.744 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.744 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.744 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.745 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.745 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.745 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.745 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.745 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.745 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.745 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.745 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rgw", 2023-11-16T17:00:01.746 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.746 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.746 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.746 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.746 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.746 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.746 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.746 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.746 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.747 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.747 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.747 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.747 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.747 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.747 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.747 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.747 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.747 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.748 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.748 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.748 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.748 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.748 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.748 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.748 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.748 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.748 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.749 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.749 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.749 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.749 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.749 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.749 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.749 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.749 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.750 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.750 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.750 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.750 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.750 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.750 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.750 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.750 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.751 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.751 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.751 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.751 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.751 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.751 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.751 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.751 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.751 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.751 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.752 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.752 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.752 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.752 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.752 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.752 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.752 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.752 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.752 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.753 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.753 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.753 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.753 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.753 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.753 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.753 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.753 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.753 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.754 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.754 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.754 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.754 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.754 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.754 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rook", 2023-11-16T17:00:01.754 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.754 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.754 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.755 INFO:teuthology.orchestra.run.smithi002.stdout: "drive_group_interval": { 2023-11-16T17:00:01.755 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "drive_group_interval", 2023-11-16T17:00:01.755 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:01.755 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.755 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.755 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "300.0", 2023-11-16T17:00:01.755 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.755 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.756 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.756 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-11-16T17:00:01.756 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.756 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.756 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.756 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.756 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.756 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.756 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.757 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.757 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.757 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.757 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.757 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.757 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.757 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.757 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.757 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.758 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.758 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.758 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.758 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.758 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.758 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.758 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.758 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.759 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.759 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.759 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.759 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.759 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.759 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.759 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.759 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.759 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.760 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.760 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.760 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.760 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.760 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.760 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.760 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.760 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.760 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.761 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.761 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.761 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.761 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.761 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.761 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.761 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.761 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.762 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.762 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.762 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.762 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.762 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.762 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.762 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.762 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.762 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.763 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.763 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.763 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.763 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.763 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.763 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.763 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.763 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.763 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.764 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.764 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.764 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.764 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.764 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.764 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.764 INFO:teuthology.orchestra.run.smithi002.stdout: "storage_class": { 2023-11-16T17:00:01.764 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "storage_class", 2023-11-16T17:00:01.765 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.765 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.765 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.765 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "local", 2023-11-16T17:00:01.765 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.765 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.765 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.765 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-11-16T17:00:01.765 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.766 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.766 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.766 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.766 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.766 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.766 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.766 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "selftest", 2023-11-16T17:00:01.766 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.766 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.767 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.767 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.767 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.767 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.767 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.767 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.767 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.767 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.767 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.768 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.768 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.768 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.768 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.768 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.768 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.768 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.768 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.768 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.769 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.769 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.769 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.769 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.769 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.769 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.769 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.769 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.769 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.770 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.770 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.770 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.770 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.770 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.770 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.770 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.770 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.771 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.771 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.771 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.771 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.771 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.771 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.771 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.771 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.771 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.771 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.772 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.772 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.772 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.772 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.772 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.772 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.772 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.772 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.772 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.773 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.773 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.773 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.773 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.773 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.773 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.773 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.773 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.773 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.774 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.774 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.774 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.774 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.774 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.774 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.774 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.774 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.775 INFO:teuthology.orchestra.run.smithi002.stdout: "roption1": { 2023-11-16T17:00:01.775 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "roption1", 2023-11-16T17:00:01.775 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.775 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.775 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.775 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.775 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.775 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.775 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.775 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.776 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.776 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.776 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.776 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.776 INFO:teuthology.orchestra.run.smithi002.stdout: "roption2": { 2023-11-16T17:00:01.776 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "roption2", 2023-11-16T17:00:01.776 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.776 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.776 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.777 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "xyz", 2023-11-16T17:00:01.777 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.777 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.777 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.777 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.777 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.777 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.777 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.777 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.777 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption1": { 2023-11-16T17:00:01.778 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption1", 2023-11-16T17:00:01.778 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.778 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.778 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.778 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.778 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.778 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.778 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.778 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.779 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.779 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.779 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.779 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.779 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption2": { 2023-11-16T17:00:01.779 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption2", 2023-11-16T17:00:01.779 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.779 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.779 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.780 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.780 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.780 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.780 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.780 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.780 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.780 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.780 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.780 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.781 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption3": { 2023-11-16T17:00:01.781 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption3", 2023-11-16T17:00:01.781 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:01.781 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.781 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.781 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.781 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.781 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.781 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.781 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.782 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.782 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.782 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.782 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.782 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption4": { 2023-11-16T17:00:01.782 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption4", 2023-11-16T17:00:01.782 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.782 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.782 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.783 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.783 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.783 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.783 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.783 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.783 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.783 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.783 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.783 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.783 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption5": { 2023-11-16T17:00:01.784 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption5", 2023-11-16T17:00:01.784 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.784 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.784 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.784 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.784 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.784 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.784 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.784 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.785 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.785 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.785 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.785 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.785 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption6": { 2023-11-16T17:00:01.785 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption6", 2023-11-16T17:00:01.785 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.785 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.785 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.786 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.786 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.786 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.786 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.786 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.786 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.786 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.786 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.786 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.787 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption7": { 2023-11-16T17:00:01.787 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption7", 2023-11-16T17:00:01.787 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.787 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.787 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.787 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.787 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T17:00:01.787 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "42", 2023-11-16T17:00:01.787 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.788 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.788 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.788 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.788 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.788 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.788 INFO:teuthology.orchestra.run.smithi002.stdout: "testkey": { 2023-11-16T17:00:01.788 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testkey", 2023-11-16T17:00:01.788 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.788 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.789 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.789 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.789 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.789 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.789 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.789 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.789 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.789 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.790 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.790 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.790 INFO:teuthology.orchestra.run.smithi002.stdout: "testlkey": { 2023-11-16T17:00:01.790 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testlkey", 2023-11-16T17:00:01.790 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.790 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.790 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.791 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.791 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.791 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.791 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.791 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.791 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.791 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.791 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.791 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.791 INFO:teuthology.orchestra.run.smithi002.stdout: "testnewline": { 2023-11-16T17:00:01.792 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testnewline", 2023-11-16T17:00:01.792 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.792 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.792 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.792 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.792 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.792 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.792 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.792 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.793 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.793 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.793 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.793 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.793 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.793 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.793 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.793 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snap_schedule", 2023-11-16T17:00:01.793 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.794 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.794 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.794 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_m_granularity": { 2023-11-16T17:00:01.794 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_m_granularity", 2023-11-16T17:00:01.794 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.794 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.794 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.794 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.795 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.795 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.795 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.795 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow minute scheduled snapshots", 2023-11-16T17:00:01.795 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.795 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.795 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.795 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.795 INFO:teuthology.orchestra.run.smithi002.stdout: "dump_on_update": { 2023-11-16T17:00:01.796 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "dump_on_update", 2023-11-16T17:00:01.796 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.796 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.796 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.796 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.796 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.796 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.796 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.796 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "dump database to debug log on update", 2023-11-16T17:00:01.797 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.797 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.797 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.797 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.797 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.797 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.797 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.797 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.797 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.797 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.798 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.798 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.798 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.798 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.798 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.798 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.798 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.798 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.799 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.799 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.799 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.799 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.799 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.799 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.799 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.799 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.799 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.801 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.801 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.801 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.801 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.801 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.801 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.801 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.801 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.801 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.802 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.802 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.802 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.802 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.802 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.802 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.802 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.802 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.802 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.802 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.803 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.803 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.803 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.803 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.803 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.803 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.803 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.803 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.803 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.804 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.804 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.804 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.804 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.804 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.804 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.804 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.804 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.804 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.805 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.805 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.805 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.805 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.805 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "stats", 2023-11-16T17:00:01.805 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.805 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.805 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.805 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.806 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.806 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.806 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.806 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.806 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.806 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.806 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.806 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.806 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.807 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.807 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.807 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.807 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.807 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.807 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.807 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.807 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.807 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.807 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.808 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.808 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.808 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.808 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.808 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.808 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.808 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.808 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.808 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.809 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.809 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.809 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.809 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.809 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.809 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.809 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.809 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.809 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.810 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.810 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.810 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.810 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.810 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.810 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.810 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.810 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.810 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.811 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.811 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.811 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.811 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.811 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.811 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.811 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.811 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.811 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.812 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.812 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.812 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.812 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.812 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.812 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.812 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.812 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.812 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.812 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.813 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.813 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.813 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.813 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.813 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.813 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.813 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.813 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "status", 2023-11-16T17:00:01.813 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.814 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.814 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.814 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.814 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.814 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.814 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.814 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.814 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.814 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.815 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.815 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.815 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.815 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.815 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.815 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.815 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.815 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.815 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.816 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.816 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.816 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.816 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.816 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.816 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.816 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.816 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.816 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.817 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.817 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.817 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.817 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.817 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.817 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.817 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.817 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.817 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.818 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.818 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.818 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.818 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.818 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.818 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.818 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.818 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.818 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.819 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.819 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.819 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.819 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.819 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.819 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.819 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.819 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.820 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.820 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.820 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.820 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.820 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.820 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.820 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.820 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.821 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.821 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.821 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.821 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.821 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.821 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.821 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.821 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.821 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.821 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.822 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.822 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.822 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.822 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.822 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "telegraf", 2023-11-16T17:00:01.822 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.822 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.822 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.822 INFO:teuthology.orchestra.run.smithi002.stdout: "address": { 2023-11-16T17:00:01.823 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "address", 2023-11-16T17:00:01.823 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.823 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.823 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.823 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-11-16T17:00:01.823 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.823 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.823 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.823 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.823 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.824 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.824 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.824 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.824 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T17:00:01.824 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T17:00:01.824 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.824 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.824 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.824 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15", 2023-11-16T17:00:01.825 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.825 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.825 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.825 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.825 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.825 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.832 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.832 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.832 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.832 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.832 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.832 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.833 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.833 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.833 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.833 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.833 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.833 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.833 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.833 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.833 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.834 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.834 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.834 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.834 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.834 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.834 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.834 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.834 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.835 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.835 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.835 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.835 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.835 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.835 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.835 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.835 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.835 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.836 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.836 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.836 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.836 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.836 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.836 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.836 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.836 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.836 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.837 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.837 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.837 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.837 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.837 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.837 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.837 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.837 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.837 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.838 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.838 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.838 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.838 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.838 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.838 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.838 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.838 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.838 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.839 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.839 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.839 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.839 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.839 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.839 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.839 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.839 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.839 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.840 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.840 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.840 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.840 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.840 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.840 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.840 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.840 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "telemetry", 2023-11-16T17:00:01.840 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.841 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.841 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.841 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_basic": { 2023-11-16T17:00:01.841 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_basic", 2023-11-16T17:00:01.841 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.841 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.841 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.841 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.841 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.841 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.842 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.842 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share basic cluster information (size, version)", 2023-11-16T17:00:01.842 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.842 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.842 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.842 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.842 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_crash": { 2023-11-16T17:00:01.842 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_crash", 2023-11-16T17:00:01.842 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.843 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.843 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.843 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.843 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.843 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.843 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.843 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-11-16T17:00:01.843 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.844 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.844 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.844 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.844 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_device": { 2023-11-16T17:00:01.844 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_device", 2023-11-16T17:00:01.844 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.844 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.844 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.844 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.845 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.845 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.845 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.845 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-11-16T17:00:01.845 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.845 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.845 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.845 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.846 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_ident": { 2023-11-16T17:00:01.846 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_ident", 2023-11-16T17:00:01.846 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.846 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.846 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.846 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.846 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.846 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.847 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.847 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-11-16T17:00:01.847 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.847 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.847 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.847 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.847 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_perf": { 2023-11-16T17:00:01.847 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_perf", 2023-11-16T17:00:01.847 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.848 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.848 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.848 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.848 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.848 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.848 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.848 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share various performance metrics of a cluster", 2023-11-16T17:00:01.848 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.848 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.849 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.849 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.849 INFO:teuthology.orchestra.run.smithi002.stdout: "contact": { 2023-11-16T17:00:01.849 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "contact", 2023-11-16T17:00:01.849 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.849 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.849 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.849 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.849 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.850 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.850 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.850 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.850 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.850 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.850 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.850 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.850 INFO:teuthology.orchestra.run.smithi002.stdout: "description": { 2023-11-16T17:00:01.850 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "description", 2023-11-16T17:00:01.851 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.851 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.851 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.851 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.851 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.851 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.851 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.851 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.851 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.852 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.852 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.852 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.852 INFO:teuthology.orchestra.run.smithi002.stdout: "device_url": { 2023-11-16T17:00:01.852 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_url", 2023-11-16T17:00:01.852 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.852 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.852 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.852 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-11-16T17:00:01.852 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.853 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.853 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.853 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.853 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.853 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.853 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.853 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.853 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": { 2023-11-16T17:00:01.853 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enabled", 2023-11-16T17:00:01.854 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.854 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.854 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.854 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.854 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.854 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.854 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.854 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.854 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.855 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.855 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.855 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.855 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T17:00:01.855 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T17:00:01.855 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.855 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.855 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.855 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "24", 2023-11-16T17:00:01.855 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "8", 2023-11-16T17:00:01.856 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.856 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.856 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.856 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.856 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.856 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.856 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.856 INFO:teuthology.orchestra.run.smithi002.stdout: "last_opt_revision": { 2023-11-16T17:00:01.856 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "last_opt_revision", 2023-11-16T17:00:01.857 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.857 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.857 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.857 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1", 2023-11-16T17:00:01.857 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.857 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.857 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.857 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.857 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.857 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.858 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.858 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.858 INFO:teuthology.orchestra.run.smithi002.stdout: "leaderboard": { 2023-11-16T17:00:01.858 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "leaderboard", 2023-11-16T17:00:01.858 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.858 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.858 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.858 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.859 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.859 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.859 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.859 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.859 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.859 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.859 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.859 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.859 INFO:teuthology.orchestra.run.smithi002.stdout: "leaderboard_description": { 2023-11-16T17:00:01.860 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "leaderboard_description", 2023-11-16T17:00:01.860 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.860 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.860 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.860 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.860 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.860 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.860 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.861 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.861 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.861 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.861 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.861 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.861 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.861 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.861 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.861 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.862 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.862 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.862 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.862 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.862 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.862 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.862 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.862 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.863 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.863 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.863 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.863 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.863 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.863 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.863 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.863 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.863 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.863 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.864 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.864 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.864 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.864 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.864 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.864 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.864 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.864 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.864 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.865 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.865 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.865 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.865 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.865 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.865 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.865 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.865 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.865 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.865 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.866 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.866 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.866 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.866 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.866 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.866 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.866 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.866 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.866 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.867 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.867 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.867 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.867 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.867 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.867 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.867 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.867 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.867 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.868 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.868 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.868 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.868 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.868 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.868 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.868 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.868 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.868 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.868 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.869 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.869 INFO:teuthology.orchestra.run.smithi002.stdout: "organization": { 2023-11-16T17:00:01.869 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "organization", 2023-11-16T17:00:01.869 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.869 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.869 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.869 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.870 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.870 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.870 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.870 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.870 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.870 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.870 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.870 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.870 INFO:teuthology.orchestra.run.smithi002.stdout: "proxy": { 2023-11-16T17:00:01.871 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "proxy", 2023-11-16T17:00:01.871 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.871 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.871 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.871 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.871 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.871 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.871 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.871 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.872 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.872 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.872 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.872 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.872 INFO:teuthology.orchestra.run.smithi002.stdout: "url": { 2023-11-16T17:00:01.872 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "url", 2023-11-16T17:00:01.872 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.872 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.872 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.873 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-11-16T17:00:01.873 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.873 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.873 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.873 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.873 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.873 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.873 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.873 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.873 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.874 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.874 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.874 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "test_orchestrator", 2023-11-16T17:00:01.874 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.874 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.874 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.874 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.874 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.875 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.875 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.875 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.875 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.875 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.875 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.875 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.875 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.875 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.876 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.876 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.876 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.876 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.876 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.876 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.876 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.876 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.876 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.877 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.877 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.877 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.877 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.877 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.877 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.877 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.877 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.877 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.877 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.878 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.878 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.878 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.878 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.878 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.878 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.878 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.878 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.878 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.879 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.879 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.879 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.879 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.879 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.879 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.879 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.879 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.880 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.880 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.880 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.880 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.880 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.880 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.880 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.880 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.880 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.880 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.881 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.881 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.881 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.881 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.881 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.881 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.881 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.881 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.881 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.882 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.882 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.882 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.882 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.882 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.882 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.882 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.882 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "volumes", 2023-11-16T17:00:01.882 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.882 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.883 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.883 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.883 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.883 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.883 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.883 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.883 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.883 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.883 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.884 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.884 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.884 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.884 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.884 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.884 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.884 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.884 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.884 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.884 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.885 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.885 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.885 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.885 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.885 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.885 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.885 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.885 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.885 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.886 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.886 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.886 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.886 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.886 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.886 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.886 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.886 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.886 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.886 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.887 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.887 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.887 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.887 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.887 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.887 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.887 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.887 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.887 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.888 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.888 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.888 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.888 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.888 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.888 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.888 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.888 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.888 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.888 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.889 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.889 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.889 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.889 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.889 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.889 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.889 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.889 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.889 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.890 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.890 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.890 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.890 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.890 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.890 INFO:teuthology.orchestra.run.smithi002.stdout: "max_concurrent_clones": { 2023-11-16T17:00:01.890 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_concurrent_clones", 2023-11-16T17:00:01.890 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.890 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.891 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.891 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "4", 2023-11-16T17:00:01.891 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.891 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.891 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.891 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Number of asynchronous cloner threads", 2023-11-16T17:00:01.891 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.891 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.891 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.892 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.892 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot_clone_delay": { 2023-11-16T17:00:01.892 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snapshot_clone_delay", 2023-11-16T17:00:01.892 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.892 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.892 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.892 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T17:00:01.892 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.892 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.892 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.893 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-11-16T17:00:01.893 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.893 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.893 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.893 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.893 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.893 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.893 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.893 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix", 2023-11-16T17:00:01.894 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.894 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.894 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.894 INFO:teuthology.orchestra.run.smithi002.stdout: "discovery_interval": { 2023-11-16T17:00:01.894 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "discovery_interval", 2023-11-16T17:00:01.894 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T17:00:01.894 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.894 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.894 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "100", 2023-11-16T17:00:01.895 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.895 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.895 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.895 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.895 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.895 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.895 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.895 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.895 INFO:teuthology.orchestra.run.smithi002.stdout: "identifier": { 2023-11-16T17:00:01.895 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "identifier", 2023-11-16T17:00:01.896 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.896 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.896 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.896 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.896 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.896 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.896 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.896 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.896 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.897 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.897 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.897 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.897 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T17:00:01.897 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T17:00:01.897 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.897 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.897 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.897 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T17:00:01.898 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.898 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.898 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.898 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.898 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.898 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.898 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.898 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.898 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.899 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.899 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.899 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.899 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.899 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.899 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.899 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.899 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.900 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.900 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.900 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.900 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.900 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.900 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.900 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.900 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.900 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.900 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.901 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.901 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.901 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.901 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.901 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.901 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.901 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.901 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.901 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.902 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.902 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.902 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.902 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.902 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.902 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.902 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.902 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.902 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.902 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.903 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.903 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.903 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.903 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.903 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.903 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.903 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.903 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.903 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.904 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.904 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.904 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.904 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.904 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.904 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.904 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.904 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.905 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.905 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.905 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.905 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.905 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.905 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.905 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.905 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.906 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.906 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.906 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.906 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.906 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.906 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.906 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.906 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_host": { 2023-11-16T17:00:01.907 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_host", 2023-11-16T17:00:01.907 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.907 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.907 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.907 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.907 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.907 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.907 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.908 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.908 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.908 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.908 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.908 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.908 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_port": { 2023-11-16T17:00:01.908 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_port", 2023-11-16T17:00:01.908 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.909 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.909 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.909 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10051", 2023-11-16T17:00:01.909 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.909 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.909 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.909 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.910 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.910 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.910 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.910 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.910 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_sender": { 2023-11-16T17:00:01.910 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_sender", 2023-11-16T17:00:01.910 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.910 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.911 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.911 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-11-16T17:00:01.911 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.911 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.911 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.911 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.911 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.911 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.911 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.912 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.912 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.912 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.912 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T17:00:01.912 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.912 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.912 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 25397, 2023-11-16T17:00:01.912 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "y", 2023-11-16T17:00:01.912 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_features": 4540138322906710015, 2023-11-16T17:00:01.913 INFO:teuthology.orchestra.run.smithi002.stdout: "available_modules": [ 2023-11-16T17:00:01.913 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.913 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alerts", 2023-11-16T17:00:01.913 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.913 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.913 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.913 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T17:00:01.913 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T17:00:01.913 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.914 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.914 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.914 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T17:00:01.914 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.914 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.914 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.914 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How frequently to reexamine health status", 2023-11-16T17:00:01.914 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.915 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.915 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.915 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.915 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.915 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.915 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.915 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.915 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.915 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.916 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.916 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.916 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.916 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.916 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.916 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.916 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.916 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.916 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.917 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.917 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.917 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.917 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.917 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.917 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.917 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.917 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.917 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.918 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.918 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.918 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.918 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.918 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.918 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.918 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.918 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.918 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.919 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.919 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.919 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.919 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.919 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.919 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.919 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.919 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.920 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.920 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.920 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.920 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.920 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.920 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.920 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.920 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.921 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.921 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.921 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.921 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.921 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.921 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.921 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.921 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.921 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.922 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.922 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.922 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.922 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.922 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.922 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.922 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.922 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.922 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.923 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.923 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.923 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.923 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_destination": { 2023-11-16T17:00:01.923 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_destination", 2023-11-16T17:00:01.923 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.923 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.923 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.923 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.924 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.924 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.924 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.924 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Email address to send alerts to", 2023-11-16T17:00:01.924 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.924 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.924 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.924 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.924 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_from_name": { 2023-11-16T17:00:01.925 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_from_name", 2023-11-16T17:00:01.925 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.925 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.925 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.925 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "Ceph", 2023-11-16T17:00:01.925 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.925 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.925 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.925 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Email From: name", 2023-11-16T17:00:01.926 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.926 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.926 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.926 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.926 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_host": { 2023-11-16T17:00:01.926 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_host", 2023-11-16T17:00:01.926 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.926 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.927 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.927 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.927 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.927 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.927 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.927 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP server", 2023-11-16T17:00:01.927 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.927 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.927 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.928 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.928 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_password": { 2023-11-16T17:00:01.928 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_password", 2023-11-16T17:00:01.928 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.928 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.928 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.928 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.928 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.929 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.929 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.929 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Password to authenticate with", 2023-11-16T17:00:01.929 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.929 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.929 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.929 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.929 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_port": { 2023-11-16T17:00:01.930 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_port", 2023-11-16T17:00:01.930 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.930 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.930 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.930 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "465", 2023-11-16T17:00:01.930 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.930 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.930 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.930 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP port", 2023-11-16T17:00:01.931 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.931 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.931 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.931 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.931 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_sender": { 2023-11-16T17:00:01.931 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_sender", 2023-11-16T17:00:01.931 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.931 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.931 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.932 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.932 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.932 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.932 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.932 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP envelope sender", 2023-11-16T17:00:01.932 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.932 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.932 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.933 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.933 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_ssl": { 2023-11-16T17:00:01.933 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_ssl", 2023-11-16T17:00:01.933 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.933 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.933 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.933 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.933 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.933 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.934 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.934 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use SSL to connect to SMTP server", 2023-11-16T17:00:01.934 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.934 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.934 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.934 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.934 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_user": { 2023-11-16T17:00:01.934 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_user", 2023-11-16T17:00:01.935 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.935 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.935 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.935 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.935 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.935 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.935 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.935 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "User to authenticate as", 2023-11-16T17:00:01.935 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.936 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.936 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.936 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.936 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.936 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.936 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.936 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "balancer", 2023-11-16T17:00:01.936 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.936 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.937 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.937 INFO:teuthology.orchestra.run.smithi002.stdout: "active": { 2023-11-16T17:00:01.937 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "active", 2023-11-16T17:00:01.937 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.937 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.937 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.937 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.937 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.937 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.938 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.938 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "automatically balance PGs across cluster", 2023-11-16T17:00:01.938 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.938 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.938 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.938 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.938 INFO:teuthology.orchestra.run.smithi002.stdout: "begin_time": { 2023-11-16T17:00:01.938 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "begin_time", 2023-11-16T17:00:01.938 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.939 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.939 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.939 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0000", 2023-11-16T17:00:01.939 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.939 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.939 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.939 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "beginning time of day to automatically balance", 2023-11-16T17:00:01.939 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-16T17:00:01.939 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.940 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.940 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.940 INFO:teuthology.orchestra.run.smithi002.stdout: "begin_weekday": { 2023-11-16T17:00:01.940 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "begin_weekday", 2023-11-16T17:00:01.940 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T17:00:01.940 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.940 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.940 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T17:00:01.941 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0", 2023-11-16T17:00:01.941 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "6", 2023-11-16T17:00:01.941 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.941 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-11-16T17:00:01.941 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-16T17:00:01.941 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.941 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.941 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.941 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_max_iterations": { 2023-11-16T17:00:01.942 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_max_iterations", 2023-11-16T17:00:01.942 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T17:00:01.942 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.942 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.942 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "25", 2023-11-16T17:00:01.942 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T17:00:01.942 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "250", 2023-11-16T17:00:01.942 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.943 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-11-16T17:00:01.943 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.943 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.943 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.943 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.943 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_metrics": { 2023-11-16T17:00:01.943 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_metrics", 2023-11-16T17:00:01.943 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.943 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.944 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.944 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "pgs,objects,bytes", 2023-11-16T17:00:01.944 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.944 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.944 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.944 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-11-16T17:00:01.944 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-11-16T17:00:01.944 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.944 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.945 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.945 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_step": { 2023-11-16T17:00:01.945 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_step", 2023-11-16T17:00:01.945 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:01.945 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.945 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.945 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0.5", 2023-11-16T17:00:01.945 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0.001", 2023-11-16T17:00:01.945 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "0.999", 2023-11-16T17:00:01.946 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.946 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "aggressiveness of optimization", 2023-11-16T17:00:01.946 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-11-16T17:00:01.946 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.946 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.946 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.946 INFO:teuthology.orchestra.run.smithi002.stdout: "end_time": { 2023-11-16T17:00:01.946 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_time", 2023-11-16T17:00:01.947 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.947 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.947 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.947 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "2359", 2023-11-16T17:00:01.947 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.947 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.947 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.947 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "ending time of day to automatically balance", 2023-11-16T17:00:01.947 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-16T17:00:01.948 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.948 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.948 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.948 INFO:teuthology.orchestra.run.smithi002.stdout: "end_weekday": { 2023-11-16T17:00:01.948 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_weekday", 2023-11-16T17:00:01.948 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T17:00:01.948 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.948 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.949 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T17:00:01.949 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0", 2023-11-16T17:00:01.949 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "6", 2023-11-16T17:00:01.949 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.949 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-11-16T17:00:01.949 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-16T17:00:01.949 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.949 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.950 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.950 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:01.950 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:01.950 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.950 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.950 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.950 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.950 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.950 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.951 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.951 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.951 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.951 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.951 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.951 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.951 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.951 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.952 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.952 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.952 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.952 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.952 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.952 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:01.952 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:01.952 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.952 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.953 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.953 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.953 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.953 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.953 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.953 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.953 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.953 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.953 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.954 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.954 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:01.954 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:01.954 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.954 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.954 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.954 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:01.954 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.954 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.955 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.955 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:01.955 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:01.955 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:01.955 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:01.955 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:01.955 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:01.955 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.956 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.956 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.956 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.956 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.956 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.956 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:01.956 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:01.956 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.956 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.957 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.957 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.957 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.957 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.957 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.957 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:01.957 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.957 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.957 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.958 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.958 INFO:teuthology.orchestra.run.smithi002.stdout: "min_score": { 2023-11-16T17:00:01.958 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "min_score", 2023-11-16T17:00:01.958 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:01.958 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.958 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.958 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T17:00:01.958 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.958 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.959 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.959 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-11-16T17:00:01.959 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.959 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.959 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.959 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.959 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": { 2023-11-16T17:00:01.959 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mode", 2023-11-16T17:00:01.960 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.960 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.960 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.960 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "upmap", 2023-11-16T17:00:01.960 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.960 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.960 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:01.960 INFO:teuthology.orchestra.run.smithi002.stdout: "crush-compat", 2023-11-16T17:00:01.960 INFO:teuthology.orchestra.run.smithi002.stdout: "none", 2023-11-16T17:00:01.961 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap" 2023-11-16T17:00:01.961 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:01.961 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Balancer mode", 2023-11-16T17:00:01.961 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.961 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.961 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.961 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.961 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_ids": { 2023-11-16T17:00:01.962 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pool_ids", 2023-11-16T17:00:01.962 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.962 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.962 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.962 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:01.962 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.962 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.962 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.962 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-11-16T17:00:01.963 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.963 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.963 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.963 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.963 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T17:00:01.963 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T17:00:01.963 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.963 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.964 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.964 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T17:00:01.964 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.964 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.964 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.964 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-11-16T17:00:01.964 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.964 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.964 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.965 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.965 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap_max_deviation": { 2023-11-16T17:00:01.965 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "upmap_max_deviation", 2023-11-16T17:00:01.965 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.965 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.965 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.965 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T17:00:01.965 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T17:00:01.965 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.966 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.966 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "deviation below which no optimization is attempted", 2023-11-16T17:00:01.966 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-11-16T17:00:01.966 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.966 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.966 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.966 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap_max_optimizations": { 2023-11-16T17:00:01.966 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "upmap_max_optimizations", 2023-11-16T17:00:01.967 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T17:00:01.967 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.967 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.967 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:01.967 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.967 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.967 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.967 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-11-16T17:00:01.967 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.968 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.968 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.968 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.968 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:01.968 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.968 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:01.968 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephadm", 2023-11-16T17:00:01.968 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:01.968 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:01.969 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:01.969 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_down_multiplier": { 2023-11-16T17:00:01.969 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_down_multiplier", 2023-11-16T17:00:01.969 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:01.969 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.969 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.969 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3.0", 2023-11-16T17:00:01.969 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.970 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.970 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.970 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-11-16T17:00:01.970 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.970 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.970 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.970 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.970 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_refresh_rate": { 2023-11-16T17:00:01.971 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_refresh_rate", 2023-11-16T17:00:01.971 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.971 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.971 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.971 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "20", 2023-11-16T17:00:01.971 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.971 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.971 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.971 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-11-16T17:00:01.972 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.972 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.972 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.972 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.972 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_starting_port": { 2023-11-16T17:00:01.972 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_starting_port", 2023-11-16T17:00:01.972 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:01.972 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.972 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.973 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "4721", 2023-11-16T17:00:01.973 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.973 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.973 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.973 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-11-16T17:00:01.973 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.973 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.973 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.973 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.974 INFO:teuthology.orchestra.run.smithi002.stdout: "alertmanager_web_password": { 2023-11-16T17:00:01.974 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alertmanager_web_password", 2023-11-16T17:00:01.974 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.974 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.974 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.974 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:01.974 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.974 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.975 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.975 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Alertmanager web password", 2023-11-16T17:00:01.975 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.975 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.975 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.975 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.975 INFO:teuthology.orchestra.run.smithi002.stdout: "alertmanager_web_user": { 2023-11-16T17:00:01.975 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alertmanager_web_user", 2023-11-16T17:00:01.975 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.976 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.976 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.976 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:01.976 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.976 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.976 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.976 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Alertmanager web user", 2023-11-16T17:00:01.976 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.977 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.977 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.977 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.977 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_ptrace": { 2023-11-16T17:00:01.977 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_ptrace", 2023-11-16T17:00:01.977 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.977 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.977 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.977 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.978 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.978 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.978 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.978 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-11-16T17:00:01.978 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-11-16T17:00:01.978 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.978 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.978 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.978 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_interval": { 2023-11-16T17:00:01.979 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_interval", 2023-11-16T17:00:01.979 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:01.979 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.979 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.979 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T17:00:01.979 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.979 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.979 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.980 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to autotune daemon memory", 2023-11-16T17:00:01.980 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.980 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.980 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.980 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.980 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_memory_target_ratio": { 2023-11-16T17:00:01.980 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_memory_target_ratio", 2023-11-16T17:00:01.980 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:01.981 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.981 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.981 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0.7", 2023-11-16T17:00:01.981 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.981 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.981 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.981 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-11-16T17:00:01.981 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.981 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.982 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.982 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.982 INFO:teuthology.orchestra.run.smithi002.stdout: "cgroups_split": { 2023-11-16T17:00:01.982 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cgroups_split", 2023-11-16T17:00:01.982 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.982 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.982 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.982 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.983 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.983 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.983 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.983 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-11-16T17:00:01.983 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.983 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.983 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.983 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.983 INFO:teuthology.orchestra.run.smithi002.stdout: "config_checks_enabled": { 2023-11-16T17:00:01.984 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_checks_enabled", 2023-11-16T17:00:01.984 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.984 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.984 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.984 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:01.984 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.984 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.984 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.984 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-11-16T17:00:01.985 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.985 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.985 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.985 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.985 INFO:teuthology.orchestra.run.smithi002.stdout: "config_dashboard": { 2023-11-16T17:00:01.985 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_dashboard", 2023-11-16T17:00:01.985 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:01.985 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.985 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.986 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:01.986 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.986 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.986 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.986 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-11-16T17:00:01.986 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.986 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.986 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.987 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.987 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_alertmanager": { 2023-11-16T17:00:01.987 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_alertmanager", 2023-11-16T17:00:01.987 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.987 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.987 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.987 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-11-16T17:00:01.987 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.988 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.988 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.988 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:01.988 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.988 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.988 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.988 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.988 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_base": { 2023-11-16T17:00:01.988 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_base", 2023-11-16T17:00:01.989 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.989 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.989 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:01.989 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/ceph", 2023-11-16T17:00:01.989 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.989 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.989 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.990 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Container image name, without the tag", 2023-11-16T17:00:01.990 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.990 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.990 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.990 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.990 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_elasticsearch": { 2023-11-16T17:00:01.990 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_elasticsearch", 2023-11-16T17:00:01.990 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.991 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.991 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.991 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-11-16T17:00:01.991 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.991 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.991 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.991 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "elasticsearch container image", 2023-11-16T17:00:01.991 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.991 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.992 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.992 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.992 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_grafana": { 2023-11-16T17:00:01.992 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_grafana", 2023-11-16T17:00:01.992 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.992 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.992 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.992 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-11-16T17:00:01.993 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.993 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.993 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.993 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:01.993 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.993 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.993 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.993 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.993 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_haproxy": { 2023-11-16T17:00:01.994 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_haproxy", 2023-11-16T17:00:01.994 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.994 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.994 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.994 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-11-16T17:00:01.994 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.994 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.995 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.995 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "HAproxy container image", 2023-11-16T17:00:01.995 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.995 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.995 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.995 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.995 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_agent": { 2023-11-16T17:00:01.995 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_agent", 2023-11-16T17:00:01.996 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.996 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.996 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.996 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-11-16T17:00:01.996 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.996 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.996 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.996 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger agent container image", 2023-11-16T17:00:01.996 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.997 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.997 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.997 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.997 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_collector": { 2023-11-16T17:00:01.997 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_collector", 2023-11-16T17:00:01.997 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.997 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.997 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.997 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-11-16T17:00:01.998 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.998 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:01.998 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:01.998 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger collector container image", 2023-11-16T17:00:01.998 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:01.998 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:01.998 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:01.999 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:01.999 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_query": { 2023-11-16T17:00:01.999 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_query", 2023-11-16T17:00:01.999 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:01.999 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:01.999 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:01.999 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-11-16T17:00:01.999 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:01.999 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.000 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.000 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger query container image", 2023-11-16T17:00:02.000 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.000 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.000 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.000 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.000 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_keepalived": { 2023-11-16T17:00:02.000 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_keepalived", 2023-11-16T17:00:02.001 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.001 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.001 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.001 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-11-16T17:00:02.001 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.001 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.001 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.001 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Keepalived container image", 2023-11-16T17:00:02.002 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.002 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.002 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.002 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.002 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_loki": { 2023-11-16T17:00:02.002 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_loki", 2023-11-16T17:00:02.002 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.002 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.003 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.003 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-11-16T17:00:02.003 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.003 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.003 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.003 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Loki container image", 2023-11-16T17:00:02.003 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.003 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.003 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.004 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.004 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_node_exporter": { 2023-11-16T17:00:02.004 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_node_exporter", 2023-11-16T17:00:02.004 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.004 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.004 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.004 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-11-16T17:00:02.004 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.004 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.005 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.005 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:02.005 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.005 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.005 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.005 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.005 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_prometheus": { 2023-11-16T17:00:02.005 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_prometheus", 2023-11-16T17:00:02.005 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.006 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.006 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.006 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-11-16T17:00:02.006 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.006 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.006 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.006 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:02.006 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.006 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.007 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.007 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.007 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_promtail": { 2023-11-16T17:00:02.007 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_promtail", 2023-11-16T17:00:02.007 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.007 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.007 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.007 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-11-16T17:00:02.008 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.008 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.008 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.008 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Promtail container image", 2023-11-16T17:00:02.008 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.008 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.008 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.008 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.009 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_snmp_gateway": { 2023-11-16T17:00:02.009 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_snmp_gateway", 2023-11-16T17:00:02.009 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.009 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.009 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.009 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-11-16T17:00:02.009 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.009 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.010 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.010 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SNMP Gateway container image", 2023-11-16T17:00:02.010 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.010 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.010 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.010 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.010 INFO:teuthology.orchestra.run.smithi002.stdout: "container_init": { 2023-11-16T17:00:02.010 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_init", 2023-11-16T17:00:02.010 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.011 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.011 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.011 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.011 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.011 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.011 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.011 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Run podman/docker with `--init`", 2023-11-16T17:00:02.011 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.012 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.012 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.012 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.012 INFO:teuthology.orchestra.run.smithi002.stdout: "daemon_cache_timeout": { 2023-11-16T17:00:02.012 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "daemon_cache_timeout", 2023-11-16T17:00:02.012 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.012 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.012 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.013 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T17:00:02.013 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.013 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.013 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.013 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-11-16T17:00:02.013 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.013 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.013 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.013 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.014 INFO:teuthology.orchestra.run.smithi002.stdout: "default_cephadm_command_timeout": { 2023-11-16T17:00:02.014 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default_cephadm_command_timeout", 2023-11-16T17:00:02.014 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.014 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.014 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.014 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "900", 2023-11-16T17:00:02.014 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.014 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.015 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.015 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-11-16T17:00:02.015 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.015 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.015 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.015 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.015 INFO:teuthology.orchestra.run.smithi002.stdout: "default_registry": { 2023-11-16T17:00:02.015 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default_registry", 2023-11-16T17:00:02.015 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.016 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.016 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.016 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io", 2023-11-16T17:00:02.016 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.016 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.016 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.016 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-11-16T17:00:02.016 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.017 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.017 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.017 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.017 INFO:teuthology.orchestra.run.smithi002.stdout: "device_cache_timeout": { 2023-11-16T17:00:02.017 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_cache_timeout", 2023-11-16T17:00:02.017 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.017 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.017 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.017 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1800", 2023-11-16T17:00:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache device inventory", 2023-11-16T17:00:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.018 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: "device_enhanced_scan": { 2023-11-16T17:00:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_enhanced_scan", 2023-11-16T17:00:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.019 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use libstoragemgmt during device scans", 2023-11-16T17:00:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: "facts_cache_timeout": { 2023-11-16T17:00:02.020 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "facts_cache_timeout", 2023-11-16T17:00:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T17:00:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.021 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache host facts data", 2023-11-16T17:00:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: "host_check_interval": { 2023-11-16T17:00:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "host_check_interval", 2023-11-16T17:00:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.022 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T17:00:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to perform a host check", 2023-11-16T17:00:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.023 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.024 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.025 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.026 INFO:teuthology.orchestra.run.smithi002.stdout: "log_refresh_metadata": { 2023-11-16T17:00:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_refresh_metadata", 2023-11-16T17:00:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.027 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-11-16T17:00:02.028 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.028 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.028 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.028 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.028 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.028 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.028 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.028 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.028 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.029 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.029 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.029 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.029 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.029 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-11-16T17:00:02.029 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.029 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.029 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.030 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.030 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.030 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.030 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.030 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.030 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.030 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.030 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.031 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.031 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.031 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.031 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.031 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.031 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.031 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.031 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.032 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.032 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.032 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.032 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.032 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.032 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.032 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.032 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.032 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.033 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.033 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.033 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.033 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.033 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.033 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.033 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.033 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.034 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.034 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.034 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.034 INFO:teuthology.orchestra.run.smithi002.stdout: "manage_etc_ceph_ceph_conf": { 2023-11-16T17:00:02.034 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-11-16T17:00:02.034 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.034 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.034 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.034 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.035 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.035 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.035 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.035 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-11-16T17:00:02.035 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.035 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.035 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.035 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.036 INFO:teuthology.orchestra.run.smithi002.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-11-16T17:00:02.036 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-11-16T17:00:02.036 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.036 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.036 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.036 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "*", 2023-11-16T17:00:02.036 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.036 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.036 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.037 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-11-16T17:00:02.037 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.037 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.037 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.037 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.037 INFO:teuthology.orchestra.run.smithi002.stdout: "max_count_per_host": { 2023-11-16T17:00:02.037 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_count_per_host", 2023-11-16T17:00:02.037 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.038 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.038 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.038 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:02.038 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.038 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.038 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.038 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "max number of daemons per service per host", 2023-11-16T17:00:02.038 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.039 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.039 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.039 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.039 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd_draining_count": { 2023-11-16T17:00:02.039 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_osd_draining_count", 2023-11-16T17:00:02.039 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.039 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.039 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.040 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:02.040 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.040 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.040 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.040 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-11-16T17:00:02.040 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.040 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.041 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.041 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.041 INFO:teuthology.orchestra.run.smithi002.stdout: "migration_current": { 2023-11-16T17:00:02.041 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "migration_current", 2023-11-16T17:00:02.041 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.041 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.041 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.041 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.041 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.042 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.042 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.042 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "internal - do not modify", 2023-11-16T17:00:02.042 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.042 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.042 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.042 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.042 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": { 2023-11-16T17:00:02.043 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mode", 2023-11-16T17:00:02.043 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.043 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.043 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.043 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "root", 2023-11-16T17:00:02.043 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.043 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.043 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.043 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm-package", 2023-11-16T17:00:02.044 INFO:teuthology.orchestra.run.smithi002.stdout: "root" 2023-11-16T17:00:02.044 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.044 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "mode for remote execution of cephadm", 2023-11-16T17:00:02.044 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.044 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.044 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.044 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.044 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_alerts_path": { 2023-11-16T17:00:02.045 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_alerts_path", 2023-11-16T17:00:02.045 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.045 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.045 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.045 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-11-16T17:00:02.045 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.045 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.045 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.045 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-11-16T17:00:02.046 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.046 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.046 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.046 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.046 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_web_password": { 2023-11-16T17:00:02.046 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_web_password", 2023-11-16T17:00:02.046 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.046 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.047 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.047 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:02.047 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.047 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.047 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.047 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus web password", 2023-11-16T17:00:02.047 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.047 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.047 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.048 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.048 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_web_user": { 2023-11-16T17:00:02.048 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_web_user", 2023-11-16T17:00:02.048 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.048 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.048 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.048 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:02.048 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.049 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.049 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.049 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus web user", 2023-11-16T17:00:02.049 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.049 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.049 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.049 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.049 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_insecure": { 2023-11-16T17:00:02.049 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_insecure", 2023-11-16T17:00:02.050 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.050 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.050 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.050 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.050 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.050 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.050 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.050 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-11-16T17:00:02.051 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.051 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.051 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.051 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.051 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_password": { 2023-11-16T17:00:02.051 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_password", 2023-11-16T17:00:02.051 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.051 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.052 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.052 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.052 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.052 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.052 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.052 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-11-16T17:00:02.052 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.052 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.052 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.053 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.053 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_url": { 2023-11-16T17:00:02.053 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_url", 2023-11-16T17:00:02.053 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.053 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.053 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.053 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.053 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.053 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.054 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.054 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-11-16T17:00:02.054 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.054 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.054 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.054 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.054 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_username": { 2023-11-16T17:00:02.055 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_username", 2023-11-16T17:00:02.055 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.055 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.055 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.055 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.055 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.055 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.055 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.055 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-11-16T17:00:02.056 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.056 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.056 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.056 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.056 INFO:teuthology.orchestra.run.smithi002.stdout: "secure_monitoring_stack": { 2023-11-16T17:00:02.056 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "secure_monitoring_stack", 2023-11-16T17:00:02.056 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.056 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.056 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.057 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.057 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.057 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.057 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.057 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-11-16T17:00:02.057 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.057 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.057 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.057 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.058 INFO:teuthology.orchestra.run.smithi002.stdout: "service_discovery_port": { 2023-11-16T17:00:02.058 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "service_discovery_port", 2023-11-16T17:00:02.058 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.058 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.058 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.058 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8765", 2023-11-16T17:00:02.058 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.058 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.059 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.059 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "cephadm service discovery port", 2023-11-16T17:00:02.059 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.059 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.059 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.059 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.059 INFO:teuthology.orchestra.run.smithi002.stdout: "ssh_config_file": { 2023-11-16T17:00:02.060 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssh_config_file", 2023-11-16T17:00:02.060 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.060 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.060 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.060 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.060 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.060 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.060 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.060 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-11-16T17:00:02.061 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.061 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.061 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.061 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.061 INFO:teuthology.orchestra.run.smithi002.stdout: "use_agent": { 2023-11-16T17:00:02.061 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "use_agent", 2023-11-16T17:00:02.061 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.061 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.061 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.062 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.062 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.062 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.062 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.062 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-11-16T17:00:02.062 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.062 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.062 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.062 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.063 INFO:teuthology.orchestra.run.smithi002.stdout: "use_repo_digest": { 2023-11-16T17:00:02.063 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "use_repo_digest", 2023-11-16T17:00:02.063 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.063 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.063 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.063 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.063 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.063 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.064 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.064 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-11-16T17:00:02.064 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.064 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.064 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.064 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.064 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_failed_host_check": { 2023-11-16T17:00:02.064 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_failed_host_check", 2023-11-16T17:00:02.065 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.065 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.065 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.065 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.065 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.065 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.065 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.065 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if the host check fails", 2023-11-16T17:00:02.065 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.066 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.066 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.066 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.066 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_stray_daemons": { 2023-11-16T17:00:02.066 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_stray_daemons", 2023-11-16T17:00:02.066 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.066 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.066 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.066 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.067 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.067 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.067 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.067 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-11-16T17:00:02.067 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.067 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.067 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.067 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.068 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_stray_hosts": { 2023-11-16T17:00:02.068 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_stray_hosts", 2023-11-16T17:00:02.068 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.068 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.068 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.068 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.068 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.068 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.068 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.069 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-11-16T17:00:02.069 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.069 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.069 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.069 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.069 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.069 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.069 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.070 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crash", 2023-11-16T17:00:02.070 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.070 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.070 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.070 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.070 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.070 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.070 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.070 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.071 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.071 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.071 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.071 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.071 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.071 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.071 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.071 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.071 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.072 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.072 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.072 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.072 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.072 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.072 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.072 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.072 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.072 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.073 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.073 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.073 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.073 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.073 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.073 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.073 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.073 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.074 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.074 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.074 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.074 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.074 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.074 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.074 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.074 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.074 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.075 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.075 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.075 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.075 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.075 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.075 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.075 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.075 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.075 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.076 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.076 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.076 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.076 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.076 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.076 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.076 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.076 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.076 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.077 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.077 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.077 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.077 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.077 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.077 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.077 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.077 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.078 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.078 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.078 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.078 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.078 INFO:teuthology.orchestra.run.smithi002.stdout: "retain_interval": { 2023-11-16T17:00:02.078 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retain_interval", 2023-11-16T17:00:02.078 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.078 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.078 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.079 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "31536000", 2023-11-16T17:00:02.079 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.079 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.079 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.079 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long to retain crashes before pruning them", 2023-11-16T17:00:02.079 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.079 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.079 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.080 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.080 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_recent_interval": { 2023-11-16T17:00:02.080 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_recent_interval", 2023-11-16T17:00:02.080 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.080 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.080 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.080 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1209600", 2023-11-16T17:00:02.080 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.080 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.081 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.081 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "time interval in which to warn about recent crashes", 2023-11-16T17:00:02.081 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.081 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.081 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.081 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.081 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.081 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.082 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.082 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "dashboard", 2023-11-16T17:00:02.082 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.082 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.082 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.082 INFO:teuthology.orchestra.run.smithi002.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-11-16T17:00:02.082 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-11-16T17:00:02.082 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.082 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.083 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.083 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:02.083 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.083 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.083 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.083 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.083 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.083 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.084 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.084 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.084 INFO:teuthology.orchestra.run.smithi002.stdout: "ALERTMANAGER_API_HOST": { 2023-11-16T17:00:02.084 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ALERTMANAGER_API_HOST", 2023-11-16T17:00:02.084 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.084 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.084 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.084 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.085 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.085 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.085 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.085 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.085 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.085 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.085 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.085 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.085 INFO:teuthology.orchestra.run.smithi002.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-11-16T17:00:02.086 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-11-16T17:00:02.086 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.086 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.086 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.086 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.086 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.086 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.086 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.086 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.087 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.087 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.087 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.087 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.087 INFO:teuthology.orchestra.run.smithi002.stdout: "AUDIT_API_ENABLED": { 2023-11-16T17:00:02.087 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "AUDIT_API_ENABLED", 2023-11-16T17:00:02.087 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.087 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.088 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.088 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.088 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.088 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.088 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.088 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.088 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.088 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.088 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.089 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.089 INFO:teuthology.orchestra.run.smithi002.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-11-16T17:00:02.089 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-11-16T17:00:02.089 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.089 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.089 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.089 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.089 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.090 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.090 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.090 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.090 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.090 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.090 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.090 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.090 INFO:teuthology.orchestra.run.smithi002.stdout: "ENABLE_BROWSABLE_API": { 2023-11-16T17:00:02.090 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ENABLE_BROWSABLE_API", 2023-11-16T17:00:02.091 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.091 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.091 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.091 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.091 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.091 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.091 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.091 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.092 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.092 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.092 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.092 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.092 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-11-16T17:00:02.092 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-11-16T17:00:02.092 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.092 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.092 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.093 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.093 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.093 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.093 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.093 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.093 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.093 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.093 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.094 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.094 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-11-16T17:00:02.094 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-11-16T17:00:02.094 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.094 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.094 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.094 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.094 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.094 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.095 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.095 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.095 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.095 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.095 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.095 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.095 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-11-16T17:00:02.095 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-11-16T17:00:02.095 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.096 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.096 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.096 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.096 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.096 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.096 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.096 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.096 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.096 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.097 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.097 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.097 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-11-16T17:00:02.097 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-11-16T17:00:02.097 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.097 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.097 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.097 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.098 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.098 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.098 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.098 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.098 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.098 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.098 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.098 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.098 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_NFS": { 2023-11-16T17:00:02.099 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-11-16T17:00:02.099 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.099 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.099 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.099 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.099 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.099 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.100 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.100 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.100 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.100 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.100 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.100 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.100 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RBD": { 2023-11-16T17:00:02.100 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-11-16T17:00:02.100 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.101 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.101 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.101 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.101 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.101 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.101 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.101 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.101 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.101 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.102 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.102 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.102 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RGW": { 2023-11-16T17:00:02.102 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-11-16T17:00:02.102 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.102 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.102 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.102 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.103 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.103 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.103 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.103 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.103 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.103 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.103 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.103 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.103 INFO:teuthology.orchestra.run.smithi002.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-11-16T17:00:02.104 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-11-16T17:00:02.104 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.104 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.104 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.104 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.104 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.104 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.104 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.104 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.105 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.105 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.105 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.105 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.105 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_PASSWORD": { 2023-11-16T17:00:02.105 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_PASSWORD", 2023-11-16T17:00:02.105 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.105 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.105 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.106 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:02.106 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.106 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.106 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.106 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.106 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.106 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.106 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.107 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.107 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-11-16T17:00:02.107 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-11-16T17:00:02.107 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.107 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.107 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.107 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.107 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.107 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.108 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.108 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.108 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.108 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.108 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.108 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.108 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_URL": { 2023-11-16T17:00:02.108 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_URL", 2023-11-16T17:00:02.108 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.109 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.109 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.109 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.109 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.109 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.109 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.109 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.109 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.110 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.110 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.110 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.110 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_USERNAME": { 2023-11-16T17:00:02.110 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_USERNAME", 2023-11-16T17:00:02.110 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.110 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.110 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.111 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:02.111 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.111 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.111 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.111 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.111 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.111 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.111 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.111 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.112 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-11-16T17:00:02.112 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-11-16T17:00:02.112 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.112 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.112 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.112 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.112 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.112 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.113 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.113 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.113 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.113 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.113 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.113 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.113 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-11-16T17:00:02.113 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-11-16T17:00:02.113 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.114 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.114 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.114 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.114 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.114 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.114 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.114 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.114 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.115 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.115 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.115 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.115 INFO:teuthology.orchestra.run.smithi002.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-11-16T17:00:02.115 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-11-16T17:00:02.115 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.115 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.115 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.115 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.116 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.116 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.116 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.116 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.116 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.116 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.116 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.116 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.117 INFO:teuthology.orchestra.run.smithi002.stdout: "ISSUE_TRACKER_API_KEY": { 2023-11-16T17:00:02.117 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-11-16T17:00:02.117 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.117 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.117 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.117 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.117 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.117 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.118 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.118 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.118 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.118 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.118 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.118 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.119 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_HOST": { 2023-11-16T17:00:02.119 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_HOST", 2023-11-16T17:00:02.119 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.119 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.119 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.120 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.120 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.120 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.120 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.120 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.120 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.120 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.120 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.121 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.121 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-11-16T17:00:02.121 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-11-16T17:00:02.121 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.121 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.121 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.121 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.121 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.122 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.122 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.122 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.122 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.122 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.122 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.122 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.122 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-11-16T17:00:02.123 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-11-16T17:00:02.123 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.123 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.123 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.123 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.123 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.123 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.123 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.124 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.124 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.124 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.124 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.124 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.124 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-11-16T17:00:02.124 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-11-16T17:00:02.124 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.125 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.125 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.125 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.125 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.125 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.125 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.125 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.125 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.125 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.126 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.126 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.126 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-11-16T17:00:02.126 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-11-16T17:00:02.126 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.126 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.126 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.127 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.127 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.127 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.127 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.127 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.127 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.127 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.127 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.128 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.128 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-11-16T17:00:02.128 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-11-16T17:00:02.128 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.128 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.128 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.128 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.128 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.129 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.129 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.129 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.129 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.129 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.129 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.129 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.130 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-11-16T17:00:02.130 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-11-16T17:00:02.130 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.130 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.130 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.130 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.130 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.130 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.130 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.131 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.131 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.131 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.131 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.131 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.131 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-11-16T17:00:02.131 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-11-16T17:00:02.131 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.132 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.132 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.132 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.132 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.132 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.132 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.132 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.132 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.133 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.133 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.133 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.133 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-11-16T17:00:02.133 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-11-16T17:00:02.133 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.133 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.133 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.134 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.134 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.134 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.134 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.134 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.134 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.134 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.134 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.134 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.135 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_ENABLED": { 2023-11-16T17:00:02.135 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_ENABLED", 2023-11-16T17:00:02.135 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.135 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.135 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.135 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.135 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.135 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.136 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.136 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.136 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.136 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.136 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.136 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.136 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-11-16T17:00:02.136 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-11-16T17:00:02.137 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.137 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.137 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.137 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-11-16T17:00:02.137 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.137 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.137 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.137 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.138 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.138 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.138 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.138 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.138 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-11-16T17:00:02.138 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-11-16T17:00:02.138 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.139 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.139 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.139 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:02.139 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.139 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.139 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.139 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.139 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-11-16T17:00:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-11-16T17:00:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.140 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.141 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8", 2023-11-16T17:00:02.141 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.141 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.141 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.141 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.141 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.141 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.141 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.142 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.142 INFO:teuthology.orchestra.run.smithi002.stdout: "REST_REQUESTS_TIMEOUT": { 2023-11-16T17:00:02.142 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-11-16T17:00:02.142 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.142 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.142 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.142 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "45", 2023-11-16T17:00:02.143 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.143 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.143 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.143 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.143 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.143 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.143 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.143 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.143 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_ACCESS_KEY": { 2023-11-16T17:00:02.144 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_ACCESS_KEY", 2023-11-16T17:00:02.144 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.144 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.144 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.144 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.144 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.144 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.144 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.145 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.145 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.145 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.145 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.145 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.145 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-11-16T17:00:02.145 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-11-16T17:00:02.145 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.146 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.146 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.146 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:02.146 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.146 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.146 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.146 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.146 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.147 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.147 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.147 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.147 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_SECRET_KEY": { 2023-11-16T17:00:02.147 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_SECRET_KEY", 2023-11-16T17:00:02.147 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.147 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.147 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.147 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.148 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.148 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.148 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.148 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.148 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.148 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.148 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.148 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.149 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_SSL_VERIFY": { 2023-11-16T17:00:02.149 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_SSL_VERIFY", 2023-11-16T17:00:02.149 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.149 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.149 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.149 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.149 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.149 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.150 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.150 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.150 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.150 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.150 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.150 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.150 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-11-16T17:00:02.150 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-11-16T17:00:02.151 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.151 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.151 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.151 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T17:00:02.151 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.151 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.151 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.151 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.152 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.152 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.152 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.152 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.152 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-11-16T17:00:02.152 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-11-16T17:00:02.152 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.152 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.153 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.153 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:02.153 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.153 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.153 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.153 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.153 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.153 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.154 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.154 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.154 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-11-16T17:00:02.154 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-11-16T17:00:02.154 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.154 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.154 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.154 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T17:00:02.155 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.155 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.155 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.155 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.155 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.155 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.155 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.155 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.155 INFO:teuthology.orchestra.run.smithi002.stdout: "cross_origin_url": { 2023-11-16T17:00:02.156 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cross_origin_url", 2023-11-16T17:00:02.156 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.156 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.156 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.156 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.156 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.156 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.156 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.156 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.157 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.157 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.157 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.157 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.157 INFO:teuthology.orchestra.run.smithi002.stdout: "crt_file": { 2023-11-16T17:00:02.157 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crt_file", 2023-11-16T17:00:02.157 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.157 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.158 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.158 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.158 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.158 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.158 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.158 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.158 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.158 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.159 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.159 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.159 INFO:teuthology.orchestra.run.smithi002.stdout: "debug": { 2023-11-16T17:00:02.159 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "debug", 2023-11-16T17:00:02.159 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.159 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.159 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.159 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.160 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.160 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.160 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.160 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable/disable debug options", 2023-11-16T17:00:02.160 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.160 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.160 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.160 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.161 INFO:teuthology.orchestra.run.smithi002.stdout: "jwt_token_ttl": { 2023-11-16T17:00:02.161 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "jwt_token_ttl", 2023-11-16T17:00:02.161 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.161 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.161 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.161 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "28800", 2023-11-16T17:00:02.161 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.161 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.161 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.162 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.162 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.162 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.162 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.162 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.162 INFO:teuthology.orchestra.run.smithi002.stdout: "key_file": { 2023-11-16T17:00:02.162 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "key_file", 2023-11-16T17:00:02.163 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.163 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.163 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.163 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.163 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.163 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.163 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.163 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.163 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.164 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.164 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.164 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.164 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.164 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.164 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.164 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.164 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.165 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.165 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.165 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.165 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.165 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.165 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.165 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.165 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.165 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.166 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.166 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.166 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.166 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.166 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.166 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.166 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.167 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.167 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.167 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.167 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.167 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.167 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.167 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.167 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: "motd": { 2023-11-16T17:00:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "motd", 2023-11-16T17:00:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "The message of the day", 2023-11-16T17:00:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect_resolve_ip_addr": { 2023-11-16T17:00:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "redirect_resolve_ip_addr", 2023-11-16T17:00:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2023-11-16T17:00:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2023-11-16T17:00:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-11-16T17:00:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2023-11-16T17:00:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2023-11-16T17:00:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8080", 2023-11-16T17:00:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.179 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.179 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.179 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.179 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.179 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.179 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.179 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl": { 2023-11-16T17:00:02.179 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl", 2023-11-16T17:00:02.180 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.180 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.180 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.180 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.180 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.180 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.180 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.180 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.181 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.181 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.181 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.181 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.181 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl_server_port": { 2023-11-16T17:00:02.181 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl_server_port", 2023-11-16T17:00:02.181 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.181 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.182 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.182 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8443", 2023-11-16T17:00:02.182 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.182 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.182 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.182 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.182 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.182 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.182 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.183 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.183 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_behaviour": { 2023-11-16T17:00:02.183 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_behaviour", 2023-11-16T17:00:02.183 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.183 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.183 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.183 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "redirect", 2023-11-16T17:00:02.183 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.184 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.184 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.184 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.184 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect" 2023-11-16T17:00:02.184 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.184 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.184 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.184 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.184 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.185 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.185 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_error_status_code": { 2023-11-16T17:00:02.185 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_error_status_code", 2023-11-16T17:00:02.185 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.185 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.185 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.185 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", 2023-11-16T17:00:02.186 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-11-16T17:00:02.186 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "599", 2023-11-16T17:00:02.186 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.186 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.186 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.186 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.186 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.186 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.187 INFO:teuthology.orchestra.run.smithi002.stdout: "url_prefix": { 2023-11-16T17:00:02.187 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "url_prefix", 2023-11-16T17:00:02.187 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.187 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.187 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.187 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.187 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.187 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.188 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.188 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.188 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.188 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.188 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.188 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.188 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.188 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.189 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.189 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", 2023-11-16T17:00:02.189 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.189 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.189 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.189 INFO:teuthology.orchestra.run.smithi002.stdout: "enable_monitoring": { 2023-11-16T17:00:02.189 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enable_monitoring", 2023-11-16T17:00:02.189 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.190 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.190 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.190 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.190 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.190 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.190 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.190 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "monitor device health metrics", 2023-11-16T17:00:02.191 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.191 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.191 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.191 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.191 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.191 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.191 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.191 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.192 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.192 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.192 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.192 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.192 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.192 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.192 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.192 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.193 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.193 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.193 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.193 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.193 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.193 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.193 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.193 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.194 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.195 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.196 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.196 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.196 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.196 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.196 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.196 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.196 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.196 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.197 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.197 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.197 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.197 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.197 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.197 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.197 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.197 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.198 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.198 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.198 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.198 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.198 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.198 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.198 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.199 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.199 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.199 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.199 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.199 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.199 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.199 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.199 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.200 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.200 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.200 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.200 INFO:teuthology.orchestra.run.smithi002.stdout: "mark_out_threshold": { 2023-11-16T17:00:02.200 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mark_out_threshold", 2023-11-16T17:00:02.200 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.200 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.200 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.201 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "2419200", 2023-11-16T17:00:02.201 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.201 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.201 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.201 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-11-16T17:00:02.201 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.201 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.201 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.202 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.202 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": { 2023-11-16T17:00:02.202 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pool_name", 2023-11-16T17:00:02.202 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.202 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.202 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.202 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "device_health_metrics", 2023-11-16T17:00:02.202 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.203 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.203 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.203 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "name of pool in which to store device health metrics", 2023-11-16T17:00:02.203 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.203 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.203 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.203 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.203 INFO:teuthology.orchestra.run.smithi002.stdout: "retention_period": { 2023-11-16T17:00:02.204 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retention_period", 2023-11-16T17:00:02.204 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.204 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.204 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.204 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15552000", 2023-11-16T17:00:02.204 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.204 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.204 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.205 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long to retain device health metrics", 2023-11-16T17:00:02.205 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.205 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.205 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.205 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.205 INFO:teuthology.orchestra.run.smithi002.stdout: "scrape_frequency": { 2023-11-16T17:00:02.205 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "scrape_frequency", 2023-11-16T17:00:02.205 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.206 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.206 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.206 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "86400", 2023-11-16T17:00:02.206 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.206 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.206 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.206 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to scrape device health metrics", 2023-11-16T17:00:02.206 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.206 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.207 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.207 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.207 INFO:teuthology.orchestra.run.smithi002.stdout: "self_heal": { 2023-11-16T17:00:02.207 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "self_heal", 2023-11-16T17:00:02.207 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.207 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.207 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.208 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.208 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.208 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.208 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.208 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-11-16T17:00:02.208 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.208 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.208 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.209 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.209 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T17:00:02.209 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T17:00:02.209 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.209 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.209 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.209 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T17:00:02.209 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.210 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.210 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.210 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to wake up and check device health", 2023-11-16T17:00:02.210 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.210 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.210 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.210 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_threshold": { 2023-11-16T17:00:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_threshold", 2023-11-16T17:00:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "7257600", 2023-11-16T17:00:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.211 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-11-16T17:00:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.212 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "diskprediction_local", 2023-11-16T17:00:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.213 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.214 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.214 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.214 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.214 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.214 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.214 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.214 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.214 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.215 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.215 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.215 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.215 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.215 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.215 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.215 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.215 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.216 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.216 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.216 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.216 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.216 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.216 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.216 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.216 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.217 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.217 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.217 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.217 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.217 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.217 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.217 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.217 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.217 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.218 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.218 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.218 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.218 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.218 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.218 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.218 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.219 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.219 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.219 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.219 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.219 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.219 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.219 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.219 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.220 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.220 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.220 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.220 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.220 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.220 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.220 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.220 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.221 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.221 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.221 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.221 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.221 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.221 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.221 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.221 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.222 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.222 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.222 INFO:teuthology.orchestra.run.smithi002.stdout: "predict_interval": { 2023-11-16T17:00:02.222 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "predict_interval", 2023-11-16T17:00:02.222 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.222 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.222 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.222 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "86400", 2023-11-16T17:00:02.223 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.223 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.223 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.223 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.223 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.223 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.223 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.223 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.223 INFO:teuthology.orchestra.run.smithi002.stdout: "predictor_model": { 2023-11-16T17:00:02.224 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "predictor_model", 2023-11-16T17:00:02.224 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.224 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.224 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.224 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "prophetstor", 2023-11-16T17:00:02.224 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.224 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.224 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.224 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.225 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.225 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.225 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.225 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.225 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T17:00:02.225 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T17:00:02.225 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.225 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.226 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.226 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T17:00:02.226 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.226 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.226 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.226 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.226 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.226 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.226 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.227 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.227 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.227 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.227 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.227 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "influx", 2023-11-16T17:00:02.227 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": false, 2023-11-16T17:00:02.227 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "influxdb python module not found", 2023-11-16T17:00:02.227 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.228 INFO:teuthology.orchestra.run.smithi002.stdout: "batch_size": { 2023-11-16T17:00:02.228 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "batch_size", 2023-11-16T17:00:02.228 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.228 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.228 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.228 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5000", 2023-11-16T17:00:02.228 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.228 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.228 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.229 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-11-16T17:00:02.229 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.229 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.229 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.229 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.229 INFO:teuthology.orchestra.run.smithi002.stdout: "database": { 2023-11-16T17:00:02.229 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "database", 2023-11-16T17:00:02.229 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.230 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.230 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.230 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "ceph", 2023-11-16T17:00:02.230 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.230 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.230 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.230 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-11-16T17:00:02.230 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.230 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.231 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.231 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.231 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": { 2023-11-16T17:00:02.231 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "hostname", 2023-11-16T17:00:02.231 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.231 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.231 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.231 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.232 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.232 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.232 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.232 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB server hostname", 2023-11-16T17:00:02.232 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.232 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.232 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.232 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.232 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T17:00:02.233 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T17:00:02.233 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.233 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.233 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.233 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "30", 2023-11-16T17:00:02.233 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "5", 2023-11-16T17:00:02.233 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.233 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.233 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-11-16T17:00:02.234 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.234 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.234 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.234 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.234 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.234 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.234 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.234 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.235 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.235 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.235 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.235 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.235 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.235 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.235 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.235 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.235 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.238 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.238 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.238 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.238 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.238 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.238 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.238 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.238 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "password": { 2023-11-16T17:00:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "password", 2023-11-16T17:00:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.244 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "password of InfluxDB server user", 2023-11-16T17:00:02.244 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.244 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.244 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.244 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.244 INFO:teuthology.orchestra.run.smithi002.stdout: "port": { 2023-11-16T17:00:02.244 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "port", 2023-11-16T17:00:02.244 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.244 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.245 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.245 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8086", 2023-11-16T17:00:02.245 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.245 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.245 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.245 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB server port", 2023-11-16T17:00:02.245 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.245 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.245 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.246 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.246 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl": { 2023-11-16T17:00:02.246 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl", 2023-11-16T17:00:02.246 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.246 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.246 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.246 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "false", 2023-11-16T17:00:02.246 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.246 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.247 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.247 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-11-16T17:00:02.247 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.247 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.247 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.247 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.247 INFO:teuthology.orchestra.run.smithi002.stdout: "threads": { 2023-11-16T17:00:02.247 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "threads", 2023-11-16T17:00:02.247 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.248 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.248 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.248 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T17:00:02.248 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T17:00:02.248 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "32", 2023-11-16T17:00:02.248 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.248 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-11-16T17:00:02.248 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.249 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.249 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.249 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.249 INFO:teuthology.orchestra.run.smithi002.stdout: "username": { 2023-11-16T17:00:02.249 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "username", 2023-11-16T17:00:02.249 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.249 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.249 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.249 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.250 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.250 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.250 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.250 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "username of InfluxDB server user", 2023-11-16T17:00:02.250 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.250 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.250 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.250 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: "verify_ssl": { 2023-11-16T17:00:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "verify_ssl", 2023-11-16T17:00:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "true", 2023-11-16T17:00:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.251 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.252 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-11-16T17:00:02.252 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.252 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.252 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.252 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.252 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.252 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.252 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.252 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "insights", 2023-11-16T17:00:02.253 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.253 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.253 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.253 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.253 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.253 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.253 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.253 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.254 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.254 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.254 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.254 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.254 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.254 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.254 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.254 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.254 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.255 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.255 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.255 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.255 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.255 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.255 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.255 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.255 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.255 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.256 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.256 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.256 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.256 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.256 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.256 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.256 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.256 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.257 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.257 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.257 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.257 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.257 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.257 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.257 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.257 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.257 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.258 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.258 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.258 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.258 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.258 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.258 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.258 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.259 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.259 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.259 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.259 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.259 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.259 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.259 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.259 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.260 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.260 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.260 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.260 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.260 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.260 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.260 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.260 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.261 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.261 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.261 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.261 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.261 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.261 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.261 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.261 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.261 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.262 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.262 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "iostat", 2023-11-16T17:00:02.262 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.262 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.262 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.262 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.262 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.262 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.263 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.263 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.263 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.263 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.263 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.263 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.263 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.263 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.264 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.264 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.264 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.264 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.264 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.264 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.264 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.264 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.265 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.265 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.265 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.265 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.265 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.265 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.265 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.266 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.266 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.266 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.266 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.266 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.266 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.266 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.266 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.267 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.267 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.267 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.267 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.267 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.267 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.267 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.267 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.268 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.268 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.268 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.268 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.268 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.268 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.268 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.268 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.269 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.269 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.269 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.269 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.269 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.269 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.269 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.270 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.270 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.270 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.270 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.270 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.270 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.270 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.270 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.271 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.271 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.271 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.271 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.271 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.271 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.271 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.271 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.271 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "k8sevents", 2023-11-16T17:00:02.272 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.272 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.272 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.272 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_event_retention_days": { 2023-11-16T17:00:02.272 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ceph_event_retention_days", 2023-11-16T17:00:02.272 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.272 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.272 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.273 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "7", 2023-11-16T17:00:02.273 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.273 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.273 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.273 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Days to hold ceph event information within local cache", 2023-11-16T17:00:02.273 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.273 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.273 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.273 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.274 INFO:teuthology.orchestra.run.smithi002.stdout: "config_check_secs": { 2023-11-16T17:00:02.274 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_check_secs", 2023-11-16T17:00:02.274 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.274 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.274 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.274 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:02.274 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "10", 2023-11-16T17:00:02.274 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.275 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.275 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-11-16T17:00:02.275 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.275 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.275 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.275 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.275 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.275 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.275 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.276 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.276 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.276 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.276 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.276 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.276 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.276 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.276 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.277 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.277 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.277 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.277 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.277 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.277 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.277 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.277 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.277 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.278 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.278 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.278 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.278 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.278 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.278 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.278 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.278 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.279 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.279 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.279 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.279 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.279 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.279 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.279 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.279 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.280 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.280 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.280 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.280 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.280 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.280 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.280 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.280 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.281 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.281 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.281 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.281 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.281 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.281 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.281 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.281 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.281 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.282 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.282 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.282 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.282 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.282 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.282 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.282 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.282 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.283 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.283 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.283 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.283 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.283 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.283 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.283 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.283 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.283 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.284 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.284 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.284 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.284 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "localpool", 2023-11-16T17:00:02.284 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.284 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.284 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.284 INFO:teuthology.orchestra.run.smithi002.stdout: "failure_domain": { 2023-11-16T17:00:02.285 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "failure_domain", 2023-11-16T17:00:02.285 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.285 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.285 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.285 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "host", 2023-11-16T17:00:02.285 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.285 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.285 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.286 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "failure domain for any created local pool", 2023-11-16T17:00:02.286 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-11-16T17:00:02.286 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.286 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.286 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.286 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.286 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.287 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.287 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.287 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.287 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.287 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.287 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.287 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.287 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.287 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.288 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.288 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.288 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.288 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.288 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.288 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.288 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.289 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.289 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.289 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.289 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.289 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.289 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.289 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.289 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.290 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.290 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.290 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.290 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.290 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.290 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.290 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.291 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.291 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.291 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.291 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.291 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.291 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.291 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.291 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.292 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.292 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.292 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.292 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.292 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.292 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.292 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.292 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.292 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.293 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.293 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.293 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.293 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.293 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.293 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.293 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.293 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.294 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.294 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.294 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.294 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.294 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.294 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.294 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.294 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.295 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.295 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.295 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.295 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.295 INFO:teuthology.orchestra.run.smithi002.stdout: "min_size": { 2023-11-16T17:00:02.295 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "min_size", 2023-11-16T17:00:02.295 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.295 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.296 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.296 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.296 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.296 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.296 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.296 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default min_size for any created local pool", 2023-11-16T17:00:02.296 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-11-16T17:00:02.296 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.297 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.297 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.297 INFO:teuthology.orchestra.run.smithi002.stdout: "num_rep": { 2023-11-16T17:00:02.297 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "num_rep", 2023-11-16T17:00:02.297 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.297 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.297 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.297 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3", 2023-11-16T17:00:02.297 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.298 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.298 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.298 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default replica count for any created local pool", 2023-11-16T17:00:02.298 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.298 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.298 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.298 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.299 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num": { 2023-11-16T17:00:02.299 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pg_num", 2023-11-16T17:00:02.299 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.299 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.299 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.299 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "128", 2023-11-16T17:00:02.299 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.300 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.300 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.300 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default pg_num for any created local pool", 2023-11-16T17:00:02.300 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.300 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.300 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.300 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.300 INFO:teuthology.orchestra.run.smithi002.stdout: "prefix": { 2023-11-16T17:00:02.300 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prefix", 2023-11-16T17:00:02.301 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.301 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.301 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.301 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.301 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.301 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.301 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.301 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "name prefix for any created local pool", 2023-11-16T17:00:02.302 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.302 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.302 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.302 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.302 INFO:teuthology.orchestra.run.smithi002.stdout: "subtree": { 2023-11-16T17:00:02.302 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subtree", 2023-11-16T17:00:02.302 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.302 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.303 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.303 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "rack", 2023-11-16T17:00:02.303 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.303 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.303 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.303 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "CRUSH level for which to create a local pool", 2023-11-16T17:00:02.303 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-11-16T17:00:02.303 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.304 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.304 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.304 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.304 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.304 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.304 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mds_autoscaler", 2023-11-16T17:00:02.304 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.304 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.305 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.305 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.305 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.305 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.305 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.305 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.305 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.305 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.305 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.306 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.306 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.306 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.306 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.306 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.306 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.306 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.306 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.306 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.307 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.307 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.307 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.307 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.307 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.307 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.307 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.307 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.308 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.308 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.308 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.308 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.308 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.308 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.308 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.308 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.308 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.309 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.309 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.309 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.309 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.309 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.309 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.309 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.309 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.310 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.310 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.310 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.310 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.310 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.310 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.310 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.310 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.311 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.311 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.311 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.311 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.311 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.311 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.311 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.311 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.311 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.312 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.312 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.312 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.312 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.312 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.312 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.312 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.312 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.312 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.313 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.313 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.313 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.313 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.313 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.313 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirroring", 2023-11-16T17:00:02.314 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.314 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.314 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.314 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.314 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.314 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.314 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.314 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.314 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.315 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.315 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.315 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.315 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.315 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.315 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.315 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.315 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.315 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.316 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.316 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.316 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.316 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.316 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.316 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.316 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.316 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.317 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.317 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.317 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.317 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.317 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.317 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.317 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.317 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.318 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.318 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.318 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.318 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.318 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.318 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.318 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.319 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.319 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.319 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.319 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.319 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.319 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.319 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.319 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.320 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.320 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.320 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.320 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.320 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.320 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.320 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.320 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.321 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.321 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.321 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.321 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.321 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.321 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.321 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.321 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.322 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.322 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.322 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.322 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.322 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.322 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.322 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.322 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.323 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.323 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.323 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "nfs", 2023-11-16T17:00:02.323 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.323 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.323 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.323 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.323 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.323 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.324 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.324 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.324 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.324 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.324 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.324 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.324 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.324 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.324 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.325 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.325 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.325 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.325 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.325 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.325 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.325 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.325 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.325 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.326 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.326 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.326 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.326 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.326 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.326 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.326 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.326 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.327 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.327 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.327 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.327 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.327 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.327 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.327 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.327 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.327 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.328 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.328 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.328 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.328 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.328 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.328 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.328 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.328 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.329 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.329 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.329 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.329 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.329 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.329 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.329 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.329 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.330 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.330 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.330 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.330 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.330 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.330 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.330 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.331 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.331 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.331 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.331 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.331 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.331 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.331 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.331 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.332 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.332 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.332 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.332 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.332 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "orchestrator", 2023-11-16T17:00:02.332 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.332 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.332 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.332 INFO:teuthology.orchestra.run.smithi002.stdout: "fail_fs": { 2023-11-16T17:00:02.333 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "fail_fs", 2023-11-16T17:00:02.333 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.333 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.333 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.333 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.333 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.333 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.333 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.334 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-11-16T17:00:02.334 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.334 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.334 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.334 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.334 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.334 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.334 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.334 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.335 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.335 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.335 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.335 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.335 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.335 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.335 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.335 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.335 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.336 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.336 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.336 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.336 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.336 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.336 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.336 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.336 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.336 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.337 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.337 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.337 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.337 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.337 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.337 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.337 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.337 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.338 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.338 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.338 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.338 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.338 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.338 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.338 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.338 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.338 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.339 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.339 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.339 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.339 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.339 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.339 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.339 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.339 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.340 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.340 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.340 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.340 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.340 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.340 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.340 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.340 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.341 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.341 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.341 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.341 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.341 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.341 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.341 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.341 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.341 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.342 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.342 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.342 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.342 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.342 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.342 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.342 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator": { 2023-11-16T17:00:02.342 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "orchestrator", 2023-11-16T17:00:02.342 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.343 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.343 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.343 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.343 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.343 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.343 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.343 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm", 2023-11-16T17:00:02.343 INFO:teuthology.orchestra.run.smithi002.stdout: "rook", 2023-11-16T17:00:02.344 INFO:teuthology.orchestra.run.smithi002.stdout: "test_orchestrator" 2023-11-16T17:00:02.344 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.344 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Orchestrator backend", 2023-11-16T17:00:02.344 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.344 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.344 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.344 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.344 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.344 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.344 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.345 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "osd_perf_query", 2023-11-16T17:00:02.345 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.345 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.345 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.345 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.345 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.345 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.345 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.346 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.346 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.346 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.346 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.346 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.346 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.346 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.346 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.346 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.347 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.347 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.347 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.347 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.347 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.347 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.347 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.347 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.347 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.348 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.348 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.348 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.348 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.348 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.348 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.348 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.348 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.348 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.349 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.349 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.349 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.349 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.349 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.349 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.349 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.349 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.350 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.350 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.350 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.350 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.350 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.350 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.350 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.350 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.351 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.351 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.351 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.351 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.351 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.351 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.351 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.351 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.351 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.352 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.352 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.352 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.352 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.352 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.352 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.352 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.352 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.352 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.353 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.353 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.353 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.353 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.353 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.353 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.353 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.353 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.353 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "osd_support", 2023-11-16T17:00:02.354 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.354 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.354 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.354 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.354 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.354 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.354 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.354 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.354 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.355 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.355 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.355 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.355 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.355 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.355 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.355 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.355 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.356 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.356 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.356 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.356 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.356 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.356 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.356 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.356 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.356 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.357 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.357 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.357 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.357 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.357 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.357 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.357 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.357 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.357 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.358 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.358 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.358 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.358 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.358 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.358 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.358 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.358 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.358 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.359 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.359 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.359 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.359 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.359 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.359 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.359 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.360 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.360 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.360 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.360 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.360 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.360 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.360 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.360 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.360 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.361 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.361 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.361 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.361 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.361 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.361 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.361 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.361 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.361 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.362 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.362 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.362 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.362 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.362 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.362 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.362 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.362 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pg_autoscaler", 2023-11-16T17:00:02.363 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.363 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.363 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.363 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.363 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.363 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.363 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.363 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.363 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.364 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.364 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.364 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.364 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.364 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.364 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.364 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.364 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.365 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.365 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.365 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.365 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.365 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.365 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.365 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.365 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.366 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.366 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.366 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.366 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.366 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.366 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.366 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.366 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.367 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.367 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.367 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.367 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.367 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.367 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.367 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.367 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.368 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.368 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.368 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.368 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.368 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.368 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.368 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.368 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.369 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.369 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.369 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.369 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.369 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.369 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.369 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.369 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.370 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.370 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.370 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.370 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.370 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.370 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.370 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.371 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.371 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.371 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.371 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.371 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.371 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.371 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.371 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.371 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.372 INFO:teuthology.orchestra.run.smithi002.stdout: "noautoscale": { 2023-11-16T17:00:02.372 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "noautoscale", 2023-11-16T17:00:02.372 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.372 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.372 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.372 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.372 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.372 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.373 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.373 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "global autoscale flag", 2023-11-16T17:00:02.373 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-11-16T17:00:02.373 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.373 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.373 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.373 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T17:00:02.373 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T17:00:02.374 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.374 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.374 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.374 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T17:00:02.374 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.374 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.374 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.374 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.375 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.375 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.375 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.375 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.375 INFO:teuthology.orchestra.run.smithi002.stdout: "threshold": { 2023-11-16T17:00:02.375 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "threshold", 2023-11-16T17:00:02.375 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:02.375 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.376 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.376 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3.0", 2023-11-16T17:00:02.376 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1.0", 2023-11-16T17:00:02.376 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.376 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.376 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "scaling threshold", 2023-11-16T17:00:02.376 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-11-16T17:00:02.376 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.376 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.377 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.377 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.377 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.377 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.377 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "progress", 2023-11-16T17:00:02.377 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.377 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.377 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.377 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_pg_recovery_event": { 2023-11-16T17:00:02.377 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_pg_recovery_event", 2023-11-16T17:00:02.378 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.378 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.378 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.378 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.378 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.378 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.378 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.378 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow the module to show pg recovery progress", 2023-11-16T17:00:02.379 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.379 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.379 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.379 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.379 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": { 2023-11-16T17:00:02.379 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enabled", 2023-11-16T17:00:02.379 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.379 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.379 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.380 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.380 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.380 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.380 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.380 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.380 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.380 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.380 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.380 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.380 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.381 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.381 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.381 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.381 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.381 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.381 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.381 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.381 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.382 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.382 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.382 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.382 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.382 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.382 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.382 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.382 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.383 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.383 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.383 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.383 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.383 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.383 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.383 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.383 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.383 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.384 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.384 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.384 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.384 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.384 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.384 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.384 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.384 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.384 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.385 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.385 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.385 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.385 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.385 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.385 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.385 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.385 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.385 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.386 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.386 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.386 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.386 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.386 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.386 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.386 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.386 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.386 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.387 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.387 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.387 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.387 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.387 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.387 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.387 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.387 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.387 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.388 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.388 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.388 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.388 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.388 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.388 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.388 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.388 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.389 INFO:teuthology.orchestra.run.smithi002.stdout: "max_completed_events": { 2023-11-16T17:00:02.389 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_completed_events", 2023-11-16T17:00:02.389 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.389 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.389 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.389 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "50", 2023-11-16T17:00:02.389 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.389 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.390 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.390 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "number of past completed events to remember", 2023-11-16T17:00:02.390 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.390 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.390 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.390 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.390 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T17:00:02.390 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T17:00:02.391 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.391 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.391 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.391 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T17:00:02.391 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.391 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.391 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.391 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long the module is going to sleep", 2023-11-16T17:00:02.391 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.392 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.392 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.392 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.392 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.392 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.392 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.392 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus", 2023-11-16T17:00:02.392 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.393 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.393 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.393 INFO:teuthology.orchestra.run.smithi002.stdout: "cache": { 2023-11-16T17:00:02.393 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cache", 2023-11-16T17:00:02.393 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.393 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.393 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.393 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.394 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.394 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.394 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.394 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.394 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.394 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.394 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.394 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.395 INFO:teuthology.orchestra.run.smithi002.stdout: "exclude_perf_counters": { 2023-11-16T17:00:02.395 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "exclude_perf_counters", 2023-11-16T17:00:02.395 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.395 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.395 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.395 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.395 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.395 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.396 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.396 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-11-16T17:00:02.396 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-11-16T17:00:02.396 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.396 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.396 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.396 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.396 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.397 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.397 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.397 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.397 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.397 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.397 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.397 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.397 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.397 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.398 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.398 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.398 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.398 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.398 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.398 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.398 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.399 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.399 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.399 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.399 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.399 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.399 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.399 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.399 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.400 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.400 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.400 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.400 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.400 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.400 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.400 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.400 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.401 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.401 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.401 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.401 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.401 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.401 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.401 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.401 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.401 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.402 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.402 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.402 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.402 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.402 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.402 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.402 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.402 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.402 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.403 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.403 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.403 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.403 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.403 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.403 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.403 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.403 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.404 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.404 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.404 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.404 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.404 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.404 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.404 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.404 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.405 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.405 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.405 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_stats_pools": { 2023-11-16T17:00:02.405 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_stats_pools", 2023-11-16T17:00:02.405 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.405 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.405 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.405 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.406 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.406 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.406 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.406 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.406 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.406 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.406 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.406 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.407 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_stats_pools_refresh_interval": { 2023-11-16T17:00:02.407 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-11-16T17:00:02.407 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.407 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.407 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.407 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "300", 2023-11-16T17:00:02.407 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.407 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.408 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.408 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.408 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.408 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.408 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.408 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.408 INFO:teuthology.orchestra.run.smithi002.stdout: "scrape_interval": { 2023-11-16T17:00:02.408 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "scrape_interval", 2023-11-16T17:00:02.408 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:02.409 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.409 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.409 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15.0", 2023-11-16T17:00:02.409 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.409 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.409 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.409 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2023-11-16T17:00:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2023-11-16T17:00:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.411 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-11-16T17:00:02.411 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.411 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.411 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.411 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-11-16T17:00:02.411 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.411 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.411 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.412 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.412 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2023-11-16T17:00:02.412 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2023-11-16T17:00:02.412 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.412 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.412 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.412 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "9283", 2023-11-16T17:00:02.412 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.412 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.413 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.413 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-11-16T17:00:02.413 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.413 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.413 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.413 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.413 INFO:teuthology.orchestra.run.smithi002.stdout: "stale_cache_strategy": { 2023-11-16T17:00:02.413 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "stale_cache_strategy", 2023-11-16T17:00:02.414 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.414 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.414 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.414 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "log", 2023-11-16T17:00:02.414 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.414 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.414 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.414 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.415 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.415 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.415 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.415 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.415 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_behaviour": { 2023-11-16T17:00:02.415 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_behaviour", 2023-11-16T17:00:02.415 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.415 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.415 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.416 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "default", 2023-11-16T17:00:02.416 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.416 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.416 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.416 INFO:teuthology.orchestra.run.smithi002.stdout: "default", 2023-11-16T17:00:02.416 INFO:teuthology.orchestra.run.smithi002.stdout: "error" 2023-11-16T17:00:02.416 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.416 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.416 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.417 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.417 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.417 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.417 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_error_status_code": { 2023-11-16T17:00:02.417 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_error_status_code", 2023-11-16T17:00:02.417 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.417 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.417 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.417 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", 2023-11-16T17:00:02.418 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-11-16T17:00:02.418 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "599", 2023-11-16T17:00:02.418 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.418 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.418 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.418 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.418 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.418 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.418 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.419 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.419 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.419 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_support", 2023-11-16T17:00:02.419 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.419 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.419 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.419 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.419 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.420 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.420 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.420 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.420 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.420 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.420 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.420 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.420 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.421 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.421 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.421 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.421 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.421 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.421 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.421 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.421 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.421 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.422 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.422 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.422 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.422 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.422 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.422 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.422 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.422 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.422 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.423 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.423 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.423 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.423 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.423 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.423 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.423 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.423 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.423 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.424 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.424 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.424 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.424 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.424 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.424 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.424 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.424 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.425 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.425 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.425 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.425 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.425 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.425 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.425 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.425 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.425 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.426 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.426 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.426 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.426 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.426 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.426 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.426 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.426 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.427 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.427 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.427 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.427 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.427 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.427 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.427 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.427 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.428 INFO:teuthology.orchestra.run.smithi002.stdout: "max_concurrent_snap_create": { 2023-11-16T17:00:02.428 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_concurrent_snap_create", 2023-11-16T17:00:02.428 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.428 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.428 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.428 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:02.428 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.428 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.428 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.429 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.429 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.429 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.429 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.429 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.429 INFO:teuthology.orchestra.run.smithi002.stdout: "mirror_snapshot_schedule": { 2023-11-16T17:00:02.429 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirror_snapshot_schedule", 2023-11-16T17:00:02.429 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.430 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.430 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.430 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.430 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.430 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.430 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.430 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.430 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.431 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.431 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.431 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.431 INFO:teuthology.orchestra.run.smithi002.stdout: "trash_purge_schedule": { 2023-11-16T17:00:02.431 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "trash_purge_schedule", 2023-11-16T17:00:02.431 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.431 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.431 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.431 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.432 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.432 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.432 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.432 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.432 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.432 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.432 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.432 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.432 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.433 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.433 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.433 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "restful", 2023-11-16T17:00:02.433 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.433 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.433 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.433 INFO:teuthology.orchestra.run.smithi002.stdout: "enable_auth": { 2023-11-16T17:00:02.433 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enable_auth", 2023-11-16T17:00:02.433 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.434 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.434 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.434 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.434 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.434 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.434 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.434 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.434 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.434 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.435 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.435 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.435 INFO:teuthology.orchestra.run.smithi002.stdout: "key_file": { 2023-11-16T17:00:02.435 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "key_file", 2023-11-16T17:00:02.435 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.435 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.435 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.435 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.435 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.436 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.436 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.436 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.436 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.436 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.436 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.436 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.436 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.436 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.437 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.437 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.437 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.437 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.437 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.437 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.437 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.437 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.437 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.437 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.438 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.438 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.438 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.438 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.438 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.438 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.438 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.438 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.439 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.439 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.439 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.439 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.439 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.439 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.439 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.439 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.439 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.440 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.440 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.440 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.440 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.440 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.440 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.440 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.440 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.440 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.441 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.441 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.441 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.441 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.441 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.441 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.441 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.441 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.441 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.442 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.442 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.442 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.442 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.442 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.442 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.442 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.442 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.442 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.443 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.443 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.443 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.443 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.443 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.443 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.443 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.443 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.443 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.444 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.444 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.444 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.444 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.444 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.444 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2023-11-16T17:00:02.444 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2023-11-16T17:00:02.444 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.445 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.445 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.445 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.445 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.445 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.445 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.445 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.445 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.445 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.446 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.446 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.446 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2023-11-16T17:00:02.446 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2023-11-16T17:00:02.446 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.446 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.446 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.446 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.447 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.447 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.447 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.447 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.447 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.447 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.447 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.447 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.447 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.448 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.448 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.448 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rgw", 2023-11-16T17:00:02.448 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.448 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.448 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.448 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.448 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.448 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.449 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.449 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.449 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.449 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.449 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.449 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.449 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.450 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.450 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.450 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.450 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.450 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.450 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.450 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.450 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.450 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.450 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.451 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.451 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.451 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.451 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.451 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.451 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.451 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.451 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.451 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.452 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.452 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.452 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.452 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.452 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.452 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.452 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.452 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.452 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.452 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.453 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.453 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.453 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.453 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.453 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.453 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.453 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.453 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.453 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.454 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.454 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.454 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.454 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.454 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.454 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.454 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.454 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.454 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.455 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.455 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.455 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.455 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.455 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.455 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.455 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.455 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.456 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.456 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.456 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.456 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.456 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.456 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.456 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.456 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.456 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rook", 2023-11-16T17:00:02.457 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.457 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.457 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.457 INFO:teuthology.orchestra.run.smithi002.stdout: "drive_group_interval": { 2023-11-16T17:00:02.457 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "drive_group_interval", 2023-11-16T17:00:02.457 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:02.457 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.457 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.457 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "300.0", 2023-11-16T17:00:02.458 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.458 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.458 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.458 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-11-16T17:00:02.458 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.458 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.458 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.458 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.459 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.459 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.459 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.459 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.459 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.459 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.459 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.459 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.460 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.460 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.460 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.460 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.460 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.460 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.460 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.460 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.461 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.461 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.461 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.461 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.461 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.461 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.461 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.461 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.462 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.462 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.462 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.462 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.462 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.462 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.462 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.462 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.462 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.462 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.463 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.463 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.463 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.463 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.463 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.463 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.463 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.463 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.463 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.464 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.464 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.464 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.464 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.464 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.464 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.464 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.464 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.464 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.464 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.465 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.465 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.465 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.465 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.465 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.465 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.465 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.465 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.465 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.466 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.466 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.466 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.466 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.466 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.466 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.466 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.466 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.466 INFO:teuthology.orchestra.run.smithi002.stdout: "storage_class": { 2023-11-16T17:00:02.467 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "storage_class", 2023-11-16T17:00:02.467 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.467 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.467 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.467 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "local", 2023-11-16T17:00:02.467 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.467 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.467 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.467 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-11-16T17:00:02.468 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.468 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.468 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.468 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.468 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.468 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.468 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.468 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "selftest", 2023-11-16T17:00:02.469 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.469 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.469 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.469 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.469 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.469 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.469 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.469 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.470 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.470 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.470 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.470 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.470 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.470 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.470 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.470 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.471 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.471 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.471 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.471 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.471 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.471 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.471 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.471 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.471 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.472 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.472 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.472 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.472 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.472 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.472 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.472 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.472 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.473 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.473 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.473 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.473 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.473 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.473 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.473 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.473 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.473 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.474 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.474 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.474 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.474 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.474 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.474 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.474 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.474 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.474 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.475 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.475 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.475 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.475 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.475 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.475 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.475 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.475 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.475 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.476 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.476 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.476 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.476 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.476 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.476 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.476 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.476 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.476 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.477 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.477 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.477 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.477 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.477 INFO:teuthology.orchestra.run.smithi002.stdout: "roption1": { 2023-11-16T17:00:02.477 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "roption1", 2023-11-16T17:00:02.477 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.477 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.478 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.478 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.478 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.478 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.478 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.478 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.478 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.478 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.478 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.479 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.479 INFO:teuthology.orchestra.run.smithi002.stdout: "roption2": { 2023-11-16T17:00:02.479 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "roption2", 2023-11-16T17:00:02.479 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.479 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.479 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.479 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "xyz", 2023-11-16T17:00:02.479 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.480 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.480 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.480 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.480 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.480 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.480 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.480 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.480 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption1": { 2023-11-16T17:00:02.480 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption1", 2023-11-16T17:00:02.481 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.481 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.481 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.481 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.481 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.481 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.481 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.481 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.481 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.482 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.482 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.482 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.482 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption2": { 2023-11-16T17:00:02.482 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption2", 2023-11-16T17:00:02.482 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.482 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.482 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.482 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.483 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.483 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.483 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.483 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.483 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.483 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.483 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.483 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.483 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption3": { 2023-11-16T17:00:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption3", 2023-11-16T17:00:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.485 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.485 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.485 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.485 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.485 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption4": { 2023-11-16T17:00:02.485 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption4", 2023-11-16T17:00:02.485 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.485 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.485 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.486 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.486 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.486 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.486 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.486 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.486 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.486 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.486 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.487 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.487 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption5": { 2023-11-16T17:00:02.487 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption5", 2023-11-16T17:00:02.487 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.487 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.487 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.487 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.487 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.487 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.488 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.488 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.488 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.488 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.488 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.488 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.488 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption6": { 2023-11-16T17:00:02.488 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption6", 2023-11-16T17:00:02.488 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.489 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.489 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.489 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.489 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.489 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.489 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.489 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.489 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.490 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.490 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.490 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.490 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption7": { 2023-11-16T17:00:02.490 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption7", 2023-11-16T17:00:02.490 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.490 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.490 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.491 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.491 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T17:00:02.491 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "42", 2023-11-16T17:00:02.491 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.491 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.491 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.491 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.491 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.491 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.492 INFO:teuthology.orchestra.run.smithi002.stdout: "testkey": { 2023-11-16T17:00:02.492 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testkey", 2023-11-16T17:00:02.492 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.492 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.492 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.492 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.492 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.492 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.492 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.493 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.493 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.493 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.493 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.493 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.493 INFO:teuthology.orchestra.run.smithi002.stdout: "testlkey": { 2023-11-16T17:00:02.493 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testlkey", 2023-11-16T17:00:02.493 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.493 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.494 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.494 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.494 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.494 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.494 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.494 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.494 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.494 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.495 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.495 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.495 INFO:teuthology.orchestra.run.smithi002.stdout: "testnewline": { 2023-11-16T17:00:02.495 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testnewline", 2023-11-16T17:00:02.495 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.495 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.495 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.495 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.495 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.496 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.496 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.496 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.496 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.496 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.496 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.496 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.496 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.497 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.497 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.497 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snap_schedule", 2023-11-16T17:00:02.497 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.497 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.497 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.497 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_m_granularity": { 2023-11-16T17:00:02.497 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_m_granularity", 2023-11-16T17:00:02.497 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.498 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.498 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.498 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.498 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.498 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.498 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.498 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow minute scheduled snapshots", 2023-11-16T17:00:02.498 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.499 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.499 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.499 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.499 INFO:teuthology.orchestra.run.smithi002.stdout: "dump_on_update": { 2023-11-16T17:00:02.499 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "dump_on_update", 2023-11-16T17:00:02.499 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.499 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.499 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.500 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.500 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.500 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.500 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.500 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "dump database to debug log on update", 2023-11-16T17:00:02.500 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.500 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.500 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.500 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.501 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.501 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.501 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.501 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.501 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.501 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.501 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.501 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.501 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.502 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.502 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.502 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.502 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.502 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.502 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.502 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.502 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.502 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.503 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.503 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.503 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.503 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.503 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.503 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.503 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.503 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.504 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.504 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.504 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.504 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.504 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.504 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.504 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.504 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.504 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.505 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.505 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.505 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.505 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.505 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.505 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.505 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.505 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.506 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.506 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.506 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.506 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.506 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.506 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.506 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.507 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.507 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.507 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.507 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.507 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.507 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.507 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.507 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.507 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.508 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.508 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.508 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.508 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.508 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.508 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.508 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.508 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.509 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.509 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.509 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.509 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.509 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.509 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.509 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "stats", 2023-11-16T17:00:02.509 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.510 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.510 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.510 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.510 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.510 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.510 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.510 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.510 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.511 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.511 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.511 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.511 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.511 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.511 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.511 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.511 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.512 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.512 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.512 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.512 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.512 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.512 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.512 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.512 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.512 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.513 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.513 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.513 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.513 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.513 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.513 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.513 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.513 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.513 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.514 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.514 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.514 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.514 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.514 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.514 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.514 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.514 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.515 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.515 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.515 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.515 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.515 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.515 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.515 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.515 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.516 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.516 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.516 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.516 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.516 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.516 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.516 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.516 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.517 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.517 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.517 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.517 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.517 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.517 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.517 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.517 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.517 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.518 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.518 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.518 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.518 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.518 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.518 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.518 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.519 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.519 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "status", 2023-11-16T17:00:02.519 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.519 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.519 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.519 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.519 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.519 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.519 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.520 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.520 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.520 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.520 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.520 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.520 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.520 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.520 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.521 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.521 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.521 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.521 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.521 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.521 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.521 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.521 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.521 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.522 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.522 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.522 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.522 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.522 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.522 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.522 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.522 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.522 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.522 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.523 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.523 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.523 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.523 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.523 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.523 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.523 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.523 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.523 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.524 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.524 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.524 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.524 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.524 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.524 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.524 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.524 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.524 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.524 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.525 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.525 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.525 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.525 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.525 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.525 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.525 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.525 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.525 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.526 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.526 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.526 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.526 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.526 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.526 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.526 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.526 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.527 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.527 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.527 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.527 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.527 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.527 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.527 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "telegraf", 2023-11-16T17:00:02.527 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.527 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.528 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.528 INFO:teuthology.orchestra.run.smithi002.stdout: "address": { 2023-11-16T17:00:02.528 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "address", 2023-11-16T17:00:02.528 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.528 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.528 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.528 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-11-16T17:00:02.528 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.529 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.529 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.529 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.529 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.529 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.529 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.529 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.529 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T17:00:02.529 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T17:00:02.530 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.530 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.530 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.530 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15", 2023-11-16T17:00:02.530 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.530 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.530 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.530 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.531 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.531 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.531 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.531 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.531 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.531 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.531 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.531 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.531 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.532 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.532 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.532 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.532 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.532 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.532 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.532 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.532 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.532 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.533 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.533 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.533 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.533 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.533 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.533 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.533 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.533 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.533 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.534 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.534 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.534 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.534 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.534 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.534 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.534 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.534 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.534 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.535 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.535 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.535 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.535 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.535 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.535 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.535 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.535 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.535 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.536 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.536 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.536 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.536 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.536 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.536 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.536 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.536 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.536 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.537 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.537 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.537 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.537 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.537 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.537 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.537 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.537 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.538 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.538 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.538 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.538 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.538 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.538 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.538 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.538 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.539 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.539 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.539 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.539 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.539 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.539 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.539 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.539 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "telemetry", 2023-11-16T17:00:02.540 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.540 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.540 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.540 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_basic": { 2023-11-16T17:00:02.540 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_basic", 2023-11-16T17:00:02.540 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.540 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.540 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.541 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.541 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.541 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.541 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.541 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share basic cluster information (size, version)", 2023-11-16T17:00:02.541 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.541 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.541 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.541 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.542 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_crash": { 2023-11-16T17:00:02.542 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_crash", 2023-11-16T17:00:02.542 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.542 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.542 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.542 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.542 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.542 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.543 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.543 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-11-16T17:00:02.543 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.543 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.543 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.543 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.543 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_device": { 2023-11-16T17:00:02.543 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_device", 2023-11-16T17:00:02.543 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.544 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.544 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.544 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.544 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.544 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.544 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.544 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-11-16T17:00:02.544 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.544 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.545 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.545 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.545 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_ident": { 2023-11-16T17:00:02.545 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_ident", 2023-11-16T17:00:02.545 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.545 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.545 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.545 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.545 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.546 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.546 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.546 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-11-16T17:00:02.546 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.546 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.546 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.546 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.546 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_perf": { 2023-11-16T17:00:02.547 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_perf", 2023-11-16T17:00:02.547 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.547 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.547 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.547 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.547 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.547 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.547 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.547 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share various performance metrics of a cluster", 2023-11-16T17:00:02.548 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.548 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.548 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.548 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.548 INFO:teuthology.orchestra.run.smithi002.stdout: "contact": { 2023-11-16T17:00:02.548 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "contact", 2023-11-16T17:00:02.548 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.548 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.548 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.549 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.549 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.549 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.549 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.549 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.549 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.549 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.550 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.550 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.550 INFO:teuthology.orchestra.run.smithi002.stdout: "description": { 2023-11-16T17:00:02.550 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "description", 2023-11-16T17:00:02.550 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.550 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.550 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.550 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.550 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.551 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.551 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.551 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.551 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.551 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.551 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.551 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.551 INFO:teuthology.orchestra.run.smithi002.stdout: "device_url": { 2023-11-16T17:00:02.551 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_url", 2023-11-16T17:00:02.551 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.552 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.552 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.552 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-11-16T17:00:02.552 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.552 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.552 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.552 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.552 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.553 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.553 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.553 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.553 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": { 2023-11-16T17:00:02.553 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enabled", 2023-11-16T17:00:02.553 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.553 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.553 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.553 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.554 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.554 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.554 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.554 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.554 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.554 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.554 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.554 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.555 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T17:00:02.555 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T17:00:02.555 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.555 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.555 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.555 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "24", 2023-11-16T17:00:02.555 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "8", 2023-11-16T17:00:02.555 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.555 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.556 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.556 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.556 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.556 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.556 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.556 INFO:teuthology.orchestra.run.smithi002.stdout: "last_opt_revision": { 2023-11-16T17:00:02.556 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "last_opt_revision", 2023-11-16T17:00:02.556 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.556 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.557 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.557 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1", 2023-11-16T17:00:02.557 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.557 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.557 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.557 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.557 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.557 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.557 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.558 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.558 INFO:teuthology.orchestra.run.smithi002.stdout: "leaderboard": { 2023-11-16T17:00:02.558 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "leaderboard", 2023-11-16T17:00:02.558 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.558 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.558 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.558 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.558 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.559 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.559 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.559 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.559 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.559 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.559 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.559 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.559 INFO:teuthology.orchestra.run.smithi002.stdout: "leaderboard_description": { 2023-11-16T17:00:02.559 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "leaderboard_description", 2023-11-16T17:00:02.560 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.560 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.560 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.560 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.560 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.560 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.560 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.560 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.561 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.561 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.561 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.561 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.561 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.561 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.561 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.561 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.561 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.562 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.562 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.562 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.562 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.562 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.562 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.562 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.562 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.563 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.563 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.563 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.563 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.563 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.563 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.563 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.563 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.563 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.564 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.564 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.564 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.564 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.564 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.564 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.564 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.564 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.564 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.565 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.565 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.565 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.565 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.565 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.565 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.565 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.565 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.565 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.566 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.566 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.566 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.566 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.566 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.566 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.566 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.566 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.566 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.567 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.567 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.567 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.567 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.567 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.567 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.567 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.567 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.567 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.568 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.568 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.568 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.568 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.568 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.568 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.568 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.569 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.569 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.569 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.569 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.569 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.569 INFO:teuthology.orchestra.run.smithi002.stdout: "organization": { 2023-11-16T17:00:02.569 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "organization", 2023-11-16T17:00:02.570 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.570 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.570 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.570 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.570 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.570 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.570 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.570 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.571 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.571 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.571 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.571 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.571 INFO:teuthology.orchestra.run.smithi002.stdout: "proxy": { 2023-11-16T17:00:02.571 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "proxy", 2023-11-16T17:00:02.571 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.571 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.571 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.572 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.572 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.572 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.572 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.572 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.572 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.572 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.572 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.572 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.573 INFO:teuthology.orchestra.run.smithi002.stdout: "url": { 2023-11-16T17:00:02.573 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "url", 2023-11-16T17:00:02.573 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.573 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.573 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.573 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-11-16T17:00:02.573 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.573 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.574 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.574 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.574 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.574 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.574 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.574 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.574 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.574 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.574 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.575 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "test_orchestrator", 2023-11-16T17:00:02.575 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.575 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.575 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.575 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.575 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.575 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.575 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.575 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.576 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.576 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.576 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.576 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.576 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.576 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.576 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.576 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.576 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.577 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.577 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.577 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.577 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.577 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.577 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.577 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.577 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.577 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.578 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.578 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.578 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.578 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.578 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.578 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.578 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.578 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.579 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.579 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.579 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.579 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.579 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.579 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.579 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.579 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.580 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.580 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.580 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.580 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.580 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.580 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.580 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.580 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.580 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.581 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.581 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.581 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.581 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.581 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.581 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.581 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.581 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.581 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.582 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.582 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.582 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.582 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.582 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.582 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.582 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.582 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.582 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.583 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.583 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.583 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.583 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.583 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.583 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.583 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.583 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "volumes", 2023-11-16T17:00:02.583 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.583 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.584 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.584 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.584 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.584 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.584 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.584 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.584 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.584 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.584 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.585 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.585 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.585 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.585 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.585 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.585 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.585 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.585 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.585 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.586 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.586 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.586 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.586 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.586 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.586 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.586 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.586 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.586 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.587 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.587 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.587 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.587 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.587 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.587 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.587 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.587 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.587 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.588 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.588 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.588 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.588 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.588 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.588 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.588 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.588 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.588 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.589 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.589 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.589 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.589 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.589 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.589 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.589 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.589 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.590 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.590 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.590 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.590 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.590 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.590 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.590 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.590 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.591 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.591 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.591 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.591 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.591 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.591 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.591 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.591 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.591 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.592 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.592 INFO:teuthology.orchestra.run.smithi002.stdout: "max_concurrent_clones": { 2023-11-16T17:00:02.592 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_concurrent_clones", 2023-11-16T17:00:02.592 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.592 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.592 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.592 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "4", 2023-11-16T17:00:02.592 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.592 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.593 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.593 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Number of asynchronous cloner threads", 2023-11-16T17:00:02.593 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.593 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.593 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.593 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.593 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot_clone_delay": { 2023-11-16T17:00:02.593 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snapshot_clone_delay", 2023-11-16T17:00:02.594 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.594 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.594 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.594 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T17:00:02.594 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.594 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.594 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.594 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-11-16T17:00:02.595 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.595 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.595 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.595 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.595 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.595 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.595 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.595 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix", 2023-11-16T17:00:02.595 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.596 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.596 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.596 INFO:teuthology.orchestra.run.smithi002.stdout: "discovery_interval": { 2023-11-16T17:00:02.596 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "discovery_interval", 2023-11-16T17:00:02.596 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T17:00:02.596 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.596 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.596 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "100", 2023-11-16T17:00:02.597 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.597 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.597 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.597 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.597 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.597 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.597 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.597 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.597 INFO:teuthology.orchestra.run.smithi002.stdout: "identifier": { 2023-11-16T17:00:02.598 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "identifier", 2023-11-16T17:00:02.598 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.598 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.598 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.598 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.598 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.598 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.599 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.599 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.599 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.599 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.599 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.599 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.599 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T17:00:02.599 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T17:00:02.600 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.600 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.600 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.600 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T17:00:02.600 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.600 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.600 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.600 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.601 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.601 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.601 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.601 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.601 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.601 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.601 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.601 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.602 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.602 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.602 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.602 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.602 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.602 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.602 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.602 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.602 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.603 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.603 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.603 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.603 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.603 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.603 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.603 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.603 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.604 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.604 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.604 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.604 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.604 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.604 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.604 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.604 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.605 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.605 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.605 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.605 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.605 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.605 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.605 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.605 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.605 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.606 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.606 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.606 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.606 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.606 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.606 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.606 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.606 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.607 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.607 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.607 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.607 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.607 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.607 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.607 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.607 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.608 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.608 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.608 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.608 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.608 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.608 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.608 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.608 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.609 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.609 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.609 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.609 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.609 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.609 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.609 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.609 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.610 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_host": { 2023-11-16T17:00:02.610 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_host", 2023-11-16T17:00:02.610 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.610 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.610 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.610 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.610 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.610 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.610 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.611 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.611 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.611 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.611 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.611 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.611 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_port": { 2023-11-16T17:00:02.611 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_port", 2023-11-16T17:00:02.611 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.611 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.612 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.612 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10051", 2023-11-16T17:00:02.612 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.612 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.612 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.612 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.612 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.612 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.613 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.613 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.613 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_sender": { 2023-11-16T17:00:02.613 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_sender", 2023-11-16T17:00:02.613 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.613 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.613 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.613 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-11-16T17:00:02.614 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.614 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.614 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.614 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.614 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.614 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.614 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.614 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.614 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.615 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.615 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T17:00:02.615 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.615 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.615 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2023-11-16T17:00:02.615 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm", 2023-11-16T17:00:02.615 INFO:teuthology.orchestra.run.smithi002.stdout: "dashboard", 2023-11-16T17:00:02.615 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2023-11-16T17:00:02.615 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2023-11-16T17:00:02.616 INFO:teuthology.orchestra.run.smithi002.stdout: "restful", 2023-11-16T17:00:02.616 INFO:teuthology.orchestra.run.smithi002.stdout: "snap_schedule" 2023-11-16T17:00:02.616 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.616 INFO:teuthology.orchestra.run.smithi002.stdout: "available_modules": [ 2023-11-16T17:00:02.616 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.616 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alerts", 2023-11-16T17:00:02.616 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.616 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.617 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.617 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T17:00:02.617 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T17:00:02.617 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.617 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.617 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.617 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T17:00:02.617 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.618 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.618 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.618 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How frequently to reexamine health status", 2023-11-16T17:00:02.618 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.618 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.618 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.618 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.618 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.618 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.619 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.619 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.619 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.619 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.619 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.619 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.619 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.619 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.620 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.620 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.620 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.620 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.620 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.620 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.620 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.620 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.620 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.621 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.621 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.621 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.621 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.621 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.621 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.621 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.621 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.621 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.622 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.622 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.622 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.622 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.622 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.622 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.622 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.622 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.623 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.623 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.623 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.623 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.623 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.623 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.623 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.623 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.623 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.624 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.624 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.624 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.624 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.624 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.624 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.624 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.624 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.624 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.625 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.625 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.625 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.625 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.625 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.625 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.625 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.625 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.625 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.626 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.626 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.626 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.626 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.626 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.626 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.626 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.626 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_destination": { 2023-11-16T17:00:02.627 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_destination", 2023-11-16T17:00:02.627 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.627 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.627 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.627 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.627 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.627 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.627 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.627 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Email address to send alerts to", 2023-11-16T17:00:02.628 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.628 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.628 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.628 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.628 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_from_name": { 2023-11-16T17:00:02.628 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_from_name", 2023-11-16T17:00:02.628 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.628 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.628 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.629 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "Ceph", 2023-11-16T17:00:02.629 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.629 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.629 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.629 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Email From: name", 2023-11-16T17:00:02.629 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.629 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.629 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.630 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.630 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_host": { 2023-11-16T17:00:02.630 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_host", 2023-11-16T17:00:02.630 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.630 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.630 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.630 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.630 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.631 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.631 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.631 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP server", 2023-11-16T17:00:02.631 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.631 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.631 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.631 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.631 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_password": { 2023-11-16T17:00:02.631 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_password", 2023-11-16T17:00:02.632 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.632 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.632 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.632 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.632 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.632 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.632 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.632 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Password to authenticate with", 2023-11-16T17:00:02.632 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.633 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.633 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.633 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.633 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_port": { 2023-11-16T17:00:02.633 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_port", 2023-11-16T17:00:02.633 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.633 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.633 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.634 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "465", 2023-11-16T17:00:02.634 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.634 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.634 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.634 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP port", 2023-11-16T17:00:02.634 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.634 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.634 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.634 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.635 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_sender": { 2023-11-16T17:00:02.635 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_sender", 2023-11-16T17:00:02.635 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.635 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.635 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.635 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.635 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.635 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.635 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.636 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP envelope sender", 2023-11-16T17:00:02.636 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.636 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.636 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.636 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.636 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_ssl": { 2023-11-16T17:00:02.636 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_ssl", 2023-11-16T17:00:02.636 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.637 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.637 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.637 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.637 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.637 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.637 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.637 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use SSL to connect to SMTP server", 2023-11-16T17:00:02.637 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.637 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.638 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.638 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.638 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_user": { 2023-11-16T17:00:02.638 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_user", 2023-11-16T17:00:02.638 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.638 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.638 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.638 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.638 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.639 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.639 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.639 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "User to authenticate as", 2023-11-16T17:00:02.639 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.639 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.639 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.639 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.639 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.640 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.640 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.640 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "balancer", 2023-11-16T17:00:02.640 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.640 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.640 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.640 INFO:teuthology.orchestra.run.smithi002.stdout: "active": { 2023-11-16T17:00:02.640 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "active", 2023-11-16T17:00:02.640 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.641 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.641 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.641 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.641 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.641 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.641 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.641 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "automatically balance PGs across cluster", 2023-11-16T17:00:02.641 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.642 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.642 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.642 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.642 INFO:teuthology.orchestra.run.smithi002.stdout: "begin_time": { 2023-11-16T17:00:02.642 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "begin_time", 2023-11-16T17:00:02.642 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.642 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.642 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.642 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0000", 2023-11-16T17:00:02.643 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.643 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.643 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.643 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "beginning time of day to automatically balance", 2023-11-16T17:00:02.643 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-16T17:00:02.643 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.643 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.643 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.643 INFO:teuthology.orchestra.run.smithi002.stdout: "begin_weekday": { 2023-11-16T17:00:02.644 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "begin_weekday", 2023-11-16T17:00:02.644 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T17:00:02.644 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.644 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.644 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T17:00:02.644 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0", 2023-11-16T17:00:02.644 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "6", 2023-11-16T17:00:02.644 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.644 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-11-16T17:00:02.645 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-16T17:00:02.645 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.645 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.645 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.645 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_max_iterations": { 2023-11-16T17:00:02.645 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_max_iterations", 2023-11-16T17:00:02.645 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T17:00:02.645 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.646 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.646 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "25", 2023-11-16T17:00:02.646 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T17:00:02.646 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "250", 2023-11-16T17:00:02.646 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.646 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-11-16T17:00:02.646 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.646 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.646 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.647 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.647 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_metrics": { 2023-11-16T17:00:02.647 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_metrics", 2023-11-16T17:00:02.647 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.647 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.647 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.647 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "pgs,objects,bytes", 2023-11-16T17:00:02.647 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.648 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.648 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.648 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-11-16T17:00:02.648 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-11-16T17:00:02.648 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.648 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.648 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.648 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_step": { 2023-11-16T17:00:02.649 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_step", 2023-11-16T17:00:02.649 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:02.649 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.649 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.649 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0.5", 2023-11-16T17:00:02.649 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0.001", 2023-11-16T17:00:02.649 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "0.999", 2023-11-16T17:00:02.649 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.650 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "aggressiveness of optimization", 2023-11-16T17:00:02.650 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-11-16T17:00:02.650 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.650 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.650 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.650 INFO:teuthology.orchestra.run.smithi002.stdout: "end_time": { 2023-11-16T17:00:02.650 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_time", 2023-11-16T17:00:02.650 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.650 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.651 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.651 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "2359", 2023-11-16T17:00:02.651 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.651 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.651 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.651 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "ending time of day to automatically balance", 2023-11-16T17:00:02.651 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-11-16T17:00:02.651 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.652 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.652 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.652 INFO:teuthology.orchestra.run.smithi002.stdout: "end_weekday": { 2023-11-16T17:00:02.652 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_weekday", 2023-11-16T17:00:02.652 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T17:00:02.652 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.652 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.652 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T17:00:02.652 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0", 2023-11-16T17:00:02.653 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "6", 2023-11-16T17:00:02.653 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.653 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-11-16T17:00:02.653 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-11-16T17:00:02.653 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.653 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.653 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.653 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.653 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.654 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.654 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.654 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.654 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.654 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.654 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.654 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.654 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.655 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.655 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.655 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.655 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.655 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.655 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.655 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.655 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.655 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.656 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.656 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.656 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.656 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.656 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.656 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.656 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.656 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.656 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.657 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.657 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.657 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.657 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.657 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.657 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.657 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.657 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.657 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.658 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.658 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.658 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.658 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.658 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.658 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.658 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.658 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.659 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.659 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.659 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.659 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.659 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.659 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.659 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.660 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.660 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.660 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.660 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.660 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.660 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.660 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.660 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.660 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.661 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.661 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.661 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.661 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.661 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.661 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.661 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.661 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.661 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.662 INFO:teuthology.orchestra.run.smithi002.stdout: "min_score": { 2023-11-16T17:00:02.662 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "min_score", 2023-11-16T17:00:02.662 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:02.662 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.662 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.662 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T17:00:02.662 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.662 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.663 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.663 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-11-16T17:00:02.663 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.663 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.663 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.663 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.663 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": { 2023-11-16T17:00:02.663 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mode", 2023-11-16T17:00:02.663 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.664 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.664 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.664 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "upmap", 2023-11-16T17:00:02.664 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.664 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.664 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.664 INFO:teuthology.orchestra.run.smithi002.stdout: "crush-compat", 2023-11-16T17:00:02.664 INFO:teuthology.orchestra.run.smithi002.stdout: "none", 2023-11-16T17:00:02.664 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap" 2023-11-16T17:00:02.665 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.665 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Balancer mode", 2023-11-16T17:00:02.665 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.665 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.665 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.665 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.665 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_ids": { 2023-11-16T17:00:02.665 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pool_ids", 2023-11-16T17:00:02.666 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.666 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.666 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.666 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.666 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.666 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.666 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.666 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-11-16T17:00:02.666 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.667 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.667 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.667 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.667 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T17:00:02.667 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T17:00:02.667 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.667 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.667 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.667 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T17:00:02.668 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.668 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.668 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.668 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-11-16T17:00:02.668 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.668 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.668 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.668 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.669 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap_max_deviation": { 2023-11-16T17:00:02.669 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "upmap_max_deviation", 2023-11-16T17:00:02.669 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.669 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.669 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.669 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T17:00:02.669 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T17:00:02.669 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.670 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.670 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "deviation below which no optimization is attempted", 2023-11-16T17:00:02.670 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-11-16T17:00:02.670 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.670 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.670 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.670 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap_max_optimizations": { 2023-11-16T17:00:02.670 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "upmap_max_optimizations", 2023-11-16T17:00:02.671 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T17:00:02.671 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.671 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.671 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:02.671 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.671 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.671 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.671 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-11-16T17:00:02.671 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.672 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.672 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.672 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.672 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.672 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.672 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.672 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephadm", 2023-11-16T17:00:02.672 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.672 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.673 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.673 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_down_multiplier": { 2023-11-16T17:00:02.673 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_down_multiplier", 2023-11-16T17:00:02.673 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:02.673 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.673 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.673 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3.0", 2023-11-16T17:00:02.673 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.674 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.674 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.674 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-11-16T17:00:02.674 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.674 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.674 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.674 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.674 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_refresh_rate": { 2023-11-16T17:00:02.674 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_refresh_rate", 2023-11-16T17:00:02.675 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.675 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.675 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.675 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "20", 2023-11-16T17:00:02.675 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.675 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.675 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.675 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-11-16T17:00:02.675 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.676 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.676 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.676 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.676 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_starting_port": { 2023-11-16T17:00:02.676 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_starting_port", 2023-11-16T17:00:02.676 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.676 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.676 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.677 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "4721", 2023-11-16T17:00:02.677 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.677 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.677 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.677 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-11-16T17:00:02.677 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.677 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.677 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.677 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.678 INFO:teuthology.orchestra.run.smithi002.stdout: "alertmanager_web_password": { 2023-11-16T17:00:02.678 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alertmanager_web_password", 2023-11-16T17:00:02.678 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.678 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.678 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.678 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:02.678 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.679 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.679 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.679 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Alertmanager web password", 2023-11-16T17:00:02.679 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.679 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.679 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.679 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.679 INFO:teuthology.orchestra.run.smithi002.stdout: "alertmanager_web_user": { 2023-11-16T17:00:02.680 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alertmanager_web_user", 2023-11-16T17:00:02.680 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.680 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.680 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.680 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:02.680 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.680 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.680 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.681 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Alertmanager web user", 2023-11-16T17:00:02.681 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.681 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.681 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.681 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.681 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_ptrace": { 2023-11-16T17:00:02.681 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_ptrace", 2023-11-16T17:00:02.681 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.681 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.682 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.682 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.682 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.682 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.682 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.682 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-11-16T17:00:02.682 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-11-16T17:00:02.682 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.683 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.683 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.683 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_interval": { 2023-11-16T17:00:02.683 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_interval", 2023-11-16T17:00:02.683 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.683 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.683 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.683 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T17:00:02.683 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.684 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.684 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.684 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to autotune daemon memory", 2023-11-16T17:00:02.684 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.684 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.684 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.684 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.684 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_memory_target_ratio": { 2023-11-16T17:00:02.685 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_memory_target_ratio", 2023-11-16T17:00:02.685 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:02.685 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.685 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.685 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0.7", 2023-11-16T17:00:02.685 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.685 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.685 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.685 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-11-16T17:00:02.686 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.686 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.686 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.686 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.686 INFO:teuthology.orchestra.run.smithi002.stdout: "cgroups_split": { 2023-11-16T17:00:02.686 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cgroups_split", 2023-11-16T17:00:02.686 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.686 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.686 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.687 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.687 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.687 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.687 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.687 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-11-16T17:00:02.687 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.687 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.687 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.688 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.688 INFO:teuthology.orchestra.run.smithi002.stdout: "config_checks_enabled": { 2023-11-16T17:00:02.688 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_checks_enabled", 2023-11-16T17:00:02.688 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.688 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.688 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.688 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.688 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.688 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.689 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.689 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-11-16T17:00:02.689 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.689 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.689 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.689 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.689 INFO:teuthology.orchestra.run.smithi002.stdout: "config_dashboard": { 2023-11-16T17:00:02.690 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_dashboard", 2023-11-16T17:00:02.690 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.690 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.690 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.690 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.690 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.690 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.690 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.691 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-11-16T17:00:02.691 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.691 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.691 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.691 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.691 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_alertmanager": { 2023-11-16T17:00:02.691 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_alertmanager", 2023-11-16T17:00:02.691 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.692 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.692 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.692 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-11-16T17:00:02.692 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.692 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.692 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.692 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:02.692 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.692 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.693 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.693 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.693 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_base": { 2023-11-16T17:00:02.693 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_base", 2023-11-16T17:00:02.693 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.693 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.693 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.693 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/ceph", 2023-11-16T17:00:02.694 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.694 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.694 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.694 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Container image name, without the tag", 2023-11-16T17:00:02.694 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.694 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.694 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.694 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.695 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_elasticsearch": { 2023-11-16T17:00:02.695 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_elasticsearch", 2023-11-16T17:00:02.695 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.695 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.695 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.695 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-11-16T17:00:02.695 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.695 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.696 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.696 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "elasticsearch container image", 2023-11-16T17:00:02.696 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.696 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.696 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.696 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.696 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_grafana": { 2023-11-16T17:00:02.696 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_grafana", 2023-11-16T17:00:02.697 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.697 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.697 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.697 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-11-16T17:00:02.697 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.697 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.697 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.697 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:02.697 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.698 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.698 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.698 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.698 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_haproxy": { 2023-11-16T17:00:02.698 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_haproxy", 2023-11-16T17:00:02.698 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.698 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.699 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.699 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-11-16T17:00:02.699 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.699 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.699 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.699 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "HAproxy container image", 2023-11-16T17:00:02.699 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.699 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.700 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.700 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.700 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_agent": { 2023-11-16T17:00:02.700 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_agent", 2023-11-16T17:00:02.700 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.700 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.700 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.700 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-11-16T17:00:02.701 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.701 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.701 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.701 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger agent container image", 2023-11-16T17:00:02.701 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.701 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.701 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.701 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.702 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_collector": { 2023-11-16T17:00:02.702 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_collector", 2023-11-16T17:00:02.702 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.702 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.702 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.702 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-11-16T17:00:02.702 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.702 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.702 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.703 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger collector container image", 2023-11-16T17:00:02.703 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.703 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.703 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.703 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.703 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_query": { 2023-11-16T17:00:02.703 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_query", 2023-11-16T17:00:02.703 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.704 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.704 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.704 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-11-16T17:00:02.704 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.704 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.704 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.704 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger query container image", 2023-11-16T17:00:02.704 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.705 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.705 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.705 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.705 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_keepalived": { 2023-11-16T17:00:02.705 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_keepalived", 2023-11-16T17:00:02.705 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.705 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.705 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.705 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-11-16T17:00:02.706 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.706 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.706 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.706 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Keepalived container image", 2023-11-16T17:00:02.706 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.706 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.706 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.706 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.706 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_loki": { 2023-11-16T17:00:02.707 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_loki", 2023-11-16T17:00:02.707 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.707 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.707 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.707 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-11-16T17:00:02.707 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.707 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.707 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.708 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Loki container image", 2023-11-16T17:00:02.708 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.708 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.708 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.708 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.708 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_node_exporter": { 2023-11-16T17:00:02.708 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_node_exporter", 2023-11-16T17:00:02.708 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.708 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.709 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.709 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-11-16T17:00:02.709 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.709 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.709 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.709 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:02.709 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.710 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.710 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.710 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.710 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_prometheus": { 2023-11-16T17:00:02.710 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_prometheus", 2023-11-16T17:00:02.710 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.710 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.710 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.710 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-11-16T17:00:02.711 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.711 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.711 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.711 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2023-11-16T17:00:02.711 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.711 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.711 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.711 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.712 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_promtail": { 2023-11-16T17:00:02.712 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_promtail", 2023-11-16T17:00:02.712 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.712 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.712 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.712 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-11-16T17:00:02.712 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.712 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.712 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.713 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Promtail container image", 2023-11-16T17:00:02.713 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.713 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.713 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.713 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.713 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_snmp_gateway": { 2023-11-16T17:00:02.713 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_snmp_gateway", 2023-11-16T17:00:02.713 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.713 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.714 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.714 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-11-16T17:00:02.714 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.714 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.714 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.714 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SNMP Gateway container image", 2023-11-16T17:00:02.714 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.714 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.715 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.715 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.715 INFO:teuthology.orchestra.run.smithi002.stdout: "container_init": { 2023-11-16T17:00:02.715 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_init", 2023-11-16T17:00:02.715 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.715 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.715 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.715 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.715 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.716 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.716 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.716 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Run podman/docker with `--init`", 2023-11-16T17:00:02.716 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.716 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.716 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.716 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.716 INFO:teuthology.orchestra.run.smithi002.stdout: "daemon_cache_timeout": { 2023-11-16T17:00:02.716 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "daemon_cache_timeout", 2023-11-16T17:00:02.717 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.717 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.717 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.717 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T17:00:02.717 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.717 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.717 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.717 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-11-16T17:00:02.717 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.718 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.718 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.718 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.718 INFO:teuthology.orchestra.run.smithi002.stdout: "default_cephadm_command_timeout": { 2023-11-16T17:00:02.718 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default_cephadm_command_timeout", 2023-11-16T17:00:02.718 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.718 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.718 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.719 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "900", 2023-11-16T17:00:02.719 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.719 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.719 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.719 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-11-16T17:00:02.719 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.719 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.719 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.720 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.720 INFO:teuthology.orchestra.run.smithi002.stdout: "default_registry": { 2023-11-16T17:00:02.720 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default_registry", 2023-11-16T17:00:02.720 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.720 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.720 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.720 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io", 2023-11-16T17:00:02.720 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.720 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.721 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.721 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-11-16T17:00:02.721 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.721 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.721 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.721 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.721 INFO:teuthology.orchestra.run.smithi002.stdout: "device_cache_timeout": { 2023-11-16T17:00:02.721 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_cache_timeout", 2023-11-16T17:00:02.721 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.722 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.722 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.722 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1800", 2023-11-16T17:00:02.722 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.722 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.722 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.722 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache device inventory", 2023-11-16T17:00:02.722 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.723 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.723 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.723 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.723 INFO:teuthology.orchestra.run.smithi002.stdout: "device_enhanced_scan": { 2023-11-16T17:00:02.723 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_enhanced_scan", 2023-11-16T17:00:02.723 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.723 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.723 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.723 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.724 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.724 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.724 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.724 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use libstoragemgmt during device scans", 2023-11-16T17:00:02.724 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.724 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.724 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.724 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.724 INFO:teuthology.orchestra.run.smithi002.stdout: "facts_cache_timeout": { 2023-11-16T17:00:02.725 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "facts_cache_timeout", 2023-11-16T17:00:02.725 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.725 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.725 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.725 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T17:00:02.725 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.725 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.725 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.725 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache host facts data", 2023-11-16T17:00:02.726 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.726 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.726 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.726 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.726 INFO:teuthology.orchestra.run.smithi002.stdout: "host_check_interval": { 2023-11-16T17:00:02.726 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "host_check_interval", 2023-11-16T17:00:02.726 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.726 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.727 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.727 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T17:00:02.727 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.727 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.727 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.727 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to perform a host check", 2023-11-16T17:00:02.727 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.727 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.727 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.728 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.728 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.728 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.728 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.728 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.728 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.728 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.728 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.728 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.729 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.729 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.729 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.729 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.729 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.729 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.729 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.729 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.730 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.730 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.730 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.730 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.730 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.730 INFO:teuthology.orchestra.run.smithi002.stdout: "log_refresh_metadata": { 2023-11-16T17:00:02.730 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_refresh_metadata", 2023-11-16T17:00:02.731 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.731 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.731 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.731 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.731 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.731 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.731 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.731 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-11-16T17:00:02.732 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.732 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.732 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.732 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.732 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.732 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.732 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.732 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.732 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.733 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.733 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.733 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.733 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.733 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-11-16T17:00:02.733 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.733 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.733 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.734 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.734 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.734 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.734 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.734 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.734 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.734 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.734 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.735 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.735 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.735 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.735 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.735 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.735 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.735 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.735 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.735 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.736 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.736 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.736 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.736 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.736 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.736 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.736 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.736 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.736 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.737 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.737 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.737 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.737 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.737 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.737 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.737 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.737 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.738 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.738 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.738 INFO:teuthology.orchestra.run.smithi002.stdout: "manage_etc_ceph_ceph_conf": { 2023-11-16T17:00:02.738 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-11-16T17:00:02.738 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.738 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.738 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.738 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.739 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.739 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.739 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.739 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-11-16T17:00:02.739 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.739 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.739 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.739 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.740 INFO:teuthology.orchestra.run.smithi002.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-11-16T17:00:02.740 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-11-16T17:00:02.740 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.740 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.740 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.740 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "*", 2023-11-16T17:00:02.740 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.740 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.741 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.741 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-11-16T17:00:02.741 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.741 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.741 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.741 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.741 INFO:teuthology.orchestra.run.smithi002.stdout: "max_count_per_host": { 2023-11-16T17:00:02.741 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_count_per_host", 2023-11-16T17:00:02.742 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.742 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.742 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.742 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:02.742 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.742 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.742 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.742 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "max number of daemons per service per host", 2023-11-16T17:00:02.743 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.743 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.743 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.743 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.743 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd_draining_count": { 2023-11-16T17:00:02.743 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_osd_draining_count", 2023-11-16T17:00:02.743 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.743 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.743 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.744 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:02.744 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.744 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.744 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.744 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-11-16T17:00:02.744 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.744 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.744 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.744 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.745 INFO:teuthology.orchestra.run.smithi002.stdout: "migration_current": { 2023-11-16T17:00:02.745 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "migration_current", 2023-11-16T17:00:02.745 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.745 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.745 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.745 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.745 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.745 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.745 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.746 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "internal - do not modify", 2023-11-16T17:00:02.746 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.746 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.746 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.746 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.746 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": { 2023-11-16T17:00:02.746 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mode", 2023-11-16T17:00:02.746 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.746 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.747 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.747 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "root", 2023-11-16T17:00:02.747 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.747 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.747 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.747 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm-package", 2023-11-16T17:00:02.747 INFO:teuthology.orchestra.run.smithi002.stdout: "root" 2023-11-16T17:00:02.747 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.747 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "mode for remote execution of cephadm", 2023-11-16T17:00:02.747 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.748 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.748 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.748 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.748 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_alerts_path": { 2023-11-16T17:00:02.748 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_alerts_path", 2023-11-16T17:00:02.748 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.748 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.748 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.749 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-11-16T17:00:02.749 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.749 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.749 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.749 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-11-16T17:00:02.749 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.749 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.750 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.750 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.750 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_web_password": { 2023-11-16T17:00:02.750 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_web_password", 2023-11-16T17:00:02.750 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.750 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.750 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.750 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:02.751 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.751 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.751 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.751 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus web password", 2023-11-16T17:00:02.751 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.751 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.751 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.751 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.751 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_web_user": { 2023-11-16T17:00:02.752 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_web_user", 2023-11-16T17:00:02.752 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.752 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.752 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.752 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:02.752 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.752 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.752 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.752 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus web user", 2023-11-16T17:00:02.753 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.753 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.753 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.753 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.753 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_insecure": { 2023-11-16T17:00:02.753 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_insecure", 2023-11-16T17:00:02.753 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.753 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.754 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.754 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.754 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.754 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.754 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.754 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-11-16T17:00:02.754 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.754 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.754 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.755 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.755 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_password": { 2023-11-16T17:00:02.755 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_password", 2023-11-16T17:00:02.755 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.755 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.755 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.755 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.755 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.755 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.756 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.756 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-11-16T17:00:02.756 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.756 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.756 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.756 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.756 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_url": { 2023-11-16T17:00:02.756 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_url", 2023-11-16T17:00:02.756 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.757 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.757 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.757 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.757 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.757 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.757 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.757 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-11-16T17:00:02.757 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.758 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.758 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.758 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.758 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_username": { 2023-11-16T17:00:02.758 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_username", 2023-11-16T17:00:02.758 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.758 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.759 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.759 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.759 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.759 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.759 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.759 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-11-16T17:00:02.759 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.759 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.760 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.760 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.760 INFO:teuthology.orchestra.run.smithi002.stdout: "secure_monitoring_stack": { 2023-11-16T17:00:02.760 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "secure_monitoring_stack", 2023-11-16T17:00:02.760 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.760 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.760 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.760 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.760 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.761 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.761 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.761 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-11-16T17:00:02.761 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.761 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.761 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.761 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.761 INFO:teuthology.orchestra.run.smithi002.stdout: "service_discovery_port": { 2023-11-16T17:00:02.761 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "service_discovery_port", 2023-11-16T17:00:02.762 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.762 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.762 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.762 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8765", 2023-11-16T17:00:02.762 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.762 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.762 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.762 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "cephadm service discovery port", 2023-11-16T17:00:02.762 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.763 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.763 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.763 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.763 INFO:teuthology.orchestra.run.smithi002.stdout: "ssh_config_file": { 2023-11-16T17:00:02.763 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssh_config_file", 2023-11-16T17:00:02.763 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.763 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.763 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.764 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.764 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.764 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.764 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.764 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-11-16T17:00:02.764 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.764 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.764 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.764 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.765 INFO:teuthology.orchestra.run.smithi002.stdout: "use_agent": { 2023-11-16T17:00:02.765 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "use_agent", 2023-11-16T17:00:02.765 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.765 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.765 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.765 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.765 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.765 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.765 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.766 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-11-16T17:00:02.766 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.766 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.766 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.766 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.766 INFO:teuthology.orchestra.run.smithi002.stdout: "use_repo_digest": { 2023-11-16T17:00:02.766 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "use_repo_digest", 2023-11-16T17:00:02.766 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.766 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.767 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.767 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.767 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.767 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.767 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.767 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-11-16T17:00:02.767 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.768 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.768 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.768 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.768 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_failed_host_check": { 2023-11-16T17:00:02.768 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_failed_host_check", 2023-11-16T17:00:02.768 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.768 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.768 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.769 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.769 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.769 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.769 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.769 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if the host check fails", 2023-11-16T17:00:02.769 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.769 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.769 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.770 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.770 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_stray_daemons": { 2023-11-16T17:00:02.770 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_stray_daemons", 2023-11-16T17:00:02.770 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.770 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.770 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.770 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.770 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.771 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.771 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.771 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-11-16T17:00:02.771 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.771 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.771 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.771 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.771 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_stray_hosts": { 2023-11-16T17:00:02.772 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_stray_hosts", 2023-11-16T17:00:02.772 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.772 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.772 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.772 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.772 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.772 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.772 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.773 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-11-16T17:00:02.773 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.773 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.773 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.773 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.773 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.773 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.773 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.773 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crash", 2023-11-16T17:00:02.774 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.774 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.774 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.774 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.774 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.774 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.774 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.774 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.775 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.775 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.775 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.775 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.775 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.775 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.775 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.775 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.776 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.776 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.776 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.776 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.776 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.776 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.776 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.776 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.776 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.777 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.777 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.777 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.777 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.777 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.777 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.777 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.777 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.778 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.778 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.778 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.778 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.778 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.778 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.778 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.778 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.779 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.779 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.779 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.779 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.779 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.779 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.779 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.780 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.780 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.780 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.780 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.780 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.780 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.780 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.780 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.781 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.781 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.781 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.781 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.781 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.781 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.781 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.781 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.782 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.782 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.782 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.782 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.782 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.782 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.782 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.782 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.783 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.783 INFO:teuthology.orchestra.run.smithi002.stdout: "retain_interval": { 2023-11-16T17:00:02.783 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retain_interval", 2023-11-16T17:00:02.783 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.783 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.783 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.783 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "31536000", 2023-11-16T17:00:02.783 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.784 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.784 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.784 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long to retain crashes before pruning them", 2023-11-16T17:00:02.784 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.784 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.784 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.784 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.784 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_recent_interval": { 2023-11-16T17:00:02.784 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_recent_interval", 2023-11-16T17:00:02.785 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.785 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.785 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.785 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1209600", 2023-11-16T17:00:02.785 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.785 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.785 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.785 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "time interval in which to warn about recent crashes", 2023-11-16T17:00:02.786 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.786 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.786 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.786 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.786 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.786 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.786 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.786 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "dashboard", 2023-11-16T17:00:02.786 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.787 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.787 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.787 INFO:teuthology.orchestra.run.smithi002.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-11-16T17:00:02.787 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-11-16T17:00:02.787 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.787 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.787 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.787 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:02.787 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.788 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.788 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.788 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.788 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.788 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.788 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.788 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.788 INFO:teuthology.orchestra.run.smithi002.stdout: "ALERTMANAGER_API_HOST": { 2023-11-16T17:00:02.789 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ALERTMANAGER_API_HOST", 2023-11-16T17:00:02.789 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.789 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.789 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.789 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.789 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.789 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.789 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.790 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.790 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.790 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.790 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.790 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.790 INFO:teuthology.orchestra.run.smithi002.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-11-16T17:00:02.790 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-11-16T17:00:02.791 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.791 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.791 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.791 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.791 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.791 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.791 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.791 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.791 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.792 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.792 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.792 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.792 INFO:teuthology.orchestra.run.smithi002.stdout: "AUDIT_API_ENABLED": { 2023-11-16T17:00:02.792 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "AUDIT_API_ENABLED", 2023-11-16T17:00:02.792 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.792 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.792 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.792 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.793 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.793 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.793 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.793 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.793 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.793 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.793 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.793 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.793 INFO:teuthology.orchestra.run.smithi002.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-11-16T17:00:02.794 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-11-16T17:00:02.794 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.794 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.794 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.794 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.794 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.794 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.795 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.795 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.795 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.795 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.795 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.795 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.795 INFO:teuthology.orchestra.run.smithi002.stdout: "ENABLE_BROWSABLE_API": { 2023-11-16T17:00:02.795 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ENABLE_BROWSABLE_API", 2023-11-16T17:00:02.795 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.796 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.796 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.796 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.796 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.796 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.796 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.796 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.796 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.797 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.797 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.797 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.797 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-11-16T17:00:02.797 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-11-16T17:00:02.797 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.797 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.797 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.797 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.798 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.798 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.798 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.798 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.798 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.798 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.798 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.798 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.799 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-11-16T17:00:02.799 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-11-16T17:00:02.799 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.799 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.799 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.799 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.799 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.800 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.800 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.800 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.800 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.800 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.800 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.800 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.800 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-11-16T17:00:02.800 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-11-16T17:00:02.801 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.801 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.801 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.801 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.801 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.801 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.801 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.801 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.801 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.802 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.802 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.802 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.802 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-11-16T17:00:02.802 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-11-16T17:00:02.802 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.802 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.802 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.802 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.803 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.803 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.803 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.803 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.803 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.803 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.803 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.803 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.804 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_NFS": { 2023-11-16T17:00:02.804 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-11-16T17:00:02.804 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.804 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.804 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.804 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.804 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.804 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.804 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.805 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.805 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.805 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.805 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.805 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.805 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RBD": { 2023-11-16T17:00:02.805 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-11-16T17:00:02.805 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.805 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.806 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.806 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.806 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.806 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.806 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.806 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.806 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.806 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.806 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.807 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.807 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RGW": { 2023-11-16T17:00:02.807 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-11-16T17:00:02.807 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.807 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.807 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.807 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.807 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.808 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.808 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.808 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.808 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.808 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.808 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.808 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.808 INFO:teuthology.orchestra.run.smithi002.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-11-16T17:00:02.808 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-11-16T17:00:02.809 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.809 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.809 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.809 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.809 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.809 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.809 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.809 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.810 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.810 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.810 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.810 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.810 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_PASSWORD": { 2023-11-16T17:00:02.810 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_PASSWORD", 2023-11-16T17:00:02.810 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.810 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.811 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.811 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:02.811 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.811 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.811 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.811 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.811 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.811 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.811 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.812 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.812 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-11-16T17:00:02.812 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-11-16T17:00:02.812 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.812 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.812 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.812 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.812 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.812 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.813 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.813 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.813 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.813 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.813 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.813 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.813 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_URL": { 2023-11-16T17:00:02.813 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_URL", 2023-11-16T17:00:02.813 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.814 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.814 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.814 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.814 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.814 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.814 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.814 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.814 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.815 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.815 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.815 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.815 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_USERNAME": { 2023-11-16T17:00:02.815 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_USERNAME", 2023-11-16T17:00:02.815 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.815 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.815 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.815 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:02.816 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.816 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.816 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.816 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.816 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.816 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.816 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.816 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.817 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-11-16T17:00:02.817 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-11-16T17:00:02.817 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.817 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.817 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.817 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.817 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.817 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.817 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.818 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.818 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.818 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.818 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.818 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.818 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-11-16T17:00:02.818 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-11-16T17:00:02.818 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.819 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.819 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.819 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.819 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.819 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.819 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.819 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.819 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.819 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.820 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.820 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.820 INFO:teuthology.orchestra.run.smithi002.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-11-16T17:00:02.820 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-11-16T17:00:02.820 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.820 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.820 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.820 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.821 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.821 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.821 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.821 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.821 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.821 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.821 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.821 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.821 INFO:teuthology.orchestra.run.smithi002.stdout: "ISSUE_TRACKER_API_KEY": { 2023-11-16T17:00:02.822 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-11-16T17:00:02.822 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.822 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.822 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.822 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.822 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.822 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.822 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.822 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.823 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.823 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.823 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.823 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.823 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_HOST": { 2023-11-16T17:00:02.823 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_HOST", 2023-11-16T17:00:02.823 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.823 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.823 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.824 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.824 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.824 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.824 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.824 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.824 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.824 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.824 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.825 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.825 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-11-16T17:00:02.825 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-11-16T17:00:02.825 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.825 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.825 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.825 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.825 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.825 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.826 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.826 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.826 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.826 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.826 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.826 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.826 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-11-16T17:00:02.826 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-11-16T17:00:02.826 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.827 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.827 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.827 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.827 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.827 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.827 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.827 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.827 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.827 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.828 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.828 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.828 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-11-16T17:00:02.828 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-11-16T17:00:02.828 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.828 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.828 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.828 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.829 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.829 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.829 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.829 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.829 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.829 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.829 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.829 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.830 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-11-16T17:00:02.830 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-11-16T17:00:02.830 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.830 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.830 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.830 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.830 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.830 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.831 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.831 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.831 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.831 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.831 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.831 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.831 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-11-16T17:00:02.831 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-11-16T17:00:02.831 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.832 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.832 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.832 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.832 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.832 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.832 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.832 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.832 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.833 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.833 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.833 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.833 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-11-16T17:00:02.833 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-11-16T17:00:02.833 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.833 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.833 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.833 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.834 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.834 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.834 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.834 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.834 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.834 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.834 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.834 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.835 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-11-16T17:00:02.835 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-11-16T17:00:02.835 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.835 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.835 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.835 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.835 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.835 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.836 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.836 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.836 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.836 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.836 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.836 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.836 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-11-16T17:00:02.836 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-11-16T17:00:02.837 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.837 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.837 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.837 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.837 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.837 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.837 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.837 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.837 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.838 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.838 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.838 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.838 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_ENABLED": { 2023-11-16T17:00:02.838 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_ENABLED", 2023-11-16T17:00:02.838 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.838 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.838 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.839 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.839 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.839 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.839 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.839 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.839 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.839 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.840 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.840 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.840 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-11-16T17:00:02.840 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-11-16T17:00:02.840 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.840 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.840 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.840 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-11-16T17:00:02.841 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.841 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.841 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.841 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.841 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.841 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.841 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.841 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.841 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-11-16T17:00:02.842 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-11-16T17:00:02.842 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.842 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.842 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.842 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:02.842 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.842 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.842 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.843 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.843 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.843 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.843 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.843 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.843 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-11-16T17:00:02.843 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-11-16T17:00:02.843 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.843 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.844 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.844 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8", 2023-11-16T17:00:02.844 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.844 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.844 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.844 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.844 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.844 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.845 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.845 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.845 INFO:teuthology.orchestra.run.smithi002.stdout: "REST_REQUESTS_TIMEOUT": { 2023-11-16T17:00:02.845 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-11-16T17:00:02.845 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.845 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.845 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.845 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "45", 2023-11-16T17:00:02.845 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.846 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.846 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.846 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.846 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.846 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.846 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.846 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.846 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_ACCESS_KEY": { 2023-11-16T17:00:02.847 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_ACCESS_KEY", 2023-11-16T17:00:02.847 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.847 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.847 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.847 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.847 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.847 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.847 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.847 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.848 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.848 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.848 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.848 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.848 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-11-16T17:00:02.848 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-11-16T17:00:02.848 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.848 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.848 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.849 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2023-11-16T17:00:02.849 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.849 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.849 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.849 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.849 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.849 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.850 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.850 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.850 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_SECRET_KEY": { 2023-11-16T17:00:02.850 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_SECRET_KEY", 2023-11-16T17:00:02.850 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.850 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.850 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.850 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.850 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.851 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.851 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.851 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.851 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.851 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.851 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.851 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.851 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_SSL_VERIFY": { 2023-11-16T17:00:02.852 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_SSL_VERIFY", 2023-11-16T17:00:02.852 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.852 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.852 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.852 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.852 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.852 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.852 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.852 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.853 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.853 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.853 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.853 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.853 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-11-16T17:00:02.853 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-11-16T17:00:02.853 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.853 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.853 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.854 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T17:00:02.854 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.854 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.854 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.854 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.854 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.854 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.854 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.855 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.855 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-11-16T17:00:02.855 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-11-16T17:00:02.855 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.855 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.855 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.855 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:02.855 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.855 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.856 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.856 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.856 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.856 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.856 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.856 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.856 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-11-16T17:00:02.856 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-11-16T17:00:02.857 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.857 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.857 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.857 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T17:00:02.857 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.857 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.857 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.857 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.857 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.858 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.858 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.858 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.858 INFO:teuthology.orchestra.run.smithi002.stdout: "cross_origin_url": { 2023-11-16T17:00:02.858 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cross_origin_url", 2023-11-16T17:00:02.858 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.858 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.858 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.858 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.859 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.859 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.859 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.859 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.859 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.859 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.859 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.860 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.860 INFO:teuthology.orchestra.run.smithi002.stdout: "crt_file": { 2023-11-16T17:00:02.860 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crt_file", 2023-11-16T17:00:02.860 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.860 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.860 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.860 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.860 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.860 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.861 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.861 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.861 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.861 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.861 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.861 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.861 INFO:teuthology.orchestra.run.smithi002.stdout: "debug": { 2023-11-16T17:00:02.861 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "debug", 2023-11-16T17:00:02.861 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.862 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.862 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.862 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.862 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.862 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.862 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.862 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable/disable debug options", 2023-11-16T17:00:02.862 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.863 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.863 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.863 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.863 INFO:teuthology.orchestra.run.smithi002.stdout: "jwt_token_ttl": { 2023-11-16T17:00:02.863 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "jwt_token_ttl", 2023-11-16T17:00:02.863 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.863 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.863 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.863 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "28800", 2023-11-16T17:00:02.864 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.864 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.864 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.864 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.864 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.864 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.864 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.864 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.864 INFO:teuthology.orchestra.run.smithi002.stdout: "key_file": { 2023-11-16T17:00:02.865 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "key_file", 2023-11-16T17:00:02.865 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.865 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.865 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.865 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.865 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.865 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.865 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.865 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.866 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.866 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.866 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.866 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.866 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.866 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.866 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.866 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.867 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.867 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.867 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.867 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.867 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.867 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.867 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.867 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.867 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.868 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.868 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.868 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.868 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.868 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.868 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.868 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.868 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.869 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.869 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.869 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.869 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.869 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.869 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.869 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.870 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.870 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.870 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.870 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.870 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.870 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.870 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.870 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.871 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.871 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.871 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.871 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.871 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.871 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.871 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.871 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.871 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.872 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.872 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.872 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.872 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.872 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.872 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.872 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.872 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.873 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.873 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.873 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.873 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.873 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.873 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.873 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.873 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.873 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.874 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.874 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.874 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.874 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.874 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.874 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.874 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.874 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.874 INFO:teuthology.orchestra.run.smithi002.stdout: "motd": { 2023-11-16T17:00:02.875 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "motd", 2023-11-16T17:00:02.875 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.875 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.875 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.875 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.875 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.875 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.875 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.876 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "The message of the day", 2023-11-16T17:00:02.876 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.876 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.876 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.876 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.876 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect_resolve_ip_addr": { 2023-11-16T17:00:02.876 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "redirect_resolve_ip_addr", 2023-11-16T17:00:02.876 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.876 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.877 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.877 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.877 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.877 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.877 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.877 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.877 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.877 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.877 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.878 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.878 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2023-11-16T17:00:02.878 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2023-11-16T17:00:02.878 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.878 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.878 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.878 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-11-16T17:00:02.878 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.878 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.879 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.879 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.879 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.879 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.879 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.879 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.879 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2023-11-16T17:00:02.879 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2023-11-16T17:00:02.880 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.880 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.880 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.880 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8080", 2023-11-16T17:00:02.880 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.880 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.880 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.880 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.881 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.881 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.881 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.881 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.881 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl": { 2023-11-16T17:00:02.881 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl", 2023-11-16T17:00:02.881 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.881 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.881 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.882 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.882 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.882 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.882 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.882 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.882 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.882 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.882 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.883 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.883 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl_server_port": { 2023-11-16T17:00:02.883 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl_server_port", 2023-11-16T17:00:02.883 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.883 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.883 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.883 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8443", 2023-11-16T17:00:02.883 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.883 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.884 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.884 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.884 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.884 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.884 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.884 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.884 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_behaviour": { 2023-11-16T17:00:02.884 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_behaviour", 2023-11-16T17:00:02.884 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.885 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.885 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.885 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "redirect", 2023-11-16T17:00:02.885 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.885 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.885 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.885 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.885 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect" 2023-11-16T17:00:02.885 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.886 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.886 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.886 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.886 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.886 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.886 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_error_status_code": { 2023-11-16T17:00:02.886 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_error_status_code", 2023-11-16T17:00:02.886 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.886 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.887 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.887 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", 2023-11-16T17:00:02.887 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-11-16T17:00:02.887 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "599", 2023-11-16T17:00:02.887 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.887 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.887 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.887 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.887 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.888 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.888 INFO:teuthology.orchestra.run.smithi002.stdout: "url_prefix": { 2023-11-16T17:00:02.888 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "url_prefix", 2023-11-16T17:00:02.888 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.888 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.888 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.888 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.888 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.889 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.889 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.889 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.889 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.889 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.889 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.889 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.889 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.889 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.890 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.890 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", 2023-11-16T17:00:02.890 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.890 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.890 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.890 INFO:teuthology.orchestra.run.smithi002.stdout: "enable_monitoring": { 2023-11-16T17:00:02.890 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enable_monitoring", 2023-11-16T17:00:02.891 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.891 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.891 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.891 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.891 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.891 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.891 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.891 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "monitor device health metrics", 2023-11-16T17:00:02.891 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.892 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.892 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.892 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.892 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.892 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.892 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.892 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.892 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.892 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.893 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.893 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.893 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.893 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.893 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.893 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.893 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.893 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.893 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.894 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.894 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.894 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.894 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.894 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.894 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.894 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.895 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.895 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.895 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.895 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.895 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.895 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.895 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.895 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.896 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.896 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.896 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.896 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.896 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.896 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.896 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.896 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.896 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.897 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.897 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.897 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.897 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.897 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.897 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.897 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.897 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.898 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.898 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.898 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.898 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.898 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.898 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.898 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.898 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.899 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.899 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.899 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.899 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.899 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.899 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.899 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.899 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.900 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.900 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.900 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.900 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.900 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.900 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.900 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.900 INFO:teuthology.orchestra.run.smithi002.stdout: "mark_out_threshold": { 2023-11-16T17:00:02.901 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mark_out_threshold", 2023-11-16T17:00:02.901 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.901 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.901 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.901 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "2419200", 2023-11-16T17:00:02.901 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.901 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.901 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.901 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-11-16T17:00:02.902 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.902 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.902 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.902 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.902 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": { 2023-11-16T17:00:02.902 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pool_name", 2023-11-16T17:00:02.902 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.902 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.903 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.903 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "device_health_metrics", 2023-11-16T17:00:02.903 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.903 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.903 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.903 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "name of pool in which to store device health metrics", 2023-11-16T17:00:02.903 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.903 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.903 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.904 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.904 INFO:teuthology.orchestra.run.smithi002.stdout: "retention_period": { 2023-11-16T17:00:02.904 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retention_period", 2023-11-16T17:00:02.904 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.904 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.904 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.904 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15552000", 2023-11-16T17:00:02.904 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.904 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.905 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.905 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long to retain device health metrics", 2023-11-16T17:00:02.905 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.905 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.905 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.905 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.905 INFO:teuthology.orchestra.run.smithi002.stdout: "scrape_frequency": { 2023-11-16T17:00:02.905 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "scrape_frequency", 2023-11-16T17:00:02.906 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.906 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.906 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.906 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "86400", 2023-11-16T17:00:02.906 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.906 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.906 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.906 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to scrape device health metrics", 2023-11-16T17:00:02.906 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.907 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.907 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.907 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.907 INFO:teuthology.orchestra.run.smithi002.stdout: "self_heal": { 2023-11-16T17:00:02.907 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "self_heal", 2023-11-16T17:00:02.907 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.907 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.907 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.908 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:02.908 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.908 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.908 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.908 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-11-16T17:00:02.908 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.908 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.909 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.909 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.909 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T17:00:02.909 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T17:00:02.909 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.909 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.909 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.910 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T17:00:02.910 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.910 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.910 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.910 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to wake up and check device health", 2023-11-16T17:00:02.910 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.910 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.910 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.911 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.911 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_threshold": { 2023-11-16T17:00:02.911 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_threshold", 2023-11-16T17:00:02.911 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.911 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.911 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.911 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "7257600", 2023-11-16T17:00:02.911 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.911 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.912 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.912 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-11-16T17:00:02.912 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.912 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.912 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.912 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.912 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.912 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.913 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.913 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "diskprediction_local", 2023-11-16T17:00:02.913 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.913 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.913 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.913 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.913 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.913 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.913 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.914 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.914 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.914 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.914 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.914 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.914 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.914 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.915 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.915 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.915 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.915 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.915 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.915 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.915 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.915 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.915 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.916 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.916 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.916 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.916 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.916 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.916 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.916 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.916 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.917 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.917 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.917 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.917 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.917 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.917 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.917 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.917 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.918 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.918 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.918 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.918 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.918 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.918 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.918 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.918 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.919 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.919 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.919 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.919 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.919 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.919 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.919 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.920 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.920 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.920 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.920 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.920 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.920 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.920 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.920 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.920 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.921 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.921 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.921 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.921 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.921 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.921 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.921 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.921 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.922 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.922 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.922 INFO:teuthology.orchestra.run.smithi002.stdout: "predict_interval": { 2023-11-16T17:00:02.922 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "predict_interval", 2023-11-16T17:00:02.922 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.922 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.922 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.922 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "86400", 2023-11-16T17:00:02.922 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.923 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.923 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.923 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.923 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.923 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.923 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.923 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.923 INFO:teuthology.orchestra.run.smithi002.stdout: "predictor_model": { 2023-11-16T17:00:02.924 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "predictor_model", 2023-11-16T17:00:02.924 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.924 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.924 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.924 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "prophetstor", 2023-11-16T17:00:02.924 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.924 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.924 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.924 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.925 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.925 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.925 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.925 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.925 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T17:00:02.925 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T17:00:02.925 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.925 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.926 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.926 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2023-11-16T17:00:02.926 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.926 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.926 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.926 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.926 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.926 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.926 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.927 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.927 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.927 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.927 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.927 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "influx", 2023-11-16T17:00:02.927 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": false, 2023-11-16T17:00:02.927 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "influxdb python module not found", 2023-11-16T17:00:02.927 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.927 INFO:teuthology.orchestra.run.smithi002.stdout: "batch_size": { 2023-11-16T17:00:02.928 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "batch_size", 2023-11-16T17:00:02.928 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.928 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.928 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.928 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5000", 2023-11-16T17:00:02.928 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.928 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.928 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.928 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-11-16T17:00:02.929 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.929 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.929 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.929 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.929 INFO:teuthology.orchestra.run.smithi002.stdout: "database": { 2023-11-16T17:00:02.929 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "database", 2023-11-16T17:00:02.929 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.929 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.930 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.930 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "ceph", 2023-11-16T17:00:02.930 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.930 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.930 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.930 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-11-16T17:00:02.930 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.930 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.931 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.931 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.931 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": { 2023-11-16T17:00:02.931 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "hostname", 2023-11-16T17:00:02.931 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.931 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.931 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.931 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.931 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.932 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.932 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.932 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB server hostname", 2023-11-16T17:00:02.932 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.932 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.932 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.932 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.932 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T17:00:02.933 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T17:00:02.933 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:02.933 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.933 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.933 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "30", 2023-11-16T17:00:02.933 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "5", 2023-11-16T17:00:02.933 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.933 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.934 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-11-16T17:00:02.934 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.934 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.934 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.934 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.934 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.934 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.934 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.935 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.935 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.935 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.935 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.935 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.935 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.935 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.935 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.935 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.936 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.936 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.936 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.936 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.936 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.936 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.936 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.936 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.936 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.937 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.937 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.937 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.937 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.937 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.937 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.937 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.937 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.938 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.938 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.938 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.938 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.938 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.938 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.938 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.938 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.939 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.939 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.939 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.939 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.939 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.939 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.939 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.939 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.940 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.940 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.940 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.940 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.940 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.940 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.940 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.940 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.941 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.941 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.941 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.941 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.941 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.941 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.941 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.941 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.941 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.942 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.942 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.942 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.942 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.942 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.942 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.942 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.942 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.943 INFO:teuthology.orchestra.run.smithi002.stdout: "password": { 2023-11-16T17:00:02.943 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "password", 2023-11-16T17:00:02.943 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.943 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.943 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.943 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.943 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.943 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.943 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.944 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "password of InfluxDB server user", 2023-11-16T17:00:02.944 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.944 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.944 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.944 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.944 INFO:teuthology.orchestra.run.smithi002.stdout: "port": { 2023-11-16T17:00:02.944 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "port", 2023-11-16T17:00:02.944 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.944 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.945 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.945 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8086", 2023-11-16T17:00:02.945 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.945 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.945 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.945 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB server port", 2023-11-16T17:00:02.945 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.946 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.946 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.946 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.946 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl": { 2023-11-16T17:00:02.946 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl", 2023-11-16T17:00:02.946 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.946 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.946 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.946 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "false", 2023-11-16T17:00:02.947 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.947 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.947 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.947 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-11-16T17:00:02.947 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.947 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.947 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.947 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.948 INFO:teuthology.orchestra.run.smithi002.stdout: "threads": { 2023-11-16T17:00:02.948 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "threads", 2023-11-16T17:00:02.948 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.948 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.948 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.948 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T17:00:02.948 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T17:00:02.948 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "32", 2023-11-16T17:00:02.948 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.949 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-11-16T17:00:02.949 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.949 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.949 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.949 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.949 INFO:teuthology.orchestra.run.smithi002.stdout: "username": { 2023-11-16T17:00:02.949 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "username", 2023-11-16T17:00:02.949 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.950 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.950 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.950 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.950 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.950 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.950 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.950 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "username of InfluxDB server user", 2023-11-16T17:00:02.950 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.951 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.951 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.951 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.951 INFO:teuthology.orchestra.run.smithi002.stdout: "verify_ssl": { 2023-11-16T17:00:02.951 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "verify_ssl", 2023-11-16T17:00:02.951 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.951 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.951 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.951 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "true", 2023-11-16T17:00:02.952 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.952 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.952 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.952 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-11-16T17:00:02.952 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.952 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.952 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.952 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.952 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.953 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.953 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.953 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "insights", 2023-11-16T17:00:02.953 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.953 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.953 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.953 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.953 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.953 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.954 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.954 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.954 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.954 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.954 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.954 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.954 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.954 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.955 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.955 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.955 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.955 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.955 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.955 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.955 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.955 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.955 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.956 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.956 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.956 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.956 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.956 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.956 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.956 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.956 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.956 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.957 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.957 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.957 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.957 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.957 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.957 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.957 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.957 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.957 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.958 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.958 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.958 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.958 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.958 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.958 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.958 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.958 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.959 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.959 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.959 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.959 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.959 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.959 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.959 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.959 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.960 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.960 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.960 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.960 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.960 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.960 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.960 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.960 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.961 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.961 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.961 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.961 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.961 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.961 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.961 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.961 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.961 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.962 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.962 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.962 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "iostat", 2023-11-16T17:00:02.962 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.962 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.962 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.962 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.962 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.962 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.963 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.963 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.963 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.963 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.963 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.963 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.963 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.963 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.963 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.964 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.964 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.964 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.964 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.964 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.964 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.964 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.964 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.965 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.965 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.965 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.965 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.965 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.965 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.965 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.965 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.965 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.966 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.966 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.966 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.966 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.966 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.966 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.966 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.966 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.966 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.967 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.967 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.967 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.967 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.967 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.967 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.967 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.967 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.967 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.968 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.968 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.968 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.968 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.968 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.968 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.968 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.968 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.969 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.969 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.969 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.969 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.969 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.969 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.969 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.969 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.970 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.970 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.970 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.970 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.970 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.970 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.970 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.970 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.970 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.971 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.971 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "k8sevents", 2023-11-16T17:00:02.971 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.971 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.971 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.971 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_event_retention_days": { 2023-11-16T17:00:02.971 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ceph_event_retention_days", 2023-11-16T17:00:02.971 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.971 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.972 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.972 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "7", 2023-11-16T17:00:02.972 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.972 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.972 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.972 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Days to hold ceph event information within local cache", 2023-11-16T17:00:02.972 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.972 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.973 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.973 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.973 INFO:teuthology.orchestra.run.smithi002.stdout: "config_check_secs": { 2023-11-16T17:00:02.973 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_check_secs", 2023-11-16T17:00:02.973 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.973 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.973 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:02.973 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:02.973 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "10", 2023-11-16T17:00:02.974 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.974 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.974 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-11-16T17:00:02.974 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.974 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.974 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.974 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.974 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.974 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.975 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.975 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.975 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.975 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.975 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.975 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.975 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.975 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.975 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.976 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.976 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.976 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.976 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.976 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.976 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.976 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.976 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.976 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.976 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.977 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.977 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.977 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.977 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.977 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.977 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.977 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.977 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.977 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.978 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.978 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.978 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.978 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.978 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.978 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.978 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.978 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.978 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.979 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.979 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.979 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.979 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.979 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.979 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.979 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.980 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.980 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.980 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.980 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.980 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.980 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.980 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.980 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.980 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.981 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.981 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.981 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.981 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.981 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.981 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.981 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.981 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.981 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.982 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.982 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.982 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.982 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.982 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.982 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.982 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:02.982 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.983 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:02.983 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "localpool", 2023-11-16T17:00:02.983 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:02.983 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:02.983 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:02.983 INFO:teuthology.orchestra.run.smithi002.stdout: "failure_domain": { 2023-11-16T17:00:02.983 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "failure_domain", 2023-11-16T17:00:02.983 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.983 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.984 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.984 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "host", 2023-11-16T17:00:02.984 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.984 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.984 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.984 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "failure domain for any created local pool", 2023-11-16T17:00:02.984 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-11-16T17:00:02.984 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.984 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.985 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.985 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:02.985 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:02.985 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.985 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.985 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.985 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.985 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.985 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.986 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.986 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.986 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.986 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.986 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.986 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.986 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.986 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.987 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.987 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.987 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.987 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.987 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.987 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:02.987 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:02.987 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.987 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.988 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.988 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.988 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.988 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.988 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.988 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.988 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.988 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.989 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.989 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.989 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:02.989 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:02.989 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.989 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.989 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.989 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:02.990 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.990 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.990 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:02.990 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:02.990 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:02.990 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:02.990 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:02.990 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:02.991 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:02.991 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:02.991 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.991 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.991 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.991 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.991 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.991 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:02.991 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:02.992 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:02.992 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.992 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.992 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:02.992 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.992 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.992 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.992 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:02.993 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.993 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.993 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.993 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.993 INFO:teuthology.orchestra.run.smithi002.stdout: "min_size": { 2023-11-16T17:00:02.993 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "min_size", 2023-11-16T17:00:02.993 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.993 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.993 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.994 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.994 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.994 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.994 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.994 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default min_size for any created local pool", 2023-11-16T17:00:02.994 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-11-16T17:00:02.994 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.994 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.995 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.995 INFO:teuthology.orchestra.run.smithi002.stdout: "num_rep": { 2023-11-16T17:00:02.995 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "num_rep", 2023-11-16T17:00:02.995 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.995 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.995 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.995 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3", 2023-11-16T17:00:02.995 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.995 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.996 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.996 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default replica count for any created local pool", 2023-11-16T17:00:02.996 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.996 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.996 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.996 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.996 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num": { 2023-11-16T17:00:02.996 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pg_num", 2023-11-16T17:00:02.997 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:02.997 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.997 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.997 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "128", 2023-11-16T17:00:02.997 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.997 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.997 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.997 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default pg_num for any created local pool", 2023-11-16T17:00:02.997 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.998 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.998 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:02.998 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:02.998 INFO:teuthology.orchestra.run.smithi002.stdout: "prefix": { 2023-11-16T17:00:02.998 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prefix", 2023-11-16T17:00:02.998 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:02.998 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:02.998 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:02.999 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:02.999 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:02.999 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:02.999 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:02.999 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "name prefix for any created local pool", 2023-11-16T17:00:02.999 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:02.999 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:02.999 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.000 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.000 INFO:teuthology.orchestra.run.smithi002.stdout: "subtree": { 2023-11-16T17:00:03.000 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subtree", 2023-11-16T17:00:03.000 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.000 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.000 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.000 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "rack", 2023-11-16T17:00:03.000 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.001 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.001 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.001 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "CRUSH level for which to create a local pool", 2023-11-16T17:00:03.001 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-11-16T17:00:03.001 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.001 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.001 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.001 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.001 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.002 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.002 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mds_autoscaler", 2023-11-16T17:00:03.002 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.002 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.002 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.002 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.002 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.003 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.003 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.003 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.003 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.003 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.003 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.003 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.003 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.003 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.004 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.004 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.004 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.004 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.004 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.004 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.004 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.004 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.005 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.005 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.005 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.005 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.005 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.005 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.005 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.005 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.006 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.006 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.006 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.006 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.006 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.006 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.006 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.006 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.007 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.007 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.007 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.007 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.007 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.007 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.007 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.007 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.008 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.008 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.008 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.008 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.008 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.008 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.008 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.008 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.009 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.009 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.009 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.009 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.009 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.009 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.009 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.009 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.010 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.010 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.010 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.010 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.010 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.010 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.010 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.010 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.011 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.011 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.011 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.011 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.011 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.011 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.011 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirroring", 2023-11-16T17:00:03.011 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.012 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.012 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.012 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.012 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.012 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.012 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.012 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.012 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.013 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.013 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.013 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.013 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.013 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.013 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.013 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.013 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.014 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.014 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.014 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.014 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.014 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.014 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.014 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.014 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.015 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.015 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.015 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.015 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.015 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.015 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.015 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.015 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.016 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.016 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.016 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.016 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.016 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.016 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.016 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.016 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.017 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.017 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.017 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.017 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.017 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.017 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.017 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.017 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.018 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.018 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.018 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.018 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.018 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.018 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.018 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.019 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.019 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.019 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.019 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.019 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.019 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.019 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.019 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.020 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.020 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.020 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.020 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.020 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.020 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.020 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.020 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.020 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.021 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.021 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.021 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.021 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "nfs", 2023-11-16T17:00:03.021 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.021 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.021 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.021 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.022 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.022 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.022 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.022 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.022 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.022 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.022 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.022 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.023 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.023 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.023 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.023 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.023 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.023 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.023 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.023 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.024 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.024 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.024 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.024 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.024 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.024 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.024 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.024 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.024 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.025 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.025 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.025 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.025 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.025 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.025 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.025 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.025 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.026 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.026 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.026 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.026 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.026 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.026 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.026 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.026 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.026 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.027 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.027 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.027 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.027 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.027 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.027 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.027 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.027 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.027 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.028 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.028 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.028 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.028 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.028 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.028 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.028 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.028 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.028 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.029 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.029 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.029 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.029 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.029 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.029 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.029 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.029 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.030 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.030 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.030 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.030 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.030 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "orchestrator", 2023-11-16T17:00:03.030 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.030 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.030 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.031 INFO:teuthology.orchestra.run.smithi002.stdout: "fail_fs": { 2023-11-16T17:00:03.031 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "fail_fs", 2023-11-16T17:00:03.031 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.031 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.031 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.031 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.031 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.031 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.032 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.032 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-11-16T17:00:03.032 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.032 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.032 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.032 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.032 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.032 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.032 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.033 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.033 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.033 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.033 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.033 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.033 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.033 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.033 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.033 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.034 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.034 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.034 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.034 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.034 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.034 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.034 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.034 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.034 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.035 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.035 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.035 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.035 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.035 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.035 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.035 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.035 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.035 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.036 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.036 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.036 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.036 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.036 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.036 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.036 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.036 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.036 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.037 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.037 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.037 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.037 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.037 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.037 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.037 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.037 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.037 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.038 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.038 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.038 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.038 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.038 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.038 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.038 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.038 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.039 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.039 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.039 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.039 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.039 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.039 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.039 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.039 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.040 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.040 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.040 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.040 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.040 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.040 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.040 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator": { 2023-11-16T17:00:03.041 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "orchestrator", 2023-11-16T17:00:03.041 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.041 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.041 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.041 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.041 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.041 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.041 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.041 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm", 2023-11-16T17:00:03.042 INFO:teuthology.orchestra.run.smithi002.stdout: "rook", 2023-11-16T17:00:03.042 INFO:teuthology.orchestra.run.smithi002.stdout: "test_orchestrator" 2023-11-16T17:00:03.042 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.042 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Orchestrator backend", 2023-11-16T17:00:03.042 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.042 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.042 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.043 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.043 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.043 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.043 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.043 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "osd_perf_query", 2023-11-16T17:00:03.043 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.043 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.043 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.044 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.044 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.044 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.044 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.044 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.044 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.044 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.044 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.044 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.045 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.045 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.045 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.045 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.045 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.045 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.045 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.045 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.046 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.046 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.046 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.046 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.046 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.046 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.046 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.047 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.047 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.047 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.047 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.047 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.047 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.047 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.047 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.047 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.048 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.048 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.048 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.048 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.048 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.048 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.048 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.048 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.049 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.049 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.049 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.049 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.049 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.049 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.049 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.050 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.050 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.050 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.050 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.050 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.050 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.050 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.050 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.051 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.051 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.051 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.051 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.051 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.051 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.051 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.051 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.051 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.052 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.052 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.052 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.052 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.052 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.052 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.052 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.052 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.052 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "osd_support", 2023-11-16T17:00:03.053 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.053 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.053 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.053 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.053 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.053 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.053 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.053 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.054 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.054 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.054 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.054 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.054 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.054 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.054 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.054 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.054 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.055 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.055 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.055 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.055 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.055 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.055 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.055 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.055 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.056 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.056 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.056 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.056 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.056 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.056 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.056 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.056 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.057 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.057 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.057 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.057 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.057 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.057 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.057 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.057 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.058 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.058 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.058 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.058 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.058 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.058 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.058 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.059 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.059 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.059 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.059 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.059 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.059 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.059 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.060 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.060 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.060 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.060 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.060 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.060 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.060 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.060 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.060 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.061 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.061 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.061 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.061 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.061 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.061 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.061 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.061 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.062 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.062 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.062 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.062 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.062 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pg_autoscaler", 2023-11-16T17:00:03.062 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.062 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.062 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.063 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.063 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.063 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.063 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.063 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.063 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.063 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.063 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.063 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.064 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.064 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.064 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.064 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.064 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.064 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.064 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.064 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.065 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.065 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.065 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.065 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.065 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.065 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.065 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.065 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.065 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.066 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.066 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.066 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.066 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.066 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.066 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.066 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.066 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.067 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.067 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.067 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.067 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.067 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.067 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.067 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.067 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.067 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.068 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.068 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.068 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.068 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.068 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.068 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.068 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.068 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.069 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.069 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.069 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.069 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.069 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.069 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.069 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.070 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.070 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.070 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.070 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.070 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.070 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.070 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.070 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.071 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.071 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.071 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.071 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.071 INFO:teuthology.orchestra.run.smithi002.stdout: "noautoscale": { 2023-11-16T17:00:03.071 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "noautoscale", 2023-11-16T17:00:03.071 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.071 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.072 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.072 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.072 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.072 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.072 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.072 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "global autoscale flag", 2023-11-16T17:00:03.072 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-11-16T17:00:03.072 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.072 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.073 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.073 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T17:00:03.073 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T17:00:03.073 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:03.073 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.073 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.073 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T17:00:03.073 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.073 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.074 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.074 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.074 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.074 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.074 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.074 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.074 INFO:teuthology.orchestra.run.smithi002.stdout: "threshold": { 2023-11-16T17:00:03.074 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "threshold", 2023-11-16T17:00:03.075 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:03.075 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.075 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.075 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3.0", 2023-11-16T17:00:03.075 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1.0", 2023-11-16T17:00:03.075 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.075 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.075 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "scaling threshold", 2023-11-16T17:00:03.075 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-11-16T17:00:03.076 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.076 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.076 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.076 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.076 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.076 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.076 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "progress", 2023-11-16T17:00:03.076 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.076 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.077 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.077 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_pg_recovery_event": { 2023-11-16T17:00:03.077 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_pg_recovery_event", 2023-11-16T17:00:03.077 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.077 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.077 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.077 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.077 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.078 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.078 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.078 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow the module to show pg recovery progress", 2023-11-16T17:00:03.078 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.078 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.078 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.078 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.078 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": { 2023-11-16T17:00:03.079 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enabled", 2023-11-16T17:00:03.079 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.079 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.079 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.079 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:03.079 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.079 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.079 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.080 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.080 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.080 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.080 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.080 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.080 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.080 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.080 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.081 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.081 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.081 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.081 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.081 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.081 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.081 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.081 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.081 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.082 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.082 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.082 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.082 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.082 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.082 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.082 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.082 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.083 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.083 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.083 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.083 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.083 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.083 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.083 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.083 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.083 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.084 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.084 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.084 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.084 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.084 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.084 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.084 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.084 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.085 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.085 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.085 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.085 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.085 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.085 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.085 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.085 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.085 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.086 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.086 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.086 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.086 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.086 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.086 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.086 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.086 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.086 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.087 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.087 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.087 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.087 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.087 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.087 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.087 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.087 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.087 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.088 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.088 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.088 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.088 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.088 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.088 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.088 INFO:teuthology.orchestra.run.smithi002.stdout: "max_completed_events": { 2023-11-16T17:00:03.088 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_completed_events", 2023-11-16T17:00:03.088 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:03.089 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.089 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.089 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "50", 2023-11-16T17:00:03.089 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.089 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.089 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.089 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "number of past completed events to remember", 2023-11-16T17:00:03.089 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.090 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.090 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.090 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.090 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2023-11-16T17:00:03.090 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2023-11-16T17:00:03.090 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:03.090 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.090 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.091 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2023-11-16T17:00:03.091 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.091 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.091 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.091 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long the module is going to sleep", 2023-11-16T17:00:03.091 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.091 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.091 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.091 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.092 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.092 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.092 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.092 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus", 2023-11-16T17:00:03.092 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.092 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.092 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.092 INFO:teuthology.orchestra.run.smithi002.stdout: "cache": { 2023-11-16T17:00:03.093 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cache", 2023-11-16T17:00:03.093 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.093 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.093 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.093 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:03.093 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.093 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.093 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.093 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.094 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.094 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.094 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.094 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.094 INFO:teuthology.orchestra.run.smithi002.stdout: "exclude_perf_counters": { 2023-11-16T17:00:03.094 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "exclude_perf_counters", 2023-11-16T17:00:03.094 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.094 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.095 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.095 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:03.095 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.095 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.095 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.095 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-11-16T17:00:03.095 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-11-16T17:00:03.095 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.095 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.096 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.096 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.096 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.096 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.096 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.096 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.096 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.096 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.097 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.097 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.097 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.097 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.097 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.097 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.097 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.097 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.097 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.098 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.098 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.098 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.098 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.098 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.098 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.098 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.098 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.099 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.099 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.099 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.099 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.099 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.099 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.099 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.099 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.100 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.100 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.100 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.100 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.100 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.100 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.100 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.100 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.101 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.101 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.101 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.101 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.101 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.101 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.101 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.101 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.101 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.102 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.102 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.102 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.102 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.102 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.102 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.102 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.102 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.102 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.103 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.103 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.103 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.103 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.103 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.103 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.103 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.103 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.104 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.104 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.104 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.104 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.104 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_stats_pools": { 2023-11-16T17:00:03.104 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_stats_pools", 2023-11-16T17:00:03.104 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.104 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.104 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.105 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.105 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.105 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.105 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.105 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.105 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.105 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.105 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.105 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.106 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_stats_pools_refresh_interval": { 2023-11-16T17:00:03.106 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-11-16T17:00:03.106 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:03.106 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.106 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.106 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "300", 2023-11-16T17:00:03.106 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.106 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.106 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.107 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.107 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.107 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.107 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.107 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.107 INFO:teuthology.orchestra.run.smithi002.stdout: "scrape_interval": { 2023-11-16T17:00:03.107 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "scrape_interval", 2023-11-16T17:00:03.107 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:03.107 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.108 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.108 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15.0", 2023-11-16T17:00:03.108 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.108 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.108 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.108 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.108 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.108 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.108 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.109 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.109 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2023-11-16T17:00:03.109 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2023-11-16T17:00:03.109 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.109 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.109 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.109 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2023-11-16T17:00:03.109 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.110 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.110 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.110 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-11-16T17:00:03.110 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.110 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.110 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.110 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.110 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2023-11-16T17:00:03.111 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2023-11-16T17:00:03.111 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:03.111 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.111 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.111 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "9283", 2023-11-16T17:00:03.111 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.111 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.111 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.111 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-11-16T17:00:03.112 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.112 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.112 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.112 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.112 INFO:teuthology.orchestra.run.smithi002.stdout: "stale_cache_strategy": { 2023-11-16T17:00:03.112 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "stale_cache_strategy", 2023-11-16T17:00:03.112 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.112 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.113 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.113 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "log", 2023-11-16T17:00:03.113 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.113 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.113 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.113 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.113 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.113 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.114 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.114 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.114 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_behaviour": { 2023-11-16T17:00:03.114 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_behaviour", 2023-11-16T17:00:03.114 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.114 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.114 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.114 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "default", 2023-11-16T17:00:03.115 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.115 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.115 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.115 INFO:teuthology.orchestra.run.smithi002.stdout: "default", 2023-11-16T17:00:03.115 INFO:teuthology.orchestra.run.smithi002.stdout: "error" 2023-11-16T17:00:03.115 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.115 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.115 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.116 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.116 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.116 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.116 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_error_status_code": { 2023-11-16T17:00:03.116 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_error_status_code", 2023-11-16T17:00:03.116 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:03.116 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.116 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.117 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", 2023-11-16T17:00:03.117 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2023-11-16T17:00:03.117 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "599", 2023-11-16T17:00:03.117 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.117 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.117 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.117 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.117 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.117 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.118 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.118 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.118 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.118 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_support", 2023-11-16T17:00:03.118 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.118 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.118 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.119 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.119 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.119 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.119 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.119 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.119 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.119 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.120 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.120 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.120 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.120 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.120 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.120 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.120 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.120 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.120 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.121 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.121 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.121 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.121 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.121 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.121 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.121 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.121 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.122 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.122 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.122 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.122 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.122 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.122 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.122 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.122 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.122 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.123 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.123 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.123 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.123 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.123 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.123 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.123 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.124 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.124 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.124 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.124 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.124 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.124 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.124 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.124 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.124 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.125 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.125 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.125 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.125 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.125 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.125 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.125 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.125 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.125 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.126 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.126 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.126 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.126 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.126 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.126 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.126 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.126 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.127 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.127 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.127 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.127 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.127 INFO:teuthology.orchestra.run.smithi002.stdout: "max_concurrent_snap_create": { 2023-11-16T17:00:03.127 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_concurrent_snap_create", 2023-11-16T17:00:03.127 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:03.127 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.127 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.128 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2023-11-16T17:00:03.128 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.128 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.128 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.128 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.128 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.128 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.128 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.129 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.129 INFO:teuthology.orchestra.run.smithi002.stdout: "mirror_snapshot_schedule": { 2023-11-16T17:00:03.129 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirror_snapshot_schedule", 2023-11-16T17:00:03.129 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.129 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.129 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.129 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.129 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.130 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.130 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.130 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.130 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.130 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.130 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.130 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.130 INFO:teuthology.orchestra.run.smithi002.stdout: "trash_purge_schedule": { 2023-11-16T17:00:03.131 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "trash_purge_schedule", 2023-11-16T17:00:03.131 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.131 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.131 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.131 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.131 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.131 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.131 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.131 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.132 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.132 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.132 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.132 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.132 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.132 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.132 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.132 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "restful", 2023-11-16T17:00:03.132 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.133 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.133 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.133 INFO:teuthology.orchestra.run.smithi002.stdout: "enable_auth": { 2023-11-16T17:00:03.133 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enable_auth", 2023-11-16T17:00:03.133 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.133 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.133 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.133 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:03.134 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.134 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.134 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.134 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.134 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.134 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.134 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.134 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.135 INFO:teuthology.orchestra.run.smithi002.stdout: "key_file": { 2023-11-16T17:00:03.135 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "key_file", 2023-11-16T17:00:03.135 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.135 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.135 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.135 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.135 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.135 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.135 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.136 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.136 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.136 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.136 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.136 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.136 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.136 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.136 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.136 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.137 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.137 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.137 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.137 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.137 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.137 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.137 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.137 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.137 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.138 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.138 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.138 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.138 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.138 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.138 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.138 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.138 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.138 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.139 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.139 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.139 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.139 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.139 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.139 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.139 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.140 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.140 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.140 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.140 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.140 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.140 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.140 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.140 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.141 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.141 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.141 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.141 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.141 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.141 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.141 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.141 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.142 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.142 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.142 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.142 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.142 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.142 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.142 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.142 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.143 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.143 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.143 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.143 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.143 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.143 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.143 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.143 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.144 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.144 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.144 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.144 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.144 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.144 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.144 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.144 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.145 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.145 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2023-11-16T17:00:03.145 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2023-11-16T17:00:03.145 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.145 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.145 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.145 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.145 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.146 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.146 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.146 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.146 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.146 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.146 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.146 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.146 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2023-11-16T17:00:03.147 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2023-11-16T17:00:03.147 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.147 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.147 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.147 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.147 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.147 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.147 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.147 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.148 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.148 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.148 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.148 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.148 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.148 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.148 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.148 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rgw", 2023-11-16T17:00:03.149 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.149 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.149 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.149 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.149 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.149 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.149 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.149 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.150 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.150 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.150 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.150 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.150 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.150 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.150 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.150 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.151 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.151 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.151 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.151 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.151 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.151 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.151 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.151 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.151 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.152 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.152 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.152 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.152 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.152 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.152 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.152 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.152 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.153 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.153 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.153 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.153 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.153 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.153 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.153 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.153 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.153 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.154 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.154 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.154 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.154 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.154 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.154 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.154 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.154 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.155 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.155 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.155 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.155 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.155 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.155 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.155 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.155 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.156 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.156 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.156 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.156 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.156 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.156 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.156 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.156 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.156 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.157 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.157 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.157 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.157 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.157 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.157 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.157 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.157 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.157 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.158 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rook", 2023-11-16T17:00:03.158 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.158 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.158 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.158 INFO:teuthology.orchestra.run.smithi002.stdout: "drive_group_interval": { 2023-11-16T17:00:03.158 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "drive_group_interval", 2023-11-16T17:00:03.158 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:03.158 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.159 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.159 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "300.0", 2023-11-16T17:00:03.159 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.159 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.159 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.159 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-11-16T17:00:03.159 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.160 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.160 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.160 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.160 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.160 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.160 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.160 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.160 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.161 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.161 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.161 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.161 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.161 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.161 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.161 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.161 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.161 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.162 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.162 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.162 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.162 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.162 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.162 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.162 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.162 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.163 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.163 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.163 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.163 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.163 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.163 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.163 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.163 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.163 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.164 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.164 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.164 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.164 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.164 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.164 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.164 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.164 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.164 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.165 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.165 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.165 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.165 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.165 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.165 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.165 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.165 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.166 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.166 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.166 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.166 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.166 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.166 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.166 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.166 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.167 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.167 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.167 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.167 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.167 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.167 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.167 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.167 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.168 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.168 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.168 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.168 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.168 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.168 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.168 INFO:teuthology.orchestra.run.smithi002.stdout: "storage_class": { 2023-11-16T17:00:03.168 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "storage_class", 2023-11-16T17:00:03.169 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.169 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.169 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.169 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "local", 2023-11-16T17:00:03.169 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.169 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.169 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.169 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-11-16T17:00:03.170 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.170 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.170 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.170 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.170 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.170 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.170 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.171 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "selftest", 2023-11-16T17:00:03.171 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.171 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.171 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.171 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.171 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.171 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.171 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.172 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.172 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.172 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.172 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.172 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.172 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.172 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.172 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.172 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.173 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.173 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.173 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.173 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.173 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.173 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.173 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.173 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.174 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.174 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.174 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.174 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.174 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.174 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.174 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.175 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.175 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.175 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.175 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.175 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.175 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.175 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.175 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.175 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.176 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.176 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.176 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.176 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.176 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.176 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.176 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.176 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.177 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.177 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.177 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.177 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.177 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.177 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.177 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.177 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.178 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.178 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.178 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.178 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.178 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.178 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.178 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.179 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.179 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.179 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.179 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.179 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.179 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.179 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.179 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.180 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.180 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.180 INFO:teuthology.orchestra.run.smithi002.stdout: "roption1": { 2023-11-16T17:00:03.180 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "roption1", 2023-11-16T17:00:03.180 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.180 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.180 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.180 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.181 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.181 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.181 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.181 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.181 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.181 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.181 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.181 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.181 INFO:teuthology.orchestra.run.smithi002.stdout: "roption2": { 2023-11-16T17:00:03.182 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "roption2", 2023-11-16T17:00:03.182 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.182 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.182 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.182 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "xyz", 2023-11-16T17:00:03.182 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.182 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.182 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.183 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.183 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.183 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.183 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.183 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.183 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption1": { 2023-11-16T17:00:03.183 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption1", 2023-11-16T17:00:03.183 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.183 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.184 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.185 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.185 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.185 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption2": { 2023-11-16T17:00:03.185 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption2", 2023-11-16T17:00:03.185 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:03.185 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.185 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.185 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.186 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.186 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.186 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.186 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.186 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.186 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.186 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.186 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.186 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption3": { 2023-11-16T17:00:03.187 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption3", 2023-11-16T17:00:03.187 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2023-11-16T17:00:03.187 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.187 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.187 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.187 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.187 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.187 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.187 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.188 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.188 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.188 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.188 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.188 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption4": { 2023-11-16T17:00:03.188 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption4", 2023-11-16T17:00:03.188 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.188 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.189 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.189 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.189 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.189 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.189 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.189 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.189 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.189 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.190 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.190 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.190 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption5": { 2023-11-16T17:00:03.190 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption5", 2023-11-16T17:00:03.190 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.190 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.190 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.190 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.191 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.191 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.191 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.191 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.191 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.191 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.191 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.193 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.193 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption6": { 2023-11-16T17:00:03.193 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption6", 2023-11-16T17:00:03.193 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.193 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.193 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.193 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:03.194 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.194 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.194 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.194 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.194 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.194 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.194 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.194 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.195 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption7": { 2023-11-16T17:00:03.195 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption7", 2023-11-16T17:00:03.195 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:03.195 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.195 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.195 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.195 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2023-11-16T17:00:03.195 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "42", 2023-11-16T17:00:03.196 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.196 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.196 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.196 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.196 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.196 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.196 INFO:teuthology.orchestra.run.smithi002.stdout: "testkey": { 2023-11-16T17:00:03.196 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testkey", 2023-11-16T17:00:03.197 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.197 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.197 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.197 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.197 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.197 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.197 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.197 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.198 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.198 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.198 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.198 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.198 INFO:teuthology.orchestra.run.smithi002.stdout: "testlkey": { 2023-11-16T17:00:03.198 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testlkey", 2023-11-16T17:00:03.198 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.199 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.199 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.199 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.199 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.199 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.199 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.199 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.200 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.200 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.200 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.200 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.200 INFO:teuthology.orchestra.run.smithi002.stdout: "testnewline": { 2023-11-16T17:00:03.200 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testnewline", 2023-11-16T17:00:03.200 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.200 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.200 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.201 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.201 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.201 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.201 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.201 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.201 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.201 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.201 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.202 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.202 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.202 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.202 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.202 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snap_schedule", 2023-11-16T17:00:03.202 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.202 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.202 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.203 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_m_granularity": { 2023-11-16T17:00:03.203 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_m_granularity", 2023-11-16T17:00:03.203 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.203 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.203 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.203 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.203 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.203 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.204 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.204 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow minute scheduled snapshots", 2023-11-16T17:00:03.204 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.204 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.204 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.204 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.204 INFO:teuthology.orchestra.run.smithi002.stdout: "dump_on_update": { 2023-11-16T17:00:03.204 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "dump_on_update", 2023-11-16T17:00:03.205 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.205 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.205 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.205 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.205 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.205 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.205 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.205 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "dump database to debug log on update", 2023-11-16T17:00:03.206 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.206 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.206 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.206 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.206 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.206 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.206 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.206 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.207 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.207 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.207 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.207 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.207 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.207 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.207 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.207 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.207 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.208 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.208 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.208 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.208 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.208 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.208 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.208 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.209 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.209 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.209 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.209 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.209 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.209 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.209 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.210 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.210 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.210 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.210 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.210 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.210 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.210 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.210 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.211 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.211 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.211 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.211 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.211 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.211 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.211 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.211 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.211 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.212 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.212 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.212 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.212 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.212 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.212 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.212 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.212 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.212 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.213 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.213 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.213 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.213 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.213 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.213 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.213 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.213 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.213 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.214 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.214 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.214 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.214 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.214 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.214 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.214 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.214 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.215 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.215 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.215 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "stats", 2023-11-16T17:00:03.215 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.215 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.215 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.215 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.215 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.216 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.216 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.216 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.216 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.216 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.216 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.216 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.216 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.217 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.217 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.217 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.217 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.217 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.217 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.217 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.217 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.218 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.218 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.218 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.218 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.218 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.218 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.218 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.218 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.219 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.219 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.219 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.219 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.219 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.219 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.219 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.220 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.220 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.220 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.220 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.220 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.220 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.220 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.220 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.221 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.221 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.221 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.221 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.221 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.221 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.221 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.222 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.222 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.222 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.222 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.222 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.222 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.222 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.222 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.223 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.223 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.223 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.223 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.223 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.223 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.223 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.223 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.223 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.224 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.224 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.224 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.224 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.224 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.224 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.224 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.224 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.225 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "status", 2023-11-16T17:00:03.225 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.225 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.225 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.225 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.225 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.225 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.225 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.226 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.226 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.226 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.226 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.226 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.226 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.226 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.226 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.227 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.227 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.227 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.227 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.227 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.227 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.227 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.227 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.227 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.228 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.228 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.228 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.228 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.228 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.228 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.228 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.228 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.229 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.229 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.229 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.229 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.229 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.229 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.229 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.229 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.230 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.230 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.230 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.230 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.230 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.230 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.230 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.230 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.231 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.231 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.231 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.231 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.231 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.231 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.231 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.231 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.232 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.232 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.232 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.232 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.232 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.232 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.232 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.232 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.232 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.233 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.233 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.233 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.233 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.233 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.233 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.233 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.233 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.233 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.234 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.234 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.234 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "telegraf", 2023-11-16T17:00:03.234 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.234 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.234 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.234 INFO:teuthology.orchestra.run.smithi002.stdout: "address": { 2023-11-16T17:00:03.234 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "address", 2023-11-16T17:00:03.235 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.235 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.235 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.235 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-11-16T17:00:03.235 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.235 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.235 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.235 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.236 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.236 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.236 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.236 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.236 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T17:00:03.236 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T17:00:03.236 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:03.236 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.236 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.237 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15", 2023-11-16T17:00:03.237 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.237 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.237 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.237 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.237 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.237 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.237 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.237 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.238 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.238 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.238 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.238 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.238 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.238 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.238 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.238 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.238 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.239 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.239 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.239 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.239 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.239 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.239 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.239 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.240 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.240 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.240 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.240 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.240 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.240 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.240 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.240 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.240 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.241 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.241 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.241 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.241 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.241 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.241 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.241 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.241 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.241 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.242 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.242 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.242 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.242 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.242 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.242 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.242 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.243 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.243 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.243 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.243 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.243 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.243 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.243 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.243 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.243 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.244 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.244 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.244 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.244 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.244 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.244 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.244 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.244 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.245 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.245 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.245 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.245 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.245 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.245 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.245 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.245 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.246 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.246 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.246 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.246 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.246 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.246 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.246 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.246 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "telemetry", 2023-11-16T17:00:03.246 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.247 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.247 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.247 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_basic": { 2023-11-16T17:00:03.247 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_basic", 2023-11-16T17:00:03.247 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.247 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.247 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.247 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:03.248 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.248 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.248 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.248 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share basic cluster information (size, version)", 2023-11-16T17:00:03.248 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.248 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.248 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.248 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.249 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_crash": { 2023-11-16T17:00:03.249 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_crash", 2023-11-16T17:00:03.249 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.249 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.249 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.249 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:03.249 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.250 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.250 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.250 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-11-16T17:00:03.250 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.250 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.250 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.250 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.250 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_device": { 2023-11-16T17:00:03.250 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_device", 2023-11-16T17:00:03.251 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.251 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.251 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.251 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2023-11-16T17:00:03.251 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.251 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.251 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.251 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-11-16T17:00:03.251 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.252 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.252 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.252 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.252 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_ident": { 2023-11-16T17:00:03.252 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_ident", 2023-11-16T17:00:03.252 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.252 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.252 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.252 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.253 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.253 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.253 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.253 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-11-16T17:00:03.253 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.253 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.253 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.253 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.254 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_perf": { 2023-11-16T17:00:03.254 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_perf", 2023-11-16T17:00:03.254 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.254 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.254 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.254 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.254 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.254 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.254 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.255 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share various performance metrics of a cluster", 2023-11-16T17:00:03.255 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.255 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.255 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.255 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.255 INFO:teuthology.orchestra.run.smithi002.stdout: "contact": { 2023-11-16T17:00:03.255 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "contact", 2023-11-16T17:00:03.255 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.255 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.256 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.256 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.256 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.256 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.256 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.256 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.256 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.256 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.256 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.257 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.257 INFO:teuthology.orchestra.run.smithi002.stdout: "description": { 2023-11-16T17:00:03.257 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "description", 2023-11-16T17:00:03.257 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.257 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.257 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.257 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.257 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.257 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.257 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.258 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.258 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.258 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.258 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.258 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.258 INFO:teuthology.orchestra.run.smithi002.stdout: "device_url": { 2023-11-16T17:00:03.258 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_url", 2023-11-16T17:00:03.258 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.258 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.259 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.259 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-11-16T17:00:03.259 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.259 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.259 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.259 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.259 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.259 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.260 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.260 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.260 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": { 2023-11-16T17:00:03.260 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enabled", 2023-11-16T17:00:03.260 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.260 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.260 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.260 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.260 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.260 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.261 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.261 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.261 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.261 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.261 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.261 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.261 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T17:00:03.261 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T17:00:03.261 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:03.262 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.262 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.262 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "24", 2023-11-16T17:00:03.262 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "8", 2023-11-16T17:00:03.262 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.262 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.262 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.262 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.262 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.263 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.263 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.263 INFO:teuthology.orchestra.run.smithi002.stdout: "last_opt_revision": { 2023-11-16T17:00:03.263 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "last_opt_revision", 2023-11-16T17:00:03.263 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:03.263 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.263 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.263 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1", 2023-11-16T17:00:03.263 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.264 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.264 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.264 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.264 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.264 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.264 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.264 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.264 INFO:teuthology.orchestra.run.smithi002.stdout: "leaderboard": { 2023-11-16T17:00:03.264 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "leaderboard", 2023-11-16T17:00:03.265 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.265 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.265 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.265 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.265 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.265 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.265 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.265 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.265 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.266 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.266 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.266 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.266 INFO:teuthology.orchestra.run.smithi002.stdout: "leaderboard_description": { 2023-11-16T17:00:03.266 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "leaderboard_description", 2023-11-16T17:00:03.266 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.266 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.266 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.266 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.267 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.267 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.267 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.267 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.267 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.267 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.267 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.267 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.267 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.267 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.268 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.268 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.268 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.268 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.268 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.268 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.268 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.268 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.268 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.269 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.269 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.269 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.269 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.269 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.269 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.269 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.269 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.269 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.270 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.270 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.270 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.270 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.270 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.270 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.270 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.270 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.271 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.271 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.271 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.271 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.271 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.271 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.271 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.271 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.271 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.271 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.272 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.272 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.272 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.272 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.272 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.272 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.272 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.272 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.272 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.273 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.273 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.273 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.273 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.273 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.273 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.273 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.273 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.273 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.274 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.274 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.274 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.274 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.274 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.274 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.274 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.274 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.274 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.275 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.275 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.275 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.275 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.275 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.275 INFO:teuthology.orchestra.run.smithi002.stdout: "organization": { 2023-11-16T17:00:03.275 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "organization", 2023-11-16T17:00:03.275 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.275 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.275 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.276 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.276 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.276 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.276 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.276 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.276 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.276 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.276 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.276 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.277 INFO:teuthology.orchestra.run.smithi002.stdout: "proxy": { 2023-11-16T17:00:03.277 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "proxy", 2023-11-16T17:00:03.277 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.277 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.277 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.277 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.277 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.277 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.277 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.277 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.278 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.278 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.278 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.278 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.278 INFO:teuthology.orchestra.run.smithi002.stdout: "url": { 2023-11-16T17:00:03.278 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "url", 2023-11-16T17:00:03.278 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.278 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.278 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.279 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-11-16T17:00:03.279 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.279 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.279 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.279 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.279 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.279 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.279 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.280 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.280 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.280 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.280 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.280 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "test_orchestrator", 2023-11-16T17:00:03.280 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.280 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.280 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.280 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.281 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.281 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.281 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.281 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.281 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.281 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.281 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.281 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.281 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.282 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.282 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.282 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.282 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.282 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.282 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.282 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.282 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.282 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.282 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.283 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.283 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.283 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.283 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.283 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.283 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.283 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.283 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.283 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.284 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.284 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.284 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.284 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.284 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.284 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.284 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.284 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.284 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.284 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.285 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.285 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.285 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.285 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.285 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.285 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.285 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.285 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.285 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.286 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.286 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.286 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.286 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.286 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.286 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.286 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.286 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.286 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.286 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.287 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.287 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.287 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.287 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.287 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.287 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.287 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.287 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.287 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.288 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.288 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.288 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.288 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.288 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.288 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.288 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "volumes", 2023-11-16T17:00:03.288 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.288 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.289 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.289 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.289 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.289 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.289 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.289 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.289 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.289 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.289 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.290 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.290 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.290 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.290 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.290 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.290 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.290 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.290 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.290 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.291 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.291 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.291 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.291 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.291 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.291 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.291 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.291 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.291 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.292 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.292 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.292 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.292 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.292 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.292 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.292 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.292 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.292 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.293 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.293 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.293 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.293 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.293 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.293 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.293 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.293 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.293 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.294 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.294 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.294 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.294 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.294 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.294 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.294 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.294 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.294 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.295 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.295 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.295 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.295 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.295 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.295 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.295 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.295 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.295 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.296 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.296 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.296 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.296 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.296 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.296 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.296 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.296 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.296 INFO:teuthology.orchestra.run.smithi002.stdout: "max_concurrent_clones": { 2023-11-16T17:00:03.296 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_concurrent_clones", 2023-11-16T17:00:03.297 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:03.297 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.297 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.297 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "4", 2023-11-16T17:00:03.297 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.297 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.297 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.297 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Number of asynchronous cloner threads", 2023-11-16T17:00:03.297 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.298 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.298 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.298 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.298 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot_clone_delay": { 2023-11-16T17:00:03.298 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snapshot_clone_delay", 2023-11-16T17:00:03.298 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:03.298 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.298 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.298 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2023-11-16T17:00:03.299 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.299 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.299 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.299 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-11-16T17:00:03.299 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.299 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.299 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.299 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.300 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.300 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.300 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.300 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix", 2023-11-16T17:00:03.300 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2023-11-16T17:00:03.300 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2023-11-16T17:00:03.300 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2023-11-16T17:00:03.300 INFO:teuthology.orchestra.run.smithi002.stdout: "discovery_interval": { 2023-11-16T17:00:03.300 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "discovery_interval", 2023-11-16T17:00:03.301 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2023-11-16T17:00:03.301 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.301 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.301 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "100", 2023-11-16T17:00:03.301 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.301 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.301 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.301 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.301 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.301 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.302 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.302 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.302 INFO:teuthology.orchestra.run.smithi002.stdout: "identifier": { 2023-11-16T17:00:03.302 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "identifier", 2023-11-16T17:00:03.302 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.302 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.302 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.302 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.302 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.303 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.303 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.303 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.303 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.303 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.303 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.303 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.303 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2023-11-16T17:00:03.303 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2023-11-16T17:00:03.304 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2023-11-16T17:00:03.304 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.304 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.304 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2023-11-16T17:00:03.304 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.304 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.304 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.304 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.304 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.304 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.305 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.305 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.305 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2023-11-16T17:00:03.305 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2023-11-16T17:00:03.305 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.305 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.305 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.305 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.305 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.306 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.306 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.306 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.306 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.306 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.306 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.306 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.306 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.306 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.307 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.307 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.307 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.307 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.307 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.307 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2023-11-16T17:00:03.307 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2023-11-16T17:00:03.307 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.307 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.308 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.308 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.308 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.308 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.308 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.308 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.308 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.308 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.309 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.309 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.309 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2023-11-16T17:00:03.309 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2023-11-16T17:00:03.309 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.309 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.309 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.309 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2023-11-16T17:00:03.310 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.310 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.310 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2023-11-16T17:00:03.310 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2023-11-16T17:00:03.310 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2023-11-16T17:00:03.310 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2023-11-16T17:00:03.310 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2023-11-16T17:00:03.310 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2023-11-16T17:00:03.310 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2023-11-16T17:00:03.311 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.311 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.311 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.311 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.311 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.311 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.311 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2023-11-16T17:00:03.311 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2023-11-16T17:00:03.312 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2023-11-16T17:00:03.312 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.312 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2023-11-16T17:00:03.312 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2023-11-16T17:00:03.312 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.312 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.312 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.312 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.312 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.313 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.313 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.313 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.313 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_host": { 2023-11-16T17:00:03.313 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_host", 2023-11-16T17:00:03.313 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.313 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.313 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.313 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2023-11-16T17:00:03.314 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.314 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.314 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.314 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.314 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.314 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.314 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.314 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.314 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_port": { 2023-11-16T17:00:03.315 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_port", 2023-11-16T17:00:03.315 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2023-11-16T17:00:03.315 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.315 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.315 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10051", 2023-11-16T17:00:03.315 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.315 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.315 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.315 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.315 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.316 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.316 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.316 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.316 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_sender": { 2023-11-16T17:00:03.316 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_sender", 2023-11-16T17:00:03.316 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2023-11-16T17:00:03.316 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2023-11-16T17:00:03.316 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2023-11-16T17:00:03.316 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-11-16T17:00:03.317 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2023-11-16T17:00:03.317 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2023-11-16T17:00:03.317 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2023-11-16T17:00:03.317 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2023-11-16T17:00:03.317 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2023-11-16T17:00:03.317 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2023-11-16T17:00:03.317 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2023-11-16T17:00:03.317 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.317 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.318 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.318 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.318 INFO:teuthology.orchestra.run.smithi002.stdout: "services": { 2023-11-16T17:00:03.318 INFO:teuthology.orchestra.run.smithi002.stdout: "dashboard": "https://172.21.15.2:8443/" 2023-11-16T17:00:03.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.318 INFO:teuthology.orchestra.run.smithi002.stdout: "always_on_modules": { 2023-11-16T17:00:03.318 INFO:teuthology.orchestra.run.smithi002.stdout: "octopus": [ 2023-11-16T17:00:03.318 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", 2023-11-16T17:00:03.318 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2023-11-16T17:00:03.319 INFO:teuthology.orchestra.run.smithi002.stdout: "devicehealth", 2023-11-16T17:00:03.319 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator", 2023-11-16T17:00:03.319 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_autoscaler", 2023-11-16T17:00:03.319 INFO:teuthology.orchestra.run.smithi002.stdout: "progress", 2023-11-16T17:00:03.319 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_support", 2023-11-16T17:00:03.319 INFO:teuthology.orchestra.run.smithi002.stdout: "status", 2023-11-16T17:00:03.319 INFO:teuthology.orchestra.run.smithi002.stdout: "telemetry", 2023-11-16T17:00:03.319 INFO:teuthology.orchestra.run.smithi002.stdout: "volumes" 2023-11-16T17:00:03.319 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.320 INFO:teuthology.orchestra.run.smithi002.stdout: "pacific": [ 2023-11-16T17:00:03.320 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", 2023-11-16T17:00:03.320 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2023-11-16T17:00:03.320 INFO:teuthology.orchestra.run.smithi002.stdout: "devicehealth", 2023-11-16T17:00:03.320 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator", 2023-11-16T17:00:03.320 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_autoscaler", 2023-11-16T17:00:03.320 INFO:teuthology.orchestra.run.smithi002.stdout: "progress", 2023-11-16T17:00:03.320 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_support", 2023-11-16T17:00:03.321 INFO:teuthology.orchestra.run.smithi002.stdout: "status", 2023-11-16T17:00:03.321 INFO:teuthology.orchestra.run.smithi002.stdout: "telemetry", 2023-11-16T17:00:03.321 INFO:teuthology.orchestra.run.smithi002.stdout: "volumes" 2023-11-16T17:00:03.321 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.321 INFO:teuthology.orchestra.run.smithi002.stdout: "quincy": [ 2023-11-16T17:00:03.321 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", 2023-11-16T17:00:03.321 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2023-11-16T17:00:03.321 INFO:teuthology.orchestra.run.smithi002.stdout: "devicehealth", 2023-11-16T17:00:03.321 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator", 2023-11-16T17:00:03.322 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_autoscaler", 2023-11-16T17:00:03.322 INFO:teuthology.orchestra.run.smithi002.stdout: "progress", 2023-11-16T17:00:03.322 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_support", 2023-11-16T17:00:03.322 INFO:teuthology.orchestra.run.smithi002.stdout: "status", 2023-11-16T17:00:03.322 INFO:teuthology.orchestra.run.smithi002.stdout: "telemetry", 2023-11-16T17:00:03.322 INFO:teuthology.orchestra.run.smithi002.stdout: "volumes" 2023-11-16T17:00:03.322 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2023-11-16T17:00:03.322 INFO:teuthology.orchestra.run.smithi002.stdout: "reef": [ 2023-11-16T17:00:03.322 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", 2023-11-16T17:00:03.323 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2023-11-16T17:00:03.323 INFO:teuthology.orchestra.run.smithi002.stdout: "devicehealth", 2023-11-16T17:00:03.323 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator", 2023-11-16T17:00:03.323 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_autoscaler", 2023-11-16T17:00:03.323 INFO:teuthology.orchestra.run.smithi002.stdout: "progress", 2023-11-16T17:00:03.323 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_support", 2023-11-16T17:00:03.323 INFO:teuthology.orchestra.run.smithi002.stdout: "status", 2023-11-16T17:00:03.323 INFO:teuthology.orchestra.run.smithi002.stdout: "telemetry", 2023-11-16T17:00:03.323 INFO:teuthology.orchestra.run.smithi002.stdout: "volumes" 2023-11-16T17:00:03.324 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T17:00:03.324 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.324 INFO:teuthology.orchestra.run.smithi002.stdout: "last_failure_osd_epoch": 94, 2023-11-16T17:00:03.324 INFO:teuthology.orchestra.run.smithi002.stdout: "active_clients": [ 2023-11-16T17:00:03.324 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.324 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", 2023-11-16T17:00:03.324 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-16T17:00:03.324 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.325 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-16T17:00:03.325 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2023-11-16T17:00:03.325 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3908718064 2023-11-16T17:00:03.325 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.325 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T17:00:03.325 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.325 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.325 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "libcephsqlite", 2023-11-16T17:00:03.326 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-16T17:00:03.326 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.326 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-16T17:00:03.326 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2023-11-16T17:00:03.326 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3608203952 2023-11-16T17:00:03.326 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.326 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T17:00:03.326 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.327 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.327 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_support", 2023-11-16T17:00:03.327 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-16T17:00:03.327 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.327 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-16T17:00:03.327 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2023-11-16T17:00:03.327 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1735617184 2023-11-16T17:00:03.327 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.327 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T17:00:03.328 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.328 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.328 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snap_schedule", 2023-11-16T17:00:03.328 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-16T17:00:03.328 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.328 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-16T17:00:03.328 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2023-11-16T17:00:03.328 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2568604358 2023-11-16T17:00:03.329 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.329 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T17:00:03.329 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.329 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.329 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snap_schedule", 2023-11-16T17:00:03.329 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-16T17:00:03.329 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.330 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-16T17:00:03.330 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2023-11-16T17:00:03.330 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 980490942 2023-11-16T17:00:03.330 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.330 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T17:00:03.330 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:03.330 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.330 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "volumes", 2023-11-16T17:00:03.331 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2023-11-16T17:00:03.331 INFO:teuthology.orchestra.run.smithi002.stdout: { 2023-11-16T17:00:03.331 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2023-11-16T17:00:03.331 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2023-11-16T17:00:03.331 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1659764962 2023-11-16T17:00:03.331 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.331 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T17:00:03.331 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:03.331 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2023-11-16T17:00:03.332 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T17:00:03.333 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:01.204+0000 7f1cc37ff700 1 -- 172.21.15.2:0/2962512053 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f1ca80a12c0 msgr2=0x7f1ca80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:03.333 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:01.204+0000 7f1cc37ff700 1 --2- 172.21.15.2:0/2962512053 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f1ca80a12c0 0x7f1ca80a3770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc0745e0 tx=0x7f1cac009260 comp rx=0 tx=0).stop 2023-11-16T17:00:03.333 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:01.204+0000 7f1cc37ff700 1 -- 172.21.15.2:0/2962512053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1cbc100550 msgr2=0x7f1cbc072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:03.333 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:01.204+0000 7f1cc37ff700 1 --2- 172.21.15.2:0/2962512053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1cbc100550 0x7f1cbc072df0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1cb80098e0 tx=0x7f1cb800f690 comp rx=0 tx=0).stop 2023-11-16T17:00:03.333 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:01.204+0000 7f1cc37ff700 1 -- 172.21.15.2:0/2962512053 shutdown_connections 2023-11-16T17:00:03.333 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:01.204+0000 7f1cc37ff700 1 --2- 172.21.15.2:0/2962512053 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f1ca80a12c0 0x7f1ca80a3770 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:03.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:01.204+0000 7f1cc37ff700 1 --2- 172.21.15.2:0/2962512053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1cbc100550 0x7f1cbc072df0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:03.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:01.204+0000 7f1cc37ff700 1 --2- 172.21.15.2:0/2962512053 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f1cbc105540 0x7f1cbc1a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:03.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:01.204+0000 7f1cc37ff700 1 --2- 172.21.15.2:0/2962512053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1cbc100ef0 0x7f1cbc073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:03.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:01.204+0000 7f1cc37ff700 1 -- 172.21.15.2:0/2962512053 >> 172.21.15.2:0/2962512053 conn(0x7f1cbc077020 msgr2=0x7f1cbc0fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:03.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:01.204+0000 7f1cc37ff700 1 -- 172.21.15.2:0/2962512053 shutdown_connections 2023-11-16T17:00:03.334 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:01.204+0000 7f1cc37ff700 1 -- 172.21.15.2:0/2962512053 wait complete. 2023-11-16T17:00:03.335 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:00 smithi002 ceph-mon[128712]: pgmap v314: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T17:00:03.335 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:00 smithi002 ceph-mon[128712]: overall HEALTH_OK 2023-11-16T17:00:03.335 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:00 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3803389771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T17:00:03.335 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:00 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/765228245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T17:00:03.335 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:00 smithi155 ceph-mon[131607]: pgmap v314: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T17:00:03.336 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:00 smithi155 ceph-mon[131607]: overall HEALTH_OK 2023-11-16T17:00:03.336 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:00 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3803389771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T17:00:03.336 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:00 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/765228245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T17:00:03.336 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:01 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2962512053' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-11-16T17:00:03.336 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:00 smithi141 ceph-mon[131184]: pgmap v314: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T17:00:03.336 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:00 smithi141 ceph-mon[131184]: overall HEALTH_OK 2023-11-16T17:00:03.337 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:00 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3803389771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T17:00:03.337 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:00 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/765228245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T17:00:03.337 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:01 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2962512053' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-11-16T17:00:03.337 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:02 smithi141 ceph-mon[131184]: pgmap v315: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2023-11-16T17:00:03.343 DEBUG:tasks.check_counter:Getting stats from a 2023-11-16T17:00:03.343 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.a perf dump 2023-11-16T17:00:03.343 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:01 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2962512053' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-11-16T17:00:03.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:02 smithi155 ceph-mon[131607]: pgmap v315: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2023-11-16T17:00:03.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:02 smithi002 ceph-mon[128712]: pgmap v315: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2023-11-16T17:00:04.216 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T17:00:05.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:04 smithi002 ceph-mon[128712]: pgmap v316: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 21 op/s 2023-11-16T17:00:05.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:04 smithi141 ceph-mon[131184]: pgmap v316: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 21 op/s 2023-11-16T17:00:05.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:04 smithi155 ceph-mon[131607]: pgmap v316: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 21 op/s 2023-11-16T17:00:07.196 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T17:00:07.197 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:00:07.197 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 217, 2023-11-16T17:00:07.197 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 207, 2023-11-16T17:00:07.197 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 1420352, 2023-11-16T17:00:07.197 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 117984, 2023-11-16T17:00:07.197 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 4, 2023-11-16T17:00:07.197 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 0, 2023-11-16T17:00:07.198 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.054520524, 2023-11-16T17:00:07.198 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.020868873, 2023-11-16T17:00:07.198 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.033581465, 2023-11-16T17:00:07.198 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.004860334, 2023-11-16T17:00:07.198 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:07.198 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 206, 2023-11-16T17:00:07.199 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.020572857, 2023-11-16T17:00:07.199 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000099868 2023-11-16T17:00:07.199 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:07.199 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:07.199 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T17:00:07.199 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T17:00:07.199 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T17:00:07.200 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:07.200 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 1420352, 2023-11-16T17:00:07.200 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 117984 2023-11-16T17:00:07.200 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:07.200 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:00:07.200 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 2, 2023-11-16T17:00:07.200 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 158, 2023-11-16T17:00:07.201 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 320, 2023-11-16T17:00:07.201 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 179536, 2023-11-16T17:00:07.201 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 5, 2023-11-16T17:00:07.201 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 3, 2023-11-16T17:00:07.201 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.022792623, 2023-11-16T17:00:07.201 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.016327893, 2023-11-16T17:00:07.201 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.002955960, 2023-11-16T17:00:07.201 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:07.202 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:07.202 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 155, 2023-11-16T17:00:07.202 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.019024921, 2023-11-16T17:00:07.202 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000122741 2023-11-16T17:00:07.202 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:07.202 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:07.203 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T17:00:07.203 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T17:00:07.203 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T17:00:07.203 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:07.203 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 320, 2023-11-16T17:00:07.203 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 179536 2023-11-16T17:00:07.203 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:07.204 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:00:07.204 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 4, 2023-11-16T17:00:07.204 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 4, 2023-11-16T17:00:07.204 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 2992, 2023-11-16T17:00:07.204 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 448, 2023-11-16T17:00:07.204 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 2, 2023-11-16T17:00:07.204 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 1, 2023-11-16T17:00:07.204 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.001421094, 2023-11-16T17:00:07.205 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.000069038, 2023-11-16T17:00:07.205 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.001307074, 2023-11-16T17:00:07.205 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:07.205 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:07.205 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 3, 2023-11-16T17:00:07.205 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.001356577, 2023-11-16T17:00:07.205 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000452192 2023-11-16T17:00:07.205 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:07.206 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:07.206 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T17:00:07.206 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T17:00:07.206 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T17:00:07.206 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:07.206 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-11-16T17:00:07.206 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 448 2023-11-16T17:00:07.207 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:07.207 INFO:teuthology.orchestra.run.smithi002.stdout: "cct": { 2023-11-16T17:00:07.207 INFO:teuthology.orchestra.run.smithi002.stdout: "total_workers": 0, 2023-11-16T17:00:07.207 INFO:teuthology.orchestra.run.smithi002.stdout: "unhealthy_workers": 0 2023-11-16T17:00:07.207 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:07.207 INFO:teuthology.orchestra.run.smithi002.stdout: "mempool": { 2023-11-16T17:00:07.207 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:00:07.208 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_items": 0, 2023-11-16T17:00:07.208 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:00:07.208 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:00:07.208 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:00:07.208 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:00:07.208 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:00:07.208 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:00:07.209 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:00:07.209 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:00:07.209 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:00:07.209 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:00:07.209 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:00:07.209 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:00:07.209 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:00:07.210 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_items": 0, 2023-11-16T17:00:07.210 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:00:07.210 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_items": 0, 2023-11-16T17:00:07.210 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:00:07.210 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:00:07.210 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:00:07.210 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:00:07.211 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:00:07.211 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:00:07.211 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:00:07.211 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_items": 0, 2023-11-16T17:00:07.211 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:00:07.211 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:00:07.211 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:00:07.212 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_items": 0, 2023-11-16T17:00:07.212 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_bytes": 0, 2023-11-16T17:00:07.212 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_items": 0, 2023-11-16T17:00:07.212 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:00:07.212 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:00:07.212 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:00:07.212 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:00:07.213 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_bytes": 148994, 2023-11-16T17:00:07.213 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_items": 31, 2023-11-16T17:00:07.213 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:00:07.213 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_items": 0, 2023-11-16T17:00:07.213 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bytes": 0, 2023-11-16T17:00:07.213 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_items": 0, 2023-11-16T17:00:07.213 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:00:07.214 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, 2023-11-16T17:00:07.214 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:00:07.214 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2023-11-16T17:00:07.214 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_bytes": 0, 2023-11-16T17:00:07.214 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_items": 0, 2023-11-16T17:00:07.214 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:00:07.214 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:00:07.214 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_bytes": 0, 2023-11-16T17:00:07.215 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_items": 0, 2023-11-16T17:00:07.215 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_bytes": 928, 2023-11-16T17:00:07.215 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_items": 2, 2023-11-16T17:00:07.215 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_bytes": 0, 2023-11-16T17:00:07.215 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_items": 0, 2023-11-16T17:00:07.215 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_bytes": 0, 2023-11-16T17:00:07.215 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_items": 0 2023-11-16T17:00:07.216 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:07.216 INFO:teuthology.orchestra.run.smithi002.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:00:07.216 INFO:teuthology.orchestra.run.smithi002.stdout: "val": 0, 2023-11-16T17:00:07.216 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 104857600, 2023-11-16T17:00:07.216 INFO:teuthology.orchestra.run.smithi002.stdout: "get_started": 0, 2023-11-16T17:00:07.216 INFO:teuthology.orchestra.run.smithi002.stdout: "get": 219, 2023-11-16T17:00:07.216 INFO:teuthology.orchestra.run.smithi002.stdout: "get_sum": 1389719, 2023-11-16T17:00:07.216 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:07.217 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_success": 219, 2023-11-16T17:00:07.217 INFO:teuthology.orchestra.run.smithi002.stdout: "take": 0, 2023-11-16T17:00:07.217 INFO:teuthology.orchestra.run.smithi002.stdout: "take_sum": 0, 2023-11-16T17:00:07.217 INFO:teuthology.orchestra.run.smithi002.stdout: "put": 219, 2023-11-16T17:00:07.217 INFO:teuthology.orchestra.run.smithi002.stdout: "put_sum": 1389719, 2023-11-16T17:00:07.217 INFO:teuthology.orchestra.run.smithi002.stdout: "wait": { 2023-11-16T17:00:07.217 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T17:00:07.218 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T17:00:07.218 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T17:00:07.218 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:07.218 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:07.218 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T17:00:07.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:06 smithi141 ceph-mon[131184]: pgmap v317: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 21 op/s 2023-11-16T17:00:07.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:06 smithi155 ceph-mon[131607]: pgmap v317: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 21 op/s 2023-11-16T17:00:07.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:06 smithi002 ceph-mon[128712]: pgmap v317: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 21 op/s 2023-11-16T17:00:07.721 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2023-11-16T17:00:07.721 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2023-11-16T17:00:07.721 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2023-11-16T17:00:07.721 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2023-11-16T17:00:07.721 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2023-11-16T17:00:07.721 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2023-11-16T17:00:07.722 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2023-11-16T17:00:07.722 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2023-11-16T17:00:07.722 DEBUG:tasks.check_counter:Getting stats from d 2023-11-16T17:00:07.722 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.d perf dump 2023-11-16T17:00:08.585 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T17:00:09.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:08 smithi002 ceph-mon[128712]: pgmap v318: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2023-11-16T17:00:09.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:08 smithi141 ceph-mon[131184]: pgmap v318: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2023-11-16T17:00:09.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:08 smithi155 ceph-mon[131607]: pgmap v318: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2023-11-16T17:00:10.099 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T17:00:10.100 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:00:10.100 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 0, 2023-11-16T17:00:10.100 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 2, 2023-11-16T17:00:10.100 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 0, 2023-11-16T17:00:10.100 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 0, 2023-11-16T17:00:10.100 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 4, 2023-11-16T17:00:10.101 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 0, 2023-11-16T17:00:10.101 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.003589601, 2023-11-16T17:00:10.101 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.000000000, 2023-11-16T17:00:10.101 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.000714110, 2023-11-16T17:00:10.101 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:10.101 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:10.101 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T17:00:10.101 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T17:00:10.102 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T17:00:10.102 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:10.102 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:10.102 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T17:00:10.102 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T17:00:10.102 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T17:00:10.103 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:10.103 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T17:00:10.103 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T17:00:10.103 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:10.103 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:00:10.103 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 6, 2023-11-16T17:00:10.103 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 163, 2023-11-16T17:00:10.103 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 3312, 2023-11-16T17:00:10.104 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 180512, 2023-11-16T17:00:10.104 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 5, 2023-11-16T17:00:10.104 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 3, 2023-11-16T17:00:10.104 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.020607086, 2023-11-16T17:00:10.104 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.013710234, 2023-11-16T17:00:10.104 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.003472792, 2023-11-16T17:00:10.104 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:10.105 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:10.105 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 161, 2023-11-16T17:00:10.105 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.020554218, 2023-11-16T17:00:10.105 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000127665 2023-11-16T17:00:10.105 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:10.105 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:10.105 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T17:00:10.106 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T17:00:10.106 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T17:00:10.106 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:10.106 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 3312, 2023-11-16T17:00:10.106 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 180512 2023-11-16T17:00:10.106 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:10.106 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:00:10.107 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 222, 2023-11-16T17:00:10.107 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 212, 2023-11-16T17:00:10.107 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 1421712, 2023-11-16T17:00:10.107 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 120944, 2023-11-16T17:00:10.107 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 2, 2023-11-16T17:00:10.107 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 1, 2023-11-16T17:00:10.107 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.054928656, 2023-11-16T17:00:10.108 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.020854483, 2023-11-16T17:00:10.108 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.038371928, 2023-11-16T17:00:10.108 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.004972309, 2023-11-16T17:00:10.108 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:10.108 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 211, 2023-11-16T17:00:10.108 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.020976593, 2023-11-16T17:00:10.108 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000099415 2023-11-16T17:00:10.109 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:10.109 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:10.109 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T17:00:10.109 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T17:00:10.109 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T17:00:10.109 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:10.110 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 1421712, 2023-11-16T17:00:10.110 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 120944 2023-11-16T17:00:10.110 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:10.110 INFO:teuthology.orchestra.run.smithi002.stdout: "cct": { 2023-11-16T17:00:10.110 INFO:teuthology.orchestra.run.smithi002.stdout: "total_workers": 0, 2023-11-16T17:00:10.110 INFO:teuthology.orchestra.run.smithi002.stdout: "unhealthy_workers": 0 2023-11-16T17:00:10.110 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:10.110 INFO:teuthology.orchestra.run.smithi002.stdout: "mempool": { 2023-11-16T17:00:10.111 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:00:10.111 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_items": 0, 2023-11-16T17:00:10.111 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:00:10.111 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:00:10.111 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:00:10.111 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:00:10.111 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:00:10.112 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:00:10.112 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:00:10.112 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:00:10.112 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:00:10.112 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:00:10.112 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:00:10.112 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:00:10.112 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:00:10.113 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_items": 0, 2023-11-16T17:00:10.113 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:00:10.113 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_items": 0, 2023-11-16T17:00:10.113 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:00:10.113 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:00:10.113 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:00:10.113 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:00:10.113 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:00:10.114 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:00:10.114 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:00:10.114 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_items": 0, 2023-11-16T17:00:10.114 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:00:10.114 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:00:10.114 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:00:10.114 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_items": 0, 2023-11-16T17:00:10.115 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_bytes": 0, 2023-11-16T17:00:10.115 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_items": 0, 2023-11-16T17:00:10.115 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:00:10.115 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:00:10.115 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:00:10.115 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:00:10.115 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_bytes": 148977, 2023-11-16T17:00:10.115 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_items": 30, 2023-11-16T17:00:10.116 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:00:10.116 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_items": 0, 2023-11-16T17:00:10.116 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bytes": 0, 2023-11-16T17:00:10.116 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_items": 0, 2023-11-16T17:00:10.116 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:00:10.116 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, 2023-11-16T17:00:10.116 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:00:10.116 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2023-11-16T17:00:10.117 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_bytes": 0, 2023-11-16T17:00:10.117 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_items": 0, 2023-11-16T17:00:10.117 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:00:10.117 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:00:10.117 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_bytes": 0, 2023-11-16T17:00:10.117 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_items": 0, 2023-11-16T17:00:10.117 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_bytes": 928, 2023-11-16T17:00:10.118 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_items": 2, 2023-11-16T17:00:10.118 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_bytes": 0, 2023-11-16T17:00:10.118 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_items": 0, 2023-11-16T17:00:10.118 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_bytes": 0, 2023-11-16T17:00:10.118 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_items": 0 2023-11-16T17:00:10.118 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:10.119 INFO:teuthology.orchestra.run.smithi002.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:00:10.119 INFO:teuthology.orchestra.run.smithi002.stdout: "val": 0, 2023-11-16T17:00:10.119 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 104857600, 2023-11-16T17:00:10.119 INFO:teuthology.orchestra.run.smithi002.stdout: "get_started": 0, 2023-11-16T17:00:10.119 INFO:teuthology.orchestra.run.smithi002.stdout: "get": 224, 2023-11-16T17:00:10.119 INFO:teuthology.orchestra.run.smithi002.stdout: "get_sum": 1390369, 2023-11-16T17:00:10.120 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:10.120 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_success": 224, 2023-11-16T17:00:10.120 INFO:teuthology.orchestra.run.smithi002.stdout: "take": 0, 2023-11-16T17:00:10.120 INFO:teuthology.orchestra.run.smithi002.stdout: "take_sum": 0, 2023-11-16T17:00:10.120 INFO:teuthology.orchestra.run.smithi002.stdout: "put": 224, 2023-11-16T17:00:10.120 INFO:teuthology.orchestra.run.smithi002.stdout: "put_sum": 1390369, 2023-11-16T17:00:10.120 INFO:teuthology.orchestra.run.smithi002.stdout: "wait": { 2023-11-16T17:00:10.120 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T17:00:10.121 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T17:00:10.121 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T17:00:10.121 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:10.121 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:10.121 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T17:00:10.583 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2023-11-16T17:00:10.583 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2023-11-16T17:00:10.583 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2023-11-16T17:00:10.583 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2023-11-16T17:00:10.583 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2023-11-16T17:00:10.584 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2023-11-16T17:00:10.584 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2023-11-16T17:00:10.584 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2023-11-16T17:00:10.584 DEBUG:tasks.check_counter:Getting stats from g 2023-11-16T17:00:10.584 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.g perf dump 2023-11-16T17:00:11.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:10 smithi141 ceph-mon[131184]: pgmap v319: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2023-11-16T17:00:11.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:10 smithi155 ceph-mon[131607]: pgmap v319: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2023-11-16T17:00:11.411 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:10 smithi002 ceph-mon[128712]: pgmap v319: 129 pgs: 129 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2023-11-16T17:00:11.624 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T17:00:12.267 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:12 smithi141 ceph-mon[131184]: osdmap e104: 12 total, 12 up, 12 in 2023-11-16T17:00:12.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:12 smithi155 ceph-mon[131607]: osdmap e104: 12 total, 12 up, 12 in 2023-11-16T17:00:12.392 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:12 smithi002 ceph-mon[128712]: osdmap e104: 12 total, 12 up, 12 in 2023-11-16T17:00:13.157 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:13 smithi002 ceph-mon[128712]: pgmap v321: 129 pgs: 14 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 64 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-11-16T17:00:13.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:13 smithi141 ceph-mon[131184]: pgmap v321: 129 pgs: 14 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 64 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-11-16T17:00:13.371 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:13 smithi155 ceph-mon[131607]: pgmap v321: 129 pgs: 14 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 64 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-11-16T17:00:13.968 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T17:00:13.969 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:00:13.969 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 223, 2023-11-16T17:00:13.969 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 211, 2023-11-16T17:00:13.969 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 1423888, 2023-11-16T17:00:13.970 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 119616, 2023-11-16T17:00:13.970 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 4, 2023-11-16T17:00:13.970 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 0, 2023-11-16T17:00:13.970 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.054492601, 2023-11-16T17:00:13.970 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.018510680, 2023-11-16T17:00:13.971 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.035017952, 2023-11-16T17:00:13.971 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.005620469, 2023-11-16T17:00:13.971 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:13.971 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 211, 2023-11-16T17:00:13.971 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.016448021, 2023-11-16T17:00:13.971 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000077952 2023-11-16T17:00:13.971 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:13.972 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:13.972 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T17:00:13.972 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T17:00:13.972 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T17:00:13.972 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:13.972 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 1423888, 2023-11-16T17:00:13.973 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 119616 2023-11-16T17:00:13.973 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:13.973 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:00:13.973 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 2, 2023-11-16T17:00:13.973 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 160, 2023-11-16T17:00:13.973 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 320, 2023-11-16T17:00:13.973 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 179888, 2023-11-16T17:00:13.974 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 5, 2023-11-16T17:00:13.974 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 3, 2023-11-16T17:00:13.974 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.021014677, 2023-11-16T17:00:13.974 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.014492127, 2023-11-16T17:00:13.974 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.002756030, 2023-11-16T17:00:13.974 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:13.974 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:13.975 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 157, 2023-11-16T17:00:13.975 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.020907008, 2023-11-16T17:00:13.975 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000133165 2023-11-16T17:00:13.975 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:13.975 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:13.975 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T17:00:13.976 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T17:00:13.976 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T17:00:13.976 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:13.976 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 320, 2023-11-16T17:00:13.976 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 179888 2023-11-16T17:00:13.976 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:13.976 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:00:13.976 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 0, 2023-11-16T17:00:13.976 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 2, 2023-11-16T17:00:13.977 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 0, 2023-11-16T17:00:13.977 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 0, 2023-11-16T17:00:13.977 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 2, 2023-11-16T17:00:13.977 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 1, 2023-11-16T17:00:13.977 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.000941728, 2023-11-16T17:00:13.977 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.000000000, 2023-11-16T17:00:13.977 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.000812928, 2023-11-16T17:00:13.978 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:13.978 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:13.978 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T17:00:13.978 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T17:00:13.978 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T17:00:13.978 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:13.979 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:13.979 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T17:00:13.979 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T17:00:13.979 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T17:00:13.979 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:13.979 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T17:00:13.980 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T17:00:13.980 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:13.980 INFO:teuthology.orchestra.run.smithi002.stdout: "cct": { 2023-11-16T17:00:13.980 INFO:teuthology.orchestra.run.smithi002.stdout: "total_workers": 0, 2023-11-16T17:00:13.980 INFO:teuthology.orchestra.run.smithi002.stdout: "unhealthy_workers": 0 2023-11-16T17:00:13.980 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:13.980 INFO:teuthology.orchestra.run.smithi002.stdout: "mempool": { 2023-11-16T17:00:13.981 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:00:13.981 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_items": 0, 2023-11-16T17:00:13.981 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:00:13.981 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:00:13.981 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:00:13.981 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:00:13.981 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:00:13.982 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:00:13.982 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:00:13.982 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:00:13.982 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:00:13.982 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:00:13.982 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:00:13.982 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:00:13.983 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:00:13.983 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_items": 0, 2023-11-16T17:00:13.983 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:00:13.983 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_items": 0, 2023-11-16T17:00:13.983 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:00:13.983 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:00:13.983 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:00:13.983 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:00:13.984 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:00:13.984 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:00:13.984 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:00:13.984 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_items": 0, 2023-11-16T17:00:13.984 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:00:13.984 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:00:13.984 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:00:13.985 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_items": 0, 2023-11-16T17:00:13.985 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_bytes": 0, 2023-11-16T17:00:13.985 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_items": 0, 2023-11-16T17:00:13.985 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:00:13.985 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:00:13.985 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:00:13.985 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:00:13.985 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_bytes": 148977, 2023-11-16T17:00:13.986 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_items": 30, 2023-11-16T17:00:13.986 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:00:13.986 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_items": 0, 2023-11-16T17:00:13.986 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bytes": 0, 2023-11-16T17:00:13.986 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_items": 0, 2023-11-16T17:00:13.986 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:00:13.986 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, 2023-11-16T17:00:13.987 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:00:13.987 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2023-11-16T17:00:13.987 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_bytes": 0, 2023-11-16T17:00:13.987 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_items": 0, 2023-11-16T17:00:13.987 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:00:13.987 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:00:13.987 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_bytes": 0, 2023-11-16T17:00:13.987 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_items": 0, 2023-11-16T17:00:13.988 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_bytes": 928, 2023-11-16T17:00:13.988 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_items": 2, 2023-11-16T17:00:13.988 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_bytes": 0, 2023-11-16T17:00:13.988 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_items": 0, 2023-11-16T17:00:13.988 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_bytes": 0, 2023-11-16T17:00:13.988 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_items": 0 2023-11-16T17:00:13.988 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:13.989 INFO:teuthology.orchestra.run.smithi002.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:00:13.989 INFO:teuthology.orchestra.run.smithi002.stdout: "val": 0, 2023-11-16T17:00:13.989 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 104857600, 2023-11-16T17:00:13.989 INFO:teuthology.orchestra.run.smithi002.stdout: "get_started": 0, 2023-11-16T17:00:13.989 INFO:teuthology.orchestra.run.smithi002.stdout: "get": 221, 2023-11-16T17:00:13.989 INFO:teuthology.orchestra.run.smithi002.stdout: "get_sum": 1389979, 2023-11-16T17:00:13.989 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:13.990 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_success": 221, 2023-11-16T17:00:13.990 INFO:teuthology.orchestra.run.smithi002.stdout: "take": 0, 2023-11-16T17:00:13.990 INFO:teuthology.orchestra.run.smithi002.stdout: "take_sum": 0, 2023-11-16T17:00:13.990 INFO:teuthology.orchestra.run.smithi002.stdout: "put": 221, 2023-11-16T17:00:13.990 INFO:teuthology.orchestra.run.smithi002.stdout: "put_sum": 1389979, 2023-11-16T17:00:13.990 INFO:teuthology.orchestra.run.smithi002.stdout: "wait": { 2023-11-16T17:00:13.991 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T17:00:13.991 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T17:00:13.991 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T17:00:13.991 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:13.991 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:13.991 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T17:00:14.577 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2023-11-16T17:00:14.577 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2023-11-16T17:00:14.577 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2023-11-16T17:00:14.577 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2023-11-16T17:00:14.577 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2023-11-16T17:00:14.577 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2023-11-16T17:00:14.577 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2023-11-16T17:00:14.577 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2023-11-16T17:00:14.578 DEBUG:tasks.check_counter:Getting stats from j 2023-11-16T17:00:14.578 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.j perf dump 2023-11-16T17:00:15.370 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:15 smithi155 ceph-mon[131607]: pgmap v322: 129 pgs: 16 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 58 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-11-16T17:00:15.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:15 smithi002 ceph-mon[128712]: pgmap v322: 129 pgs: 16 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 58 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-11-16T17:00:15.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:15 smithi141 ceph-mon[131184]: pgmap v322: 129 pgs: 16 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 58 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-11-16T17:00:15.917 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.a/config 2023-11-16T17:00:17.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:17 smithi155 ceph-mon[131607]: pgmap v323: 129 pgs: 16 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 58 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-11-16T17:00:17.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:17 smithi002 ceph-mon[128712]: pgmap v323: 129 pgs: 16 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 58 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-11-16T17:00:17.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:17 smithi141 ceph-mon[131184]: pgmap v323: 129 pgs: 16 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 58 active+clean; 607 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-11-16T17:00:18.156 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2023-11-16T17:00:18.156 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:00:18.157 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 226, 2023-11-16T17:00:18.157 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 216, 2023-11-16T17:00:18.157 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 1422864, 2023-11-16T17:00:18.157 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 122880, 2023-11-16T17:00:18.157 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 4, 2023-11-16T17:00:18.157 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 0, 2023-11-16T17:00:18.157 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.055556094, 2023-11-16T17:00:18.157 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.019878626, 2023-11-16T17:00:18.158 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.036870316, 2023-11-16T17:00:18.158 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.005017390, 2023-11-16T17:00:18.158 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:18.158 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 215, 2023-11-16T17:00:18.158 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.025191875, 2023-11-16T17:00:18.158 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000117171 2023-11-16T17:00:18.158 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:18.159 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:18.159 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T17:00:18.159 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T17:00:18.159 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T17:00:18.159 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:18.159 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 1422864, 2023-11-16T17:00:18.159 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 122880 2023-11-16T17:00:18.159 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:18.160 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:00:18.160 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 2, 2023-11-16T17:00:18.160 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 164, 2023-11-16T17:00:18.160 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 320, 2023-11-16T17:00:18.160 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 180592, 2023-11-16T17:00:18.160 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 5, 2023-11-16T17:00:18.160 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 3, 2023-11-16T17:00:18.161 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.022123423, 2023-11-16T17:00:18.161 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.015762724, 2023-11-16T17:00:18.161 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.002933643, 2023-11-16T17:00:18.161 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:18.161 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:18.161 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 161, 2023-11-16T17:00:18.161 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.020832988, 2023-11-16T17:00:18.161 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000129397 2023-11-16T17:00:18.161 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:18.161 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:18.162 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T17:00:18.162 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T17:00:18.162 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T17:00:18.162 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:18.162 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 320, 2023-11-16T17:00:18.162 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 180592 2023-11-16T17:00:18.162 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:18.162 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:00:18.163 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 4, 2023-11-16T17:00:18.163 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 4, 2023-11-16T17:00:18.163 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 2992, 2023-11-16T17:00:18.163 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 448, 2023-11-16T17:00:18.163 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 2, 2023-11-16T17:00:18.163 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 1, 2023-11-16T17:00:18.163 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.001082708, 2023-11-16T17:00:18.163 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.000052851, 2023-11-16T17:00:18.163 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.000977886, 2023-11-16T17:00:18.164 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:18.164 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:18.164 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 3, 2023-11-16T17:00:18.164 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.001353059, 2023-11-16T17:00:18.164 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000451019 2023-11-16T17:00:18.164 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:18.164 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:18.164 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T17:00:18.164 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T17:00:18.165 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T17:00:18.165 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:18.165 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-11-16T17:00:18.165 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 448 2023-11-16T17:00:18.165 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:18.165 INFO:teuthology.orchestra.run.smithi002.stdout: "cct": { 2023-11-16T17:00:18.165 INFO:teuthology.orchestra.run.smithi002.stdout: "total_workers": 0, 2023-11-16T17:00:18.165 INFO:teuthology.orchestra.run.smithi002.stdout: "unhealthy_workers": 0 2023-11-16T17:00:18.165 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:18.166 INFO:teuthology.orchestra.run.smithi002.stdout: "mempool": { 2023-11-16T17:00:18.166 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:00:18.166 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_items": 0, 2023-11-16T17:00:18.166 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:00:18.166 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:00:18.166 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:00:18.166 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:00:18.166 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:00:18.167 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:00:18.167 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:00:18.167 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:00:18.167 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:00:18.167 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:00:18.167 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:00:18.167 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:00:18.167 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:00:18.167 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_items": 0, 2023-11-16T17:00:18.168 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:00:18.168 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_items": 0, 2023-11-16T17:00:18.168 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:00:18.168 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:00:18.168 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:00:18.168 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:00:18.168 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:00:18.168 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:00:18.168 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:00:18.169 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_items": 0, 2023-11-16T17:00:18.169 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:00:18.169 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:00:18.169 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:00:18.169 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_items": 0, 2023-11-16T17:00:18.169 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_bytes": 0, 2023-11-16T17:00:18.169 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_items": 0, 2023-11-16T17:00:18.170 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:00:18.170 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:00:18.170 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:00:18.170 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:00:18.170 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_bytes": 148994, 2023-11-16T17:00:18.170 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_items": 31, 2023-11-16T17:00:18.170 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:00:18.170 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_items": 0, 2023-11-16T17:00:18.171 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bytes": 0, 2023-11-16T17:00:18.171 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_items": 0, 2023-11-16T17:00:18.171 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:00:18.171 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, 2023-11-16T17:00:18.171 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:00:18.171 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2023-11-16T17:00:18.171 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_bytes": 0, 2023-11-16T17:00:18.171 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_items": 0, 2023-11-16T17:00:18.171 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:00:18.172 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:00:18.172 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_bytes": 0, 2023-11-16T17:00:18.172 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_items": 0, 2023-11-16T17:00:18.172 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_bytes": 928, 2023-11-16T17:00:18.172 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_items": 2, 2023-11-16T17:00:18.172 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_bytes": 0, 2023-11-16T17:00:18.172 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_items": 0, 2023-11-16T17:00:18.172 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_bytes": 0, 2023-11-16T17:00:18.173 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_items": 0 2023-11-16T17:00:18.173 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2023-11-16T17:00:18.173 INFO:teuthology.orchestra.run.smithi002.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:00:18.173 INFO:teuthology.orchestra.run.smithi002.stdout: "val": 0, 2023-11-16T17:00:18.173 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 104857600, 2023-11-16T17:00:18.173 INFO:teuthology.orchestra.run.smithi002.stdout: "get_started": 0, 2023-11-16T17:00:18.173 INFO:teuthology.orchestra.run.smithi002.stdout: "get": 228, 2023-11-16T17:00:18.173 INFO:teuthology.orchestra.run.smithi002.stdout: "get_sum": 1390953, 2023-11-16T17:00:18.174 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:18.174 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_success": 228, 2023-11-16T17:00:18.174 INFO:teuthology.orchestra.run.smithi002.stdout: "take": 0, 2023-11-16T17:00:18.174 INFO:teuthology.orchestra.run.smithi002.stdout: "take_sum": 0, 2023-11-16T17:00:18.174 INFO:teuthology.orchestra.run.smithi002.stdout: "put": 228, 2023-11-16T17:00:18.174 INFO:teuthology.orchestra.run.smithi002.stdout: "put_sum": 1390953, 2023-11-16T17:00:18.174 INFO:teuthology.orchestra.run.smithi002.stdout: "wait": { 2023-11-16T17:00:18.175 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2023-11-16T17:00:18.175 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2023-11-16T17:00:18.175 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2023-11-16T17:00:18.175 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:18.175 INFO:teuthology.orchestra.run.smithi002.stdout: } 2023-11-16T17:00:18.175 INFO:teuthology.orchestra.run.smithi002.stdout:} 2023-11-16T17:00:18.684 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2023-11-16T17:00:18.684 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2023-11-16T17:00:18.685 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2023-11-16T17:00:18.685 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2023-11-16T17:00:18.685 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2023-11-16T17:00:18.685 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2023-11-16T17:00:18.685 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2023-11-16T17:00:18.685 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2023-11-16T17:00:18.685 DEBUG:tasks.check_counter:Getting stats from b 2023-11-16T17:00:18.685 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.b perf dump 2023-11-16T17:00:19.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:19 smithi155 ceph-mon[131607]: pgmap v324: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 607 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T17:00:19.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:19 smithi002 ceph-mon[128712]: pgmap v324: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 607 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T17:00:19.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:19 smithi141 ceph-mon[131184]: pgmap v324: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 607 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T17:00:19.527 INFO:teuthology.orchestra.run.smithi141.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T17:00:20.950 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-11-16T17:00:20.950 INFO:teuthology.orchestra.run.smithi141.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:00:20.950 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_messages": 37761, 2023-11-16T17:00:20.951 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages": 36756, 2023-11-16T17:00:20.951 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_bytes": 22737925, 2023-11-16T17:00:20.951 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_bytes": 227758686, 2023-11-16T17:00:20.951 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_created_connections": 22, 2023-11-16T17:00:20.951 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_active_connections": 5, 2023-11-16T17:00:20.951 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_total_time": 2.650820174, 2023-11-16T17:00:20.951 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_send_time": 0.968378852, 2023-11-16T17:00:20.951 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_recv_time": 0.716486404, 2023-11-16T17:00:20.952 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_fast_dispatch_time": 1.004465255, 2023-11-16T17:00:20.952 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:20.952 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 36754, 2023-11-16T17:00:20.952 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 52.895416643, 2023-11-16T17:00:20.952 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.001439174 2023-11-16T17:00:20.952 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.952 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:20.952 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 19260, 2023-11-16T17:00:20.952 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.004933342, 2023-11-16T17:00:20.953 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000256 2023-11-16T17:00:20.953 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.953 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_encrypted_bytes": 160, 2023-11-16T17:00:20.953 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_encrypted_bytes": 306864 2023-11-16T17:00:20.953 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.953 INFO:teuthology.orchestra.run.smithi141.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:00:20.953 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_messages": 113926, 2023-11-16T17:00:20.953 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages": 185906, 2023-11-16T17:00:20.954 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_bytes": 42651564, 2023-11-16T17:00:20.954 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_bytes": 194412915, 2023-11-16T17:00:20.954 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_created_connections": 53, 2023-11-16T17:00:20.954 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_active_connections": 8, 2023-11-16T17:00:20.954 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_total_time": 6.388672205, 2023-11-16T17:00:20.954 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_send_time": 2.506389106, 2023-11-16T17:00:20.954 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_recv_time": 212.232824489, 2023-11-16T17:00:20.954 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_fast_dispatch_time": 1.079646181, 2023-11-16T17:00:20.954 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:20.955 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 185903, 2023-11-16T17:00:20.955 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 124.320484014, 2023-11-16T17:00:20.955 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000668738 2023-11-16T17:00:20.955 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.955 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:20.955 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 194836, 2023-11-16T17:00:20.955 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.032331126, 2023-11-16T17:00:20.955 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000165 2023-11-16T17:00:20.955 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.956 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_encrypted_bytes": 139968, 2023-11-16T17:00:20.956 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_encrypted_bytes": 169696 2023-11-16T17:00:20.956 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.956 INFO:teuthology.orchestra.run.smithi141.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:00:20.956 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_messages": 37222, 2023-11-16T17:00:20.956 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages": 37583, 2023-11-16T17:00:20.956 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_bytes": 23442403, 2023-11-16T17:00:20.956 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_bytes": 129647475, 2023-11-16T17:00:20.957 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_created_connections": 13, 2023-11-16T17:00:20.957 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_active_connections": 7, 2023-11-16T17:00:20.957 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_total_time": 2.466748619, 2023-11-16T17:00:20.957 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_send_time": 0.703537941, 2023-11-16T17:00:20.957 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_recv_time": 0.671368495, 2023-11-16T17:00:20.957 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_fast_dispatch_time": 1.088607252, 2023-11-16T17:00:20.957 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:20.957 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 37583, 2023-11-16T17:00:20.957 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 48.137202570, 2023-11-16T17:00:20.958 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.001280823 2023-11-16T17:00:20.958 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.958 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:20.958 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 16073, 2023-11-16T17:00:20.958 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.002747590, 2023-11-16T17:00:20.958 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000170 2023-11-16T17:00:20.958 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.958 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_encrypted_bytes": 1872208, 2023-11-16T17:00:20.958 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_encrypted_bytes": 230848 2023-11-16T17:00:20.959 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.959 INFO:teuthology.orchestra.run.smithi141.stdout: "cct": { 2023-11-16T17:00:20.959 INFO:teuthology.orchestra.run.smithi141.stdout: "total_workers": 1, 2023-11-16T17:00:20.959 INFO:teuthology.orchestra.run.smithi141.stdout: "unhealthy_workers": 0 2023-11-16T17:00:20.959 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.959 INFO:teuthology.orchestra.run.smithi141.stdout: "finisher-MDSRank": { 2023-11-16T17:00:20.959 INFO:teuthology.orchestra.run.smithi141.stdout: "queue_len": 0, 2023-11-16T17:00:20.959 INFO:teuthology.orchestra.run.smithi141.stdout: "complete_latency": { 2023-11-16T17:00:20.960 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 2764, 2023-11-16T17:00:20.960 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 71.656824636, 2023-11-16T17:00:20.960 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.025925045 2023-11-16T17:00:20.960 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:20.960 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.960 INFO:teuthology.orchestra.run.smithi141.stdout: "finisher-PurgeQueue": { 2023-11-16T17:00:20.960 INFO:teuthology.orchestra.run.smithi141.stdout: "queue_len": 0, 2023-11-16T17:00:20.960 INFO:teuthology.orchestra.run.smithi141.stdout: "complete_latency": { 2023-11-16T17:00:20.960 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 27591, 2023-11-16T17:00:20.961 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 7.838300101, 2023-11-16T17:00:20.961 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000284089 2023-11-16T17:00:20.961 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:20.961 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.961 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": { 2023-11-16T17:00:20.961 INFO:teuthology.orchestra.run.smithi141.stdout: "request": 50430, 2023-11-16T17:00:20.961 INFO:teuthology.orchestra.run.smithi141.stdout: "reply": 42626, 2023-11-16T17:00:20.961 INFO:teuthology.orchestra.run.smithi141.stdout: "reply_latency": { 2023-11-16T17:00:20.961 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 42626, 2023-11-16T17:00:20.962 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 1987.947120992, 2023-11-16T17:00:20.962 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.046636961 2023-11-16T17:00:20.962 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.962 INFO:teuthology.orchestra.run.smithi141.stdout: "slow_reply": 0, 2023-11-16T17:00:20.962 INFO:teuthology.orchestra.run.smithi141.stdout: "forward": 7804, 2023-11-16T17:00:20.962 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_fetch_complete": 430, 2023-11-16T17:00:20.962 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_fetch_keys": 0, 2023-11-16T17:00:20.962 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_commit": 11632, 2023-11-16T17:00:20.963 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_split": 142, 2023-11-16T17:00:20.963 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_merge": 3, 2023-11-16T17:00:20.963 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes": 41054, 2023-11-16T17:00:20.963 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes_top": 28724, 2023-11-16T17:00:20.963 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes_bottom": 12318, 2023-11-16T17:00:20.963 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes_pin_tail": 12, 2023-11-16T17:00:20.963 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes_pinned": 17, 2023-11-16T17:00:20.963 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes_expired": 10910, 2023-11-16T17:00:20.963 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes_with_caps": 1, 2023-11-16T17:00:20.964 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": 1, 2023-11-16T17:00:20.964 INFO:teuthology.orchestra.run.smithi141.stdout: "subtrees": 6, 2023-11-16T17:00:20.964 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse": 59645, 2023-11-16T17:00:20.964 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse_hit": 51693, 2023-11-16T17:00:20.964 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse_forward": 7598, 2023-11-16T17:00:20.964 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse_discover": 3, 2023-11-16T17:00:20.964 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse_dir_fetch": 47, 2023-11-16T17:00:20.964 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse_remote_ino": 0, 2023-11-16T17:00:20.965 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse_lock": 0, 2023-11-16T17:00:20.965 INFO:teuthology.orchestra.run.smithi141.stdout: "load_cent": 0, 2023-11-16T17:00:20.965 INFO:teuthology.orchestra.run.smithi141.stdout: "q": 0, 2023-11-16T17:00:20.965 INFO:teuthology.orchestra.run.smithi141.stdout: "exported": 255, 2023-11-16T17:00:20.965 INFO:teuthology.orchestra.run.smithi141.stdout: "exported_inodes": 9270, 2023-11-16T17:00:20.965 INFO:teuthology.orchestra.run.smithi141.stdout: "imported": 274, 2023-11-16T17:00:20.965 INFO:teuthology.orchestra.run.smithi141.stdout: "imported_inodes": 0, 2023-11-16T17:00:20.965 INFO:teuthology.orchestra.run.smithi141.stdout: "openino_dir_fetch": 0, 2023-11-16T17:00:20.965 INFO:teuthology.orchestra.run.smithi141.stdout: "openino_backtrace_fetch": 0, 2023-11-16T17:00:20.966 INFO:teuthology.orchestra.run.smithi141.stdout: "openino_peer_discover": 0, 2023-11-16T17:00:20.966 INFO:teuthology.orchestra.run.smithi141.stdout: "root_rfiles": 4, 2023-11-16T17:00:20.966 INFO:teuthology.orchestra.run.smithi141.stdout: "root_rbytes": 246, 2023-11-16T17:00:20.966 INFO:teuthology.orchestra.run.smithi141.stdout: "root_rsnaps": 6, 2023-11-16T17:00:20.966 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_backtrace_fetch": 0, 2023-11-16T17:00:20.966 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_set_tag": 0, 2023-11-16T17:00:20.966 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_backtrace_repaired": 0, 2023-11-16T17:00:20.966 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_inotable_repaired": 0, 2023-11-16T17:00:20.967 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_dir_inodes": 0, 2023-11-16T17:00:20.967 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_dir_base_inodes": 0, 2023-11-16T17:00:20.967 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_dirfrag_rstats": 0, 2023-11-16T17:00:20.967 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_file_inodes": 0, 2023-11-16T17:00:20.967 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_inode_file_caps": 4, 2023-11-16T17:00:20.967 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_cap_op_revoke": 81, 2023-11-16T17:00:20.967 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_cap_op_grant": 33520, 2023-11-16T17:00:20.967 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_cap_op_trunc": 0, 2023-11-16T17:00:20.967 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-11-16T17:00:20.968 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-16T17:00:20.968 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_client_caps": 33881, 2023-11-16T17:00:20.968 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_client_caps_dirty": 33774, 2023-11-16T17:00:20.968 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_client_cap_release": 201, 2023-11-16T17:00:20.968 INFO:teuthology.orchestra.run.smithi141.stdout: "process_request_cap_release": 23395 2023-11-16T17:00:20.968 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.968 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_cache": { 2023-11-16T17:00:20.968 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_update": 1857, 2023-11-16T17:00:20.969 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_update_receipt": 2704, 2023-11-16T17:00:20.969 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_try_discover": 1353, 2023-11-16T17:00:20.969 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_send_discover": 1874, 2023-11-16T17:00:20.969 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_handle_discover": 2275, 2023-11-16T17:00:20.969 INFO:teuthology.orchestra.run.smithi141.stdout: "num_strays": 2, 2023-11-16T17:00:20.969 INFO:teuthology.orchestra.run.smithi141.stdout: "num_strays_delayed": 0, 2023-11-16T17:00:20.969 INFO:teuthology.orchestra.run.smithi141.stdout: "num_strays_enqueuing": 0, 2023-11-16T17:00:20.969 INFO:teuthology.orchestra.run.smithi141.stdout: "strays_created": 15707, 2023-11-16T17:00:20.970 INFO:teuthology.orchestra.run.smithi141.stdout: "strays_enqueued": 15705, 2023-11-16T17:00:20.970 INFO:teuthology.orchestra.run.smithi141.stdout: "strays_reintegrated": 0, 2023-11-16T17:00:20.970 INFO:teuthology.orchestra.run.smithi141.stdout: "strays_migrated": 0, 2023-11-16T17:00:20.970 INFO:teuthology.orchestra.run.smithi141.stdout: "num_recovering_processing": 0, 2023-11-16T17:00:20.970 INFO:teuthology.orchestra.run.smithi141.stdout: "num_recovering_enqueued": 0, 2023-11-16T17:00:20.970 INFO:teuthology.orchestra.run.smithi141.stdout: "num_recovering_prioritized": 0, 2023-11-16T17:00:20.970 INFO:teuthology.orchestra.run.smithi141.stdout: "recovery_started": 0, 2023-11-16T17:00:20.970 INFO:teuthology.orchestra.run.smithi141.stdout: "recovery_completed": 0, 2023-11-16T17:00:20.971 INFO:teuthology.orchestra.run.smithi141.stdout: "ireq_enqueue_scrub": 0, 2023-11-16T17:00:20.971 INFO:teuthology.orchestra.run.smithi141.stdout: "ireq_exportdir": 257, 2023-11-16T17:00:20.971 INFO:teuthology.orchestra.run.smithi141.stdout: "ireq_flush": 0, 2023-11-16T17:00:20.971 INFO:teuthology.orchestra.run.smithi141.stdout: "ireq_fragmentdir": 147, 2023-11-16T17:00:20.971 INFO:teuthology.orchestra.run.smithi141.stdout: "ireq_fragstats": 0, 2023-11-16T17:00:20.971 INFO:teuthology.orchestra.run.smithi141.stdout: "ireq_inodestats": 0 2023-11-16T17:00:20.971 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.971 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_log": { 2023-11-16T17:00:20.971 INFO:teuthology.orchestra.run.smithi141.stdout: "evadd": 84087, 2023-11-16T17:00:20.972 INFO:teuthology.orchestra.run.smithi141.stdout: "evex": 84046, 2023-11-16T17:00:20.972 INFO:teuthology.orchestra.run.smithi141.stdout: "evtrm": 84046, 2023-11-16T17:00:20.972 INFO:teuthology.orchestra.run.smithi141.stdout: "ev": 41, 2023-11-16T17:00:20.972 INFO:teuthology.orchestra.run.smithi141.stdout: "evexg": 0, 2023-11-16T17:00:20.972 INFO:teuthology.orchestra.run.smithi141.stdout: "evexd": 10, 2023-11-16T17:00:20.972 INFO:teuthology.orchestra.run.smithi141.stdout: "segadd": 140, 2023-11-16T17:00:20.972 INFO:teuthology.orchestra.run.smithi141.stdout: "segex": 139, 2023-11-16T17:00:20.972 INFO:teuthology.orchestra.run.smithi141.stdout: "segtrm": 139, 2023-11-16T17:00:20.972 INFO:teuthology.orchestra.run.smithi141.stdout: "seg": 2, 2023-11-16T17:00:20.973 INFO:teuthology.orchestra.run.smithi141.stdout: "segexg": 0, 2023-11-16T17:00:20.973 INFO:teuthology.orchestra.run.smithi141.stdout: "segexd": 1, 2023-11-16T17:00:20.973 INFO:teuthology.orchestra.run.smithi141.stdout: "expos": 280457945, 2023-11-16T17:00:20.973 INFO:teuthology.orchestra.run.smithi141.stdout: "wrpos": 280516815, 2023-11-16T17:00:20.973 INFO:teuthology.orchestra.run.smithi141.stdout: "rdpos": 0, 2023-11-16T17:00:20.973 INFO:teuthology.orchestra.run.smithi141.stdout: "jlat": { 2023-11-16T17:00:20.973 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 1246, 2023-11-16T17:00:20.973 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 135.004611389, 2023-11-16T17:00:20.973 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.108350410 2023-11-16T17:00:20.974 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.974 INFO:teuthology.orchestra.run.smithi141.stdout: "replayed": 0 2023-11-16T17:00:20.974 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.974 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_mem": { 2023-11-16T17:00:20.974 INFO:teuthology.orchestra.run.smithi141.stdout: "ino": 31057, 2023-11-16T17:00:20.974 INFO:teuthology.orchestra.run.smithi141.stdout: "ino+": 55941, 2023-11-16T17:00:20.974 INFO:teuthology.orchestra.run.smithi141.stdout: "ino-": 24884, 2023-11-16T17:00:20.974 INFO:teuthology.orchestra.run.smithi141.stdout: "dir": 1056, 2023-11-16T17:00:20.975 INFO:teuthology.orchestra.run.smithi141.stdout: "dir+": 2796, 2023-11-16T17:00:20.975 INFO:teuthology.orchestra.run.smithi141.stdout: "dir-": 1740, 2023-11-16T17:00:20.975 INFO:teuthology.orchestra.run.smithi141.stdout: "dn": 41054, 2023-11-16T17:00:20.975 INFO:teuthology.orchestra.run.smithi141.stdout: "dn+": 77453, 2023-11-16T17:00:20.975 INFO:teuthology.orchestra.run.smithi141.stdout: "dn-": 36399, 2023-11-16T17:00:20.975 INFO:teuthology.orchestra.run.smithi141.stdout: "cap": 1, 2023-11-16T17:00:20.975 INFO:teuthology.orchestra.run.smithi141.stdout: "cap+": 30343, 2023-11-16T17:00:20.975 INFO:teuthology.orchestra.run.smithi141.stdout: "cap-": 30342, 2023-11-16T17:00:20.975 INFO:teuthology.orchestra.run.smithi141.stdout: "rss": 305700, 2023-11-16T17:00:20.976 INFO:teuthology.orchestra.run.smithi141.stdout: "heap": 207156 2023-11-16T17:00:20.976 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.976 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_server": { 2023-11-16T17:00:20.976 INFO:teuthology.orchestra.run.smithi141.stdout: "dispatch_client_request": 52026, 2023-11-16T17:00:20.976 INFO:teuthology.orchestra.run.smithi141.stdout: "dispatch_server_request": 34, 2023-11-16T17:00:20.976 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_client_request": 50430, 2023-11-16T17:00:20.976 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_client_session": 209, 2023-11-16T17:00:20.976 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_peer_request": 76, 2023-11-16T17:00:20.977 INFO:teuthology.orchestra.run.smithi141.stdout: "req_create_latency": { 2023-11-16T17:00:20.977 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 17731, 2023-11-16T17:00:20.977 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 1908.991139796, 2023-11-16T17:00:20.977 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.107664042 2023-11-16T17:00:20.977 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.977 INFO:teuthology.orchestra.run.smithi141.stdout: "req_getattr_latency": { 2023-11-16T17:00:20.977 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 128, 2023-11-16T17:00:20.977 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.038319005, 2023-11-16T17:00:20.978 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000299367 2023-11-16T17:00:20.978 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.978 INFO:teuthology.orchestra.run.smithi141.stdout: "req_getfilelock_latency": { 2023-11-16T17:00:20.978 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:20.978 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:20.978 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:20.978 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.978 INFO:teuthology.orchestra.run.smithi141.stdout: "req_link_latency": { 2023-11-16T17:00:20.978 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:20.979 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:20.979 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:20.979 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.979 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lookup_latency": { 2023-11-16T17:00:20.979 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 7667, 2023-11-16T17:00:20.979 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 11.450746552, 2023-11-16T17:00:20.979 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.001493510 2023-11-16T17:00:20.979 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.980 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lookuphash_latency": { 2023-11-16T17:00:20.980 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:20.980 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:20.980 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:20.980 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.980 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lookupino_latency": { 2023-11-16T17:00:20.980 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 1, 2023-11-16T17:00:20.980 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000177358, 2023-11-16T17:00:20.980 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000177358 2023-11-16T17:00:20.981 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.981 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lookupname_latency": { 2023-11-16T17:00:20.981 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:20.981 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:20.981 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:20.981 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.981 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lookupparent_latency": { 2023-11-16T17:00:20.981 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:20.981 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:20.982 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:20.982 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.982 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lookupsnap_latency": { 2023-11-16T17:00:20.982 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:20.982 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:20.982 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:20.982 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.982 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lssnap_latency": { 2023-11-16T17:00:20.983 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 11, 2023-11-16T17:00:20.983 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.003523684, 2023-11-16T17:00:20.983 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000320334 2023-11-16T17:00:20.983 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.983 INFO:teuthology.orchestra.run.smithi141.stdout: "req_mkdir_latency": { 2023-11-16T17:00:20.983 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 6, 2023-11-16T17:00:20.983 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.008420744, 2023-11-16T17:00:20.983 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.001403457 2023-11-16T17:00:20.983 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.984 INFO:teuthology.orchestra.run.smithi141.stdout: "req_mknod_latency": { 2023-11-16T17:00:20.984 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:20.984 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:20.984 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:20.984 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.984 INFO:teuthology.orchestra.run.smithi141.stdout: "req_mksnap_latency": { 2023-11-16T17:00:20.984 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 11, 2023-11-16T17:00:20.984 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 2.432093744, 2023-11-16T17:00:20.984 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.221099431 2023-11-16T17:00:20.985 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.985 INFO:teuthology.orchestra.run.smithi141.stdout: "req_open_latency": { 2023-11-16T17:00:20.985 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:20.985 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:20.985 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:20.985 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.985 INFO:teuthology.orchestra.run.smithi141.stdout: "req_readdir_latency": { 2023-11-16T17:00:20.985 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 1349, 2023-11-16T17:00:20.985 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 16.121542216, 2023-11-16T17:00:20.986 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.011950735 2023-11-16T17:00:20.986 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.986 INFO:teuthology.orchestra.run.smithi141.stdout: "req_rename_latency": { 2023-11-16T17:00:20.986 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 2, 2023-11-16T17:00:20.986 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.002680145, 2023-11-16T17:00:20.986 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.001340072 2023-11-16T17:00:20.986 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.986 INFO:teuthology.orchestra.run.smithi141.stdout: "req_renamesnap_latency": { 2023-11-16T17:00:20.987 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:20.987 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:20.987 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:20.987 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.987 INFO:teuthology.orchestra.run.smithi141.stdout: "req_snapdiff_latency": { 2023-11-16T17:00:20.987 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:20.987 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:20.987 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:20.987 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.988 INFO:teuthology.orchestra.run.smithi141.stdout: "req_rmdir_latency": { 2023-11-16T17:00:20.988 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 2, 2023-11-16T17:00:20.988 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.079607259, 2023-11-16T17:00:20.988 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.039803629 2023-11-16T17:00:20.988 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.988 INFO:teuthology.orchestra.run.smithi141.stdout: "req_rmsnap_latency": { 2023-11-16T17:00:20.988 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 5, 2023-11-16T17:00:20.988 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.023339978, 2023-11-16T17:00:20.989 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.004667995 2023-11-16T17:00:20.989 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.989 INFO:teuthology.orchestra.run.smithi141.stdout: "req_rmxattr_latency": { 2023-11-16T17:00:20.989 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:20.989 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:20.989 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:20.989 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.989 INFO:teuthology.orchestra.run.smithi141.stdout: "req_setattr_latency": { 2023-11-16T17:00:20.989 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 1, 2023-11-16T17:00:20.990 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000356554, 2023-11-16T17:00:20.990 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000356554 2023-11-16T17:00:20.990 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.990 INFO:teuthology.orchestra.run.smithi141.stdout: "req_setdirlayout_latency": { 2023-11-16T17:00:20.990 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:20.990 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:20.990 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:20.990 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.991 INFO:teuthology.orchestra.run.smithi141.stdout: "req_setfilelock_latency": { 2023-11-16T17:00:20.991 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:20.991 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:20.991 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:20.991 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.991 INFO:teuthology.orchestra.run.smithi141.stdout: "req_setlayout_latency": { 2023-11-16T17:00:20.991 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:20.991 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:20.991 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:20.992 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.992 INFO:teuthology.orchestra.run.smithi141.stdout: "req_setxattr_latency": { 2023-11-16T17:00:20.992 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 7, 2023-11-16T17:00:20.992 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.019967690, 2023-11-16T17:00:20.992 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.002852527 2023-11-16T17:00:20.992 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.992 INFO:teuthology.orchestra.run.smithi141.stdout: "req_symlink_latency": { 2023-11-16T17:00:20.992 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:20.992 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:20.993 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:20.993 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.993 INFO:teuthology.orchestra.run.smithi141.stdout: "req_unlink_latency": { 2023-11-16T17:00:20.993 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 15705, 2023-11-16T17:00:20.993 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 48.775206267, 2023-11-16T17:00:20.993 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.003105711 2023-11-16T17:00:20.993 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.993 INFO:teuthology.orchestra.run.smithi141.stdout: "cap_revoke_eviction": 0, 2023-11-16T17:00:20.993 INFO:teuthology.orchestra.run.smithi141.stdout: "cap_acquisition_throttle": 0, 2023-11-16T17:00:20.994 INFO:teuthology.orchestra.run.smithi141.stdout: "req_getvxattr_latency": { 2023-11-16T17:00:20.994 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:20.994 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:20.994 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:20.994 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:20.994 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.994 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_sessions": { 2023-11-16T17:00:20.994 INFO:teuthology.orchestra.run.smithi141.stdout: "session_count": 2, 2023-11-16T17:00:20.995 INFO:teuthology.orchestra.run.smithi141.stdout: "session_add": 6, 2023-11-16T17:00:20.995 INFO:teuthology.orchestra.run.smithi141.stdout: "session_remove": 4, 2023-11-16T17:00:20.995 INFO:teuthology.orchestra.run.smithi141.stdout: "sessions_open": 2, 2023-11-16T17:00:20.995 INFO:teuthology.orchestra.run.smithi141.stdout: "sessions_stale": 0, 2023-11-16T17:00:20.995 INFO:teuthology.orchestra.run.smithi141.stdout: "total_load": 819, 2023-11-16T17:00:20.995 INFO:teuthology.orchestra.run.smithi141.stdout: "average_load": 409, 2023-11-16T17:00:20.995 INFO:teuthology.orchestra.run.smithi141.stdout: "avg_session_uptime": 621 2023-11-16T17:00:20.995 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:20.996 INFO:teuthology.orchestra.run.smithi141.stdout: "mempool": { 2023-11-16T17:00:20.996 INFO:teuthology.orchestra.run.smithi141.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:00:20.996 INFO:teuthology.orchestra.run.smithi141.stdout: "bloom_filter_items": 0, 2023-11-16T17:00:20.996 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:00:20.996 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:00:20.996 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:00:20.996 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:00:20.996 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:00:20.997 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:00:20.997 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:00:20.997 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:00:20.997 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:00:20.997 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:00:20.997 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:00:20.997 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:00:20.997 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:00:20.997 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_extent_items": 0, 2023-11-16T17:00:20.998 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:00:20.998 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_blob_items": 0, 2023-11-16T17:00:20.998 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:00:20.998 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:00:20.998 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:00:20.998 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:00:20.998 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:00:20.998 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:00:20.999 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:00:20.999 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_txc_items": 0, 2023-11-16T17:00:20.999 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:00:20.999 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:00:20.999 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:00:20.999 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_writing_items": 0, 2023-11-16T17:00:20.999 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_bytes": 0, 2023-11-16T17:00:20.999 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_items": 0, 2023-11-16T17:00:20.999 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:00:21.000 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:00:21.000 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:00:21.000 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:00:21.000 INFO:teuthology.orchestra.run.smithi141.stdout: "buffer_anon_bytes": 1154355, 2023-11-16T17:00:21.000 INFO:teuthology.orchestra.run.smithi141.stdout: "buffer_anon_items": 24333, 2023-11-16T17:00:21.000 INFO:teuthology.orchestra.run.smithi141.stdout: "buffer_meta_bytes": 1760, 2023-11-16T17:00:21.000 INFO:teuthology.orchestra.run.smithi141.stdout: "buffer_meta_items": 20, 2023-11-16T17:00:21.000 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_bytes": 0, 2023-11-16T17:00:21.001 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_items": 0, 2023-11-16T17:00:21.001 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:00:21.001 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_mapbl_items": 0, 2023-11-16T17:00:21.001 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:00:21.001 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_pglog_items": 0, 2023-11-16T17:00:21.001 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_bytes": 9944, 2023-11-16T17:00:21.001 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_items": 192, 2023-11-16T17:00:21.001 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:00:21.001 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:00:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap_bytes": 0, 2023-11-16T17:00:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap_items": 0, 2023-11-16T17:00:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_co_bytes": 99603428, 2023-11-16T17:00:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_co_items": 846783, 2023-11-16T17:00:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: "unittest_1_bytes": 0, 2023-11-16T17:00:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: "unittest_1_items": 0, 2023-11-16T17:00:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: "unittest_2_bytes": 0, 2023-11-16T17:00:21.002 INFO:teuthology.orchestra.run.smithi141.stdout: "unittest_2_items": 0 2023-11-16T17:00:21.055 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:21.055 INFO:teuthology.orchestra.run.smithi141.stdout: "objecter": { 2023-11-16T17:00:21.055 INFO:teuthology.orchestra.run.smithi141.stdout: "op_active": 0, 2023-11-16T17:00:21.055 INFO:teuthology.orchestra.run.smithi141.stdout: "op_laggy": 0, 2023-11-16T17:00:21.055 INFO:teuthology.orchestra.run.smithi141.stdout: "op_send": 85094, 2023-11-16T17:00:21.055 INFO:teuthology.orchestra.run.smithi141.stdout: "op_send_bytes": 355044278, 2023-11-16T17:00:21.056 INFO:teuthology.orchestra.run.smithi141.stdout: "op_resend": 37, 2023-11-16T17:00:21.056 INFO:teuthology.orchestra.run.smithi141.stdout: "op_reply": 85094, 2023-11-16T17:00:21.056 INFO:teuthology.orchestra.run.smithi141.stdout: "op_latency": { 2023-11-16T17:00:21.056 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 85094, 2023-11-16T17:00:21.056 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 35359.400122285, 2023-11-16T17:00:21.056 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.415533411 2023-11-16T17:00:21.056 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:21.056 INFO:teuthology.orchestra.run.smithi141.stdout: "op_inflight": 0, 2023-11-16T17:00:21.056 INFO:teuthology.orchestra.run.smithi141.stdout: "oplen_avg": { 2023-11-16T17:00:21.057 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 85094, 2023-11-16T17:00:21.057 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 161523 2023-11-16T17:00:21.057 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:21.057 INFO:teuthology.orchestra.run.smithi141.stdout: "op": 85094, 2023-11-16T17:00:21.057 INFO:teuthology.orchestra.run.smithi141.stdout: "op_r": 628, 2023-11-16T17:00:21.057 INFO:teuthology.orchestra.run.smithi141.stdout: "op_w": 84466, 2023-11-16T17:00:21.057 INFO:teuthology.orchestra.run.smithi141.stdout: "op_rmw": 0, 2023-11-16T17:00:21.057 INFO:teuthology.orchestra.run.smithi141.stdout: "op_pg": 0, 2023-11-16T17:00:21.058 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_stat": 10477, 2023-11-16T17:00:21.058 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_create": 38907, 2023-11-16T17:00:21.058 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_read": 198, 2023-11-16T17:00:21.058 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_write": 1444, 2023-11-16T17:00:21.058 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_writefull": 189, 2023-11-16T17:00:21.058 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_writesame": 0, 2023-11-16T17:00:21.058 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_append": 0, 2023-11-16T17:00:21.058 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_zero": 0, 2023-11-16T17:00:21.058 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_truncate": 0, 2023-11-16T17:00:21.059 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_delete": 32202, 2023-11-16T17:00:21.059 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_mapext": 0, 2023-11-16T17:00:21.059 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_sparse_read": 0, 2023-11-16T17:00:21.059 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_clonerange": 0, 2023-11-16T17:00:21.059 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_getxattr": 0, 2023-11-16T17:00:21.059 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_setxattr": 58564, 2023-11-16T17:00:21.059 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_cmpxattr": 0, 2023-11-16T17:00:21.059 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_rmxattr": 0, 2023-11-16T17:00:21.060 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_resetxattrs": 0, 2023-11-16T17:00:21.060 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_call": 0, 2023-11-16T17:00:21.060 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_watch": 0, 2023-11-16T17:00:21.060 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_notify": 0, 2023-11-16T17:00:21.060 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_src_cmpxattr": 0, 2023-11-16T17:00:21.060 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_pgls": 0, 2023-11-16T17:00:21.060 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_pgls_filter": 0, 2023-11-16T17:00:21.060 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_other": 0, 2023-11-16T17:00:21.060 INFO:teuthology.orchestra.run.smithi141.stdout: "linger_active": 0, 2023-11-16T17:00:21.061 INFO:teuthology.orchestra.run.smithi141.stdout: "linger_send": 0, 2023-11-16T17:00:21.061 INFO:teuthology.orchestra.run.smithi141.stdout: "linger_resend": 0, 2023-11-16T17:00:21.061 INFO:teuthology.orchestra.run.smithi141.stdout: "linger_ping": 0, 2023-11-16T17:00:21.061 INFO:teuthology.orchestra.run.smithi141.stdout: "poolop_active": 0, 2023-11-16T17:00:21.061 INFO:teuthology.orchestra.run.smithi141.stdout: "poolop_send": 0, 2023-11-16T17:00:21.061 INFO:teuthology.orchestra.run.smithi141.stdout: "poolop_resend": 0, 2023-11-16T17:00:21.061 INFO:teuthology.orchestra.run.smithi141.stdout: "poolstat_active": 0, 2023-11-16T17:00:21.061 INFO:teuthology.orchestra.run.smithi141.stdout: "poolstat_send": 0, 2023-11-16T17:00:21.061 INFO:teuthology.orchestra.run.smithi141.stdout: "poolstat_resend": 0, 2023-11-16T17:00:21.062 INFO:teuthology.orchestra.run.smithi141.stdout: "statfs_active": 0, 2023-11-16T17:00:21.062 INFO:teuthology.orchestra.run.smithi141.stdout: "statfs_send": 0, 2023-11-16T17:00:21.062 INFO:teuthology.orchestra.run.smithi141.stdout: "statfs_resend": 0, 2023-11-16T17:00:21.062 INFO:teuthology.orchestra.run.smithi141.stdout: "command_active": 0, 2023-11-16T17:00:21.062 INFO:teuthology.orchestra.run.smithi141.stdout: "command_send": 0, 2023-11-16T17:00:21.062 INFO:teuthology.orchestra.run.smithi141.stdout: "command_resend": 0, 2023-11-16T17:00:21.062 INFO:teuthology.orchestra.run.smithi141.stdout: "map_epoch": 104, 2023-11-16T17:00:21.062 INFO:teuthology.orchestra.run.smithi141.stdout: "map_full": 0, 2023-11-16T17:00:21.063 INFO:teuthology.orchestra.run.smithi141.stdout: "map_inc": 20, 2023-11-16T17:00:21.063 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_sessions": 12, 2023-11-16T17:00:21.063 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_session_open": 12, 2023-11-16T17:00:21.063 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_session_close": 0, 2023-11-16T17:00:21.063 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_laggy": 0, 2023-11-16T17:00:21.063 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_wr": 17598, 2023-11-16T17:00:21.063 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_rd": 860, 2023-11-16T17:00:21.063 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_del": 1084 2023-11-16T17:00:21.063 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:21.064 INFO:teuthology.orchestra.run.smithi141.stdout: "oft": { 2023-11-16T17:00:21.064 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_total_objs": 1, 2023-11-16T17:00:21.064 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_total_kv_pairs": 0, 2023-11-16T17:00:21.064 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_total_updates": 0, 2023-11-16T17:00:21.064 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_total_removes": 0 2023-11-16T17:00:21.064 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:21.064 INFO:teuthology.orchestra.run.smithi141.stdout: "osd-slow-ops": { 2023-11-16T17:00:21.064 INFO:teuthology.orchestra.run.smithi141.stdout: "slow_ops_count": 50869 2023-11-16T17:00:21.064 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:21.065 INFO:teuthology.orchestra.run.smithi141.stdout: "purge_queue": { 2023-11-16T17:00:21.065 INFO:teuthology.orchestra.run.smithi141.stdout: "pq_executing_ops": 0, 2023-11-16T17:00:21.065 INFO:teuthology.orchestra.run.smithi141.stdout: "pq_executing_ops_high_water": 16, 2023-11-16T17:00:21.065 INFO:teuthology.orchestra.run.smithi141.stdout: "pq_executing": 0, 2023-11-16T17:00:21.065 INFO:teuthology.orchestra.run.smithi141.stdout: "pq_executing_high_water": 8, 2023-11-16T17:00:21.065 INFO:teuthology.orchestra.run.smithi141.stdout: "pq_executed": 15705, 2023-11-16T17:00:21.065 INFO:teuthology.orchestra.run.smithi141.stdout: "pq_item_in_journal": 0 2023-11-16T17:00:21.065 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:21.065 INFO:teuthology.orchestra.run.smithi141.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:00:21.066 INFO:teuthology.orchestra.run.smithi141.stdout: "val": 0, 2023-11-16T17:00:21.066 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 104857600, 2023-11-16T17:00:21.066 INFO:teuthology.orchestra.run.smithi141.stdout: "get_started": 0, 2023-11-16T17:00:21.066 INFO:teuthology.orchestra.run.smithi141.stdout: "get": 188905, 2023-11-16T17:00:21.066 INFO:teuthology.orchestra.run.smithi141.stdout: "get_sum": 71763198, 2023-11-16T17:00:21.066 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:21.066 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_success": 188905, 2023-11-16T17:00:21.066 INFO:teuthology.orchestra.run.smithi141.stdout: "take": 0, 2023-11-16T17:00:21.066 INFO:teuthology.orchestra.run.smithi141.stdout: "take_sum": 0, 2023-11-16T17:00:21.067 INFO:teuthology.orchestra.run.smithi141.stdout: "put": 188905, 2023-11-16T17:00:21.067 INFO:teuthology.orchestra.run.smithi141.stdout: "put_sum": 71763198, 2023-11-16T17:00:21.067 INFO:teuthology.orchestra.run.smithi141.stdout: "wait": { 2023-11-16T17:00:21.067 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:21.067 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:21.067 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:21.067 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:21.067 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:21.068 INFO:teuthology.orchestra.run.smithi141.stdout: "throttle-objecter_bytes": { 2023-11-16T17:00:21.068 INFO:teuthology.orchestra.run.smithi141.stdout: "val": 0, 2023-11-16T17:00:21.068 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 104857600, 2023-11-16T17:00:21.068 INFO:teuthology.orchestra.run.smithi141.stdout: "get_started": 0, 2023-11-16T17:00:21.068 INFO:teuthology.orchestra.run.smithi141.stdout: "get": 0, 2023-11-16T17:00:21.068 INFO:teuthology.orchestra.run.smithi141.stdout: "get_sum": 0, 2023-11-16T17:00:21.068 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:21.068 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_success": 0, 2023-11-16T17:00:21.068 INFO:teuthology.orchestra.run.smithi141.stdout: "take": 85094, 2023-11-16T17:00:21.069 INFO:teuthology.orchestra.run.smithi141.stdout: "take_sum": 356969307, 2023-11-16T17:00:21.069 INFO:teuthology.orchestra.run.smithi141.stdout: "put": 85094, 2023-11-16T17:00:21.069 INFO:teuthology.orchestra.run.smithi141.stdout: "put_sum": 356969307, 2023-11-16T17:00:21.069 INFO:teuthology.orchestra.run.smithi141.stdout: "wait": { 2023-11-16T17:00:21.069 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:21.069 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:21.069 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:21.069 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:21.070 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:21.070 INFO:teuthology.orchestra.run.smithi141.stdout: "throttle-objecter_ops": { 2023-11-16T17:00:21.070 INFO:teuthology.orchestra.run.smithi141.stdout: "val": 0, 2023-11-16T17:00:21.070 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1024, 2023-11-16T17:00:21.070 INFO:teuthology.orchestra.run.smithi141.stdout: "get_started": 0, 2023-11-16T17:00:21.070 INFO:teuthology.orchestra.run.smithi141.stdout: "get": 0, 2023-11-16T17:00:21.070 INFO:teuthology.orchestra.run.smithi141.stdout: "get_sum": 0, 2023-11-16T17:00:21.070 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:21.070 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_success": 0, 2023-11-16T17:00:21.071 INFO:teuthology.orchestra.run.smithi141.stdout: "take": 85094, 2023-11-16T17:00:21.071 INFO:teuthology.orchestra.run.smithi141.stdout: "take_sum": 85094, 2023-11-16T17:00:21.071 INFO:teuthology.orchestra.run.smithi141.stdout: "put": 85094, 2023-11-16T17:00:21.071 INFO:teuthology.orchestra.run.smithi141.stdout: "put_sum": 85094, 2023-11-16T17:00:21.071 INFO:teuthology.orchestra.run.smithi141.stdout: "wait": { 2023-11-16T17:00:21.071 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:21.071 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:21.071 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:21.072 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:21.072 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:21.072 INFO:teuthology.orchestra.run.smithi141.stdout: "throttle-write_buf_throttle": { 2023-11-16T17:00:21.072 INFO:teuthology.orchestra.run.smithi141.stdout: "val": 0, 2023-11-16T17:00:21.072 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 3758096384, 2023-11-16T17:00:21.072 INFO:teuthology.orchestra.run.smithi141.stdout: "get_started": 0, 2023-11-16T17:00:21.072 INFO:teuthology.orchestra.run.smithi141.stdout: "get": 15705, 2023-11-16T17:00:21.072 INFO:teuthology.orchestra.run.smithi141.stdout: "get_sum": 1931909, 2023-11-16T17:00:21.072 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:21.073 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_success": 15705, 2023-11-16T17:00:21.073 INFO:teuthology.orchestra.run.smithi141.stdout: "take": 0, 2023-11-16T17:00:21.073 INFO:teuthology.orchestra.run.smithi141.stdout: "take_sum": 0, 2023-11-16T17:00:21.073 INFO:teuthology.orchestra.run.smithi141.stdout: "put": 198, 2023-11-16T17:00:21.073 INFO:teuthology.orchestra.run.smithi141.stdout: "put_sum": 1931909, 2023-11-16T17:00:21.073 INFO:teuthology.orchestra.run.smithi141.stdout: "wait": { 2023-11-16T17:00:21.073 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:21.073 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:21.073 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:21.074 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:21.074 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:21.074 INFO:teuthology.orchestra.run.smithi141.stdout: "throttle-write_buf_throttle-0x5586e008c0a0": { 2023-11-16T17:00:21.074 INFO:teuthology.orchestra.run.smithi141.stdout: "val": 0, 2023-11-16T17:00:21.074 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 3758096384, 2023-11-16T17:00:21.074 INFO:teuthology.orchestra.run.smithi141.stdout: "get_started": 0, 2023-11-16T17:00:21.074 INFO:teuthology.orchestra.run.smithi141.stdout: "get": 84087, 2023-11-16T17:00:21.074 INFO:teuthology.orchestra.run.smithi141.stdout: "get_sum": 276322511, 2023-11-16T17:00:21.075 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:21.075 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_success": 84087, 2023-11-16T17:00:21.075 INFO:teuthology.orchestra.run.smithi141.stdout: "take": 0, 2023-11-16T17:00:21.075 INFO:teuthology.orchestra.run.smithi141.stdout: "take_sum": 0, 2023-11-16T17:00:21.075 INFO:teuthology.orchestra.run.smithi141.stdout: "put": 1246, 2023-11-16T17:00:21.075 INFO:teuthology.orchestra.run.smithi141.stdout: "put_sum": 276322511, 2023-11-16T17:00:21.075 INFO:teuthology.orchestra.run.smithi141.stdout: "wait": { 2023-11-16T17:00:21.075 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:21.075 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:21.076 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:21.076 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:21.076 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:21.076 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-11-16T17:00:21.264 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:21 smithi141 ceph-mon[131184]: pgmap v325: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 607 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T17:00:21.371 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:21 smithi155 ceph-mon[131607]: pgmap v325: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 607 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T17:00:21.376 INFO:tasks.check_counter:Daemon mds.b mds.exported=255 2023-11-16T17:00:21.377 INFO:tasks.check_counter:Daemon mds.b mds.imported=274 2023-11-16T17:00:21.377 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=1857 2023-11-16T17:00:21.377 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=2704 2023-11-16T17:00:21.377 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2023-11-16T17:00:21.377 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=11 2023-11-16T17:00:21.377 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=5 2023-11-16T17:00:21.377 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=142 2023-11-16T17:00:21.377 DEBUG:tasks.check_counter:Getting stats from e 2023-11-16T17:00:21.378 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.e perf dump 2023-11-16T17:00:21.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:21 smithi002 ceph-mon[128712]: pgmap v325: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 607 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-11-16T17:00:22.263 INFO:teuthology.orchestra.run.smithi141.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T17:00:27.526 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:23 smithi141 ceph-mon[131184]: pgmap v326: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 98 B/s wr, 7 op/s 2023-11-16T17:00:27.526 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:25 smithi141 ceph-mon[131184]: pgmap v327: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2023-11-16T17:00:27.526 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:25 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T17:00:27.527 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:23 smithi002 ceph-mon[128712]: pgmap v326: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 98 B/s wr, 7 op/s 2023-11-16T17:00:27.527 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:25 smithi002 ceph-mon[128712]: pgmap v327: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2023-11-16T17:00:27.527 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:25 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T17:00:27.527 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:27 smithi002 ceph-mon[128712]: pgmap v328: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2023-11-16T17:00:27.527 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:23 smithi155 ceph-mon[131607]: pgmap v326: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 98 B/s wr, 7 op/s 2023-11-16T17:00:27.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:25 smithi155 ceph-mon[131607]: pgmap v327: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2023-11-16T17:00:27.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:25 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-16T17:00:27.528 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:27 smithi155 ceph-mon[131607]: pgmap v328: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2023-11-16T17:00:27.529 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-11-16T17:00:27.529 INFO:teuthology.orchestra.run.smithi141.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:00:27.529 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_messages": 869, 2023-11-16T17:00:27.529 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages": 869, 2023-11-16T17:00:27.530 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_bytes": 88847797, 2023-11-16T17:00:27.530 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_bytes": 269527, 2023-11-16T17:00:27.530 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_created_connections": 8, 2023-11-16T17:00:27.530 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_active_connections": 4, 2023-11-16T17:00:27.530 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_total_time": 0.127992895, 2023-11-16T17:00:27.530 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_send_time": 0.023204728, 2023-11-16T17:00:27.530 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_recv_time": 0.066934879, 2023-11-16T17:00:27.531 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_fast_dispatch_time": 0.029739993, 2023-11-16T17:00:27.531 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:27.531 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 868, 2023-11-16T17:00:27.531 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.042107024, 2023-11-16T17:00:27.531 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000048510 2023-11-16T17:00:27.531 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.531 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:27.531 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.532 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.532 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.532 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.532 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-11-16T17:00:27.532 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_encrypted_bytes": 448 2023-11-16T17:00:27.532 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.532 INFO:teuthology.orchestra.run.smithi141.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:00:27.533 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_messages": 1106, 2023-11-16T17:00:27.533 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages": 1079, 2023-11-16T17:00:27.533 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_bytes": 110501382, 2023-11-16T17:00:27.533 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_bytes": 388784, 2023-11-16T17:00:27.533 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_created_connections": 5, 2023-11-16T17:00:27.533 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_active_connections": 4, 2023-11-16T17:00:27.533 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_total_time": 0.183778632, 2023-11-16T17:00:27.533 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_send_time": 0.032903664, 2023-11-16T17:00:27.534 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_recv_time": 0.118047084, 2023-11-16T17:00:27.534 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_fast_dispatch_time": 0.034818427, 2023-11-16T17:00:27.534 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:27.534 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 1078, 2023-11-16T17:00:27.534 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.064029726, 2023-11-16T17:00:27.534 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000059396 2023-11-16T17:00:27.534 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.535 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:27.535 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.535 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.535 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.535 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.535 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_encrypted_bytes": 1518112, 2023-11-16T17:00:27.535 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_encrypted_bytes": 125984 2023-11-16T17:00:27.535 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.536 INFO:teuthology.orchestra.run.smithi141.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:00:27.536 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_messages": 953, 2023-11-16T17:00:27.536 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages": 1115, 2023-11-16T17:00:27.536 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_bytes": 63951372, 2023-11-16T17:00:27.536 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_bytes": 725201, 2023-11-16T17:00:27.536 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_created_connections": 8, 2023-11-16T17:00:27.536 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_active_connections": 6, 2023-11-16T17:00:27.537 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_total_time": 0.143445830, 2023-11-16T17:00:27.537 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_send_time": 0.032595533, 2023-11-16T17:00:27.537 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_recv_time": 0.061183336, 2023-11-16T17:00:27.537 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_fast_dispatch_time": 0.038030097, 2023-11-16T17:00:27.537 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:27.537 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 1112, 2023-11-16T17:00:27.537 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.079621378, 2023-11-16T17:00:27.537 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000071601 2023-11-16T17:00:27.538 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.538 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:27.538 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.538 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.538 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.538 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.538 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_encrypted_bytes": 320, 2023-11-16T17:00:27.539 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_encrypted_bytes": 429440 2023-11-16T17:00:27.539 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.539 INFO:teuthology.orchestra.run.smithi141.stdout: "cct": { 2023-11-16T17:00:27.539 INFO:teuthology.orchestra.run.smithi141.stdout: "total_workers": 1, 2023-11-16T17:00:27.539 INFO:teuthology.orchestra.run.smithi141.stdout: "unhealthy_workers": 0 2023-11-16T17:00:27.539 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.539 INFO:teuthology.orchestra.run.smithi141.stdout: "finisher-MDSRank": { 2023-11-16T17:00:27.539 INFO:teuthology.orchestra.run.smithi141.stdout: "queue_len": 0, 2023-11-16T17:00:27.540 INFO:teuthology.orchestra.run.smithi141.stdout: "complete_latency": { 2023-11-16T17:00:27.540 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 4221, 2023-11-16T17:00:27.540 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.126853496, 2023-11-16T17:00:27.540 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000030052 2023-11-16T17:00:27.540 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:27.540 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.540 INFO:teuthology.orchestra.run.smithi141.stdout: "finisher-PurgeQueue": { 2023-11-16T17:00:27.541 INFO:teuthology.orchestra.run.smithi141.stdout: "queue_len": 0, 2023-11-16T17:00:27.541 INFO:teuthology.orchestra.run.smithi141.stdout: "complete_latency": { 2023-11-16T17:00:27.541 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.541 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.541 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.541 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:27.541 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.541 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": { 2023-11-16T17:00:27.542 INFO:teuthology.orchestra.run.smithi141.stdout: "request": 0, 2023-11-16T17:00:27.542 INFO:teuthology.orchestra.run.smithi141.stdout: "reply": 0, 2023-11-16T17:00:27.542 INFO:teuthology.orchestra.run.smithi141.stdout: "reply_latency": { 2023-11-16T17:00:27.542 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.542 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.542 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.543 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.543 INFO:teuthology.orchestra.run.smithi141.stdout: "slow_reply": 0, 2023-11-16T17:00:27.543 INFO:teuthology.orchestra.run.smithi141.stdout: "forward": 0, 2023-11-16T17:00:27.543 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_fetch_complete": 0, 2023-11-16T17:00:27.543 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_fetch_keys": 0, 2023-11-16T17:00:27.543 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_commit": 0, 2023-11-16T17:00:27.543 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_split": 0, 2023-11-16T17:00:27.543 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_merge": 0, 2023-11-16T17:00:27.544 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes": 16787, 2023-11-16T17:00:27.544 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes_top": 0, 2023-11-16T17:00:27.544 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes_bottom": 16784, 2023-11-16T17:00:27.544 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes_pin_tail": 3, 2023-11-16T17:00:27.544 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes_pinned": 16786, 2023-11-16T17:00:27.544 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes_expired": 0, 2023-11-16T17:00:27.544 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes_with_caps": 0, 2023-11-16T17:00:27.544 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": 0, 2023-11-16T17:00:27.545 INFO:teuthology.orchestra.run.smithi141.stdout: "subtrees": 4, 2023-11-16T17:00:27.545 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse": 0, 2023-11-16T17:00:27.545 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse_hit": 0, 2023-11-16T17:00:27.545 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse_forward": 0, 2023-11-16T17:00:27.545 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse_discover": 0, 2023-11-16T17:00:27.545 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse_dir_fetch": 0, 2023-11-16T17:00:27.545 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse_remote_ino": 0, 2023-11-16T17:00:27.545 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse_lock": 0, 2023-11-16T17:00:27.546 INFO:teuthology.orchestra.run.smithi141.stdout: "load_cent": 0, 2023-11-16T17:00:27.546 INFO:teuthology.orchestra.run.smithi141.stdout: "q": 0, 2023-11-16T17:00:27.546 INFO:teuthology.orchestra.run.smithi141.stdout: "exported": 0, 2023-11-16T17:00:27.546 INFO:teuthology.orchestra.run.smithi141.stdout: "exported_inodes": 0, 2023-11-16T17:00:27.546 INFO:teuthology.orchestra.run.smithi141.stdout: "imported": 0, 2023-11-16T17:00:27.546 INFO:teuthology.orchestra.run.smithi141.stdout: "imported_inodes": 0, 2023-11-16T17:00:27.547 INFO:teuthology.orchestra.run.smithi141.stdout: "openino_dir_fetch": 0, 2023-11-16T17:00:27.547 INFO:teuthology.orchestra.run.smithi141.stdout: "openino_backtrace_fetch": 0, 2023-11-16T17:00:27.547 INFO:teuthology.orchestra.run.smithi141.stdout: "openino_peer_discover": 0, 2023-11-16T17:00:27.547 INFO:teuthology.orchestra.run.smithi141.stdout: "root_rfiles": 0, 2023-11-16T17:00:27.547 INFO:teuthology.orchestra.run.smithi141.stdout: "root_rbytes": 0, 2023-11-16T17:00:27.547 INFO:teuthology.orchestra.run.smithi141.stdout: "root_rsnaps": 0, 2023-11-16T17:00:27.547 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_backtrace_fetch": 0, 2023-11-16T17:00:27.547 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_set_tag": 0, 2023-11-16T17:00:27.548 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_backtrace_repaired": 0, 2023-11-16T17:00:27.548 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_inotable_repaired": 0, 2023-11-16T17:00:27.548 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_dir_inodes": 0, 2023-11-16T17:00:27.548 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_dir_base_inodes": 0, 2023-11-16T17:00:27.548 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_dirfrag_rstats": 0, 2023-11-16T17:00:27.548 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_file_inodes": 0, 2023-11-16T17:00:27.548 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_inode_file_caps": 0, 2023-11-16T17:00:27.548 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_cap_op_revoke": 0, 2023-11-16T17:00:27.549 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_cap_op_grant": 0, 2023-11-16T17:00:27.549 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_cap_op_trunc": 0, 2023-11-16T17:00:27.549 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-11-16T17:00:27.549 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-16T17:00:27.549 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_client_caps": 0, 2023-11-16T17:00:27.549 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_client_caps_dirty": 0, 2023-11-16T17:00:27.549 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_client_cap_release": 0, 2023-11-16T17:00:27.549 INFO:teuthology.orchestra.run.smithi141.stdout: "process_request_cap_release": 0 2023-11-16T17:00:27.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.550 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_cache": { 2023-11-16T17:00:27.550 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_update": 0, 2023-11-16T17:00:27.550 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_update_receipt": 0, 2023-11-16T17:00:27.550 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_try_discover": 0, 2023-11-16T17:00:27.550 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_send_discover": 0, 2023-11-16T17:00:27.550 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_handle_discover": 0, 2023-11-16T17:00:27.550 INFO:teuthology.orchestra.run.smithi141.stdout: "num_strays": 0, 2023-11-16T17:00:27.551 INFO:teuthology.orchestra.run.smithi141.stdout: "num_strays_delayed": 0, 2023-11-16T17:00:27.551 INFO:teuthology.orchestra.run.smithi141.stdout: "num_strays_enqueuing": 0, 2023-11-16T17:00:27.551 INFO:teuthology.orchestra.run.smithi141.stdout: "strays_created": 0, 2023-11-16T17:00:27.551 INFO:teuthology.orchestra.run.smithi141.stdout: "strays_enqueued": 0, 2023-11-16T17:00:27.551 INFO:teuthology.orchestra.run.smithi141.stdout: "strays_reintegrated": 0, 2023-11-16T17:00:27.551 INFO:teuthology.orchestra.run.smithi141.stdout: "strays_migrated": 0, 2023-11-16T17:00:27.551 INFO:teuthology.orchestra.run.smithi141.stdout: "num_recovering_processing": 0, 2023-11-16T17:00:27.551 INFO:teuthology.orchestra.run.smithi141.stdout: "num_recovering_enqueued": 0, 2023-11-16T17:00:27.552 INFO:teuthology.orchestra.run.smithi141.stdout: "num_recovering_prioritized": 0, 2023-11-16T17:00:27.552 INFO:teuthology.orchestra.run.smithi141.stdout: "recovery_started": 0, 2023-11-16T17:00:27.552 INFO:teuthology.orchestra.run.smithi141.stdout: "recovery_completed": 0, 2023-11-16T17:00:27.552 INFO:teuthology.orchestra.run.smithi141.stdout: "ireq_enqueue_scrub": 0, 2023-11-16T17:00:27.552 INFO:teuthology.orchestra.run.smithi141.stdout: "ireq_exportdir": 0, 2023-11-16T17:00:27.552 INFO:teuthology.orchestra.run.smithi141.stdout: "ireq_flush": 0, 2023-11-16T17:00:27.552 INFO:teuthology.orchestra.run.smithi141.stdout: "ireq_fragmentdir": 0, 2023-11-16T17:00:27.552 INFO:teuthology.orchestra.run.smithi141.stdout: "ireq_fragstats": 0, 2023-11-16T17:00:27.553 INFO:teuthology.orchestra.run.smithi141.stdout: "ireq_inodestats": 0 2023-11-16T17:00:27.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.553 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_log": { 2023-11-16T17:00:27.553 INFO:teuthology.orchestra.run.smithi141.stdout: "evadd": 0, 2023-11-16T17:00:27.553 INFO:teuthology.orchestra.run.smithi141.stdout: "evex": 0, 2023-11-16T17:00:27.553 INFO:teuthology.orchestra.run.smithi141.stdout: "evtrm": 0, 2023-11-16T17:00:27.553 INFO:teuthology.orchestra.run.smithi141.stdout: "ev": 0, 2023-11-16T17:00:27.553 INFO:teuthology.orchestra.run.smithi141.stdout: "evexg": 0, 2023-11-16T17:00:27.554 INFO:teuthology.orchestra.run.smithi141.stdout: "evexd": 0, 2023-11-16T17:00:27.554 INFO:teuthology.orchestra.run.smithi141.stdout: "segadd": 0, 2023-11-16T17:00:27.554 INFO:teuthology.orchestra.run.smithi141.stdout: "segex": 0, 2023-11-16T17:00:27.554 INFO:teuthology.orchestra.run.smithi141.stdout: "segtrm": 0, 2023-11-16T17:00:27.554 INFO:teuthology.orchestra.run.smithi141.stdout: "seg": 125, 2023-11-16T17:00:27.554 INFO:teuthology.orchestra.run.smithi141.stdout: "segexg": 0, 2023-11-16T17:00:27.554 INFO:teuthology.orchestra.run.smithi141.stdout: "segexd": 0, 2023-11-16T17:00:27.554 INFO:teuthology.orchestra.run.smithi141.stdout: "expos": 4194304, 2023-11-16T17:00:27.555 INFO:teuthology.orchestra.run.smithi141.stdout: "wrpos": 0, 2023-11-16T17:00:27.555 INFO:teuthology.orchestra.run.smithi141.stdout: "rdpos": 263950246, 2023-11-16T17:00:27.555 INFO:teuthology.orchestra.run.smithi141.stdout: "jlat": { 2023-11-16T17:00:27.555 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.555 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.555 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.556 INFO:teuthology.orchestra.run.smithi141.stdout: "replayed": 92995 2023-11-16T17:00:27.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.556 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_mem": { 2023-11-16T17:00:27.556 INFO:teuthology.orchestra.run.smithi141.stdout: "ino": 17, 2023-11-16T17:00:27.556 INFO:teuthology.orchestra.run.smithi141.stdout: "ino+": 60074, 2023-11-16T17:00:27.556 INFO:teuthology.orchestra.run.smithi141.stdout: "ino-": 60057, 2023-11-16T17:00:27.556 INFO:teuthology.orchestra.run.smithi141.stdout: "dir": 16, 2023-11-16T17:00:27.556 INFO:teuthology.orchestra.run.smithi141.stdout: "dir+": 1492, 2023-11-16T17:00:27.557 INFO:teuthology.orchestra.run.smithi141.stdout: "dir-": 1476, 2023-11-16T17:00:27.557 INFO:teuthology.orchestra.run.smithi141.stdout: "dn": 16787, 2023-11-16T17:00:27.557 INFO:teuthology.orchestra.run.smithi141.stdout: "dn+": 86896, 2023-11-16T17:00:27.557 INFO:teuthology.orchestra.run.smithi141.stdout: "dn-": 70109, 2023-11-16T17:00:27.557 INFO:teuthology.orchestra.run.smithi141.stdout: "cap": 0, 2023-11-16T17:00:27.557 INFO:teuthology.orchestra.run.smithi141.stdout: "cap+": 0, 2023-11-16T17:00:27.557 INFO:teuthology.orchestra.run.smithi141.stdout: "cap-": 0, 2023-11-16T17:00:27.557 INFO:teuthology.orchestra.run.smithi141.stdout: "rss": 96168, 2023-11-16T17:00:27.557 INFO:teuthology.orchestra.run.smithi141.stdout: "heap": 207156 2023-11-16T17:00:27.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.558 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_server": { 2023-11-16T17:00:27.558 INFO:teuthology.orchestra.run.smithi141.stdout: "dispatch_client_request": 0, 2023-11-16T17:00:27.558 INFO:teuthology.orchestra.run.smithi141.stdout: "dispatch_server_request": 0, 2023-11-16T17:00:27.558 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_client_request": 0, 2023-11-16T17:00:27.558 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_client_session": 0, 2023-11-16T17:00:27.558 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_peer_request": 0, 2023-11-16T17:00:27.559 INFO:teuthology.orchestra.run.smithi141.stdout: "req_create_latency": { 2023-11-16T17:00:27.559 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.559 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.559 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.559 INFO:teuthology.orchestra.run.smithi141.stdout: "req_getattr_latency": { 2023-11-16T17:00:27.559 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.559 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.560 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.560 INFO:teuthology.orchestra.run.smithi141.stdout: "req_getfilelock_latency": { 2023-11-16T17:00:27.560 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.560 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.560 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.560 INFO:teuthology.orchestra.run.smithi141.stdout: "req_link_latency": { 2023-11-16T17:00:27.561 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.561 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.561 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.561 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lookup_latency": { 2023-11-16T17:00:27.561 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.561 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.561 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.562 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lookuphash_latency": { 2023-11-16T17:00:27.562 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.562 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.562 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.562 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.562 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lookupino_latency": { 2023-11-16T17:00:27.562 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.563 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.563 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.563 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lookupname_latency": { 2023-11-16T17:00:27.563 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.563 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.563 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.563 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lookupparent_latency": { 2023-11-16T17:00:27.564 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.564 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.564 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.564 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lookupsnap_latency": { 2023-11-16T17:00:27.564 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.564 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.564 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.565 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lssnap_latency": { 2023-11-16T17:00:27.565 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.565 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.565 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.565 INFO:teuthology.orchestra.run.smithi141.stdout: "req_mkdir_latency": { 2023-11-16T17:00:27.565 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.566 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.566 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.566 INFO:teuthology.orchestra.run.smithi141.stdout: "req_mknod_latency": { 2023-11-16T17:00:27.566 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.566 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.566 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.567 INFO:teuthology.orchestra.run.smithi141.stdout: "req_mksnap_latency": { 2023-11-16T17:00:27.567 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.567 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.567 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.567 INFO:teuthology.orchestra.run.smithi141.stdout: "req_open_latency": { 2023-11-16T17:00:27.567 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.567 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.568 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.568 INFO:teuthology.orchestra.run.smithi141.stdout: "req_readdir_latency": { 2023-11-16T17:00:27.568 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.568 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.568 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.568 INFO:teuthology.orchestra.run.smithi141.stdout: "req_rename_latency": { 2023-11-16T17:00:27.569 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.569 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.569 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.569 INFO:teuthology.orchestra.run.smithi141.stdout: "req_renamesnap_latency": { 2023-11-16T17:00:27.569 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.569 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.569 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.570 INFO:teuthology.orchestra.run.smithi141.stdout: "req_snapdiff_latency": { 2023-11-16T17:00:27.570 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.570 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.570 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.570 INFO:teuthology.orchestra.run.smithi141.stdout: "req_rmdir_latency": { 2023-11-16T17:00:27.570 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.571 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.571 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.571 INFO:teuthology.orchestra.run.smithi141.stdout: "req_rmsnap_latency": { 2023-11-16T17:00:27.571 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.571 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.571 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.572 INFO:teuthology.orchestra.run.smithi141.stdout: "req_rmxattr_latency": { 2023-11-16T17:00:27.572 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.572 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.572 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.572 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.572 INFO:teuthology.orchestra.run.smithi141.stdout: "req_setattr_latency": { 2023-11-16T17:00:27.572 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.572 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.573 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.573 INFO:teuthology.orchestra.run.smithi141.stdout: "req_setdirlayout_latency": { 2023-11-16T17:00:27.573 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.573 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.573 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.573 INFO:teuthology.orchestra.run.smithi141.stdout: "req_setfilelock_latency": { 2023-11-16T17:00:27.574 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.574 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.574 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.574 INFO:teuthology.orchestra.run.smithi141.stdout: "req_setlayout_latency": { 2023-11-16T17:00:27.574 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.574 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.574 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.575 INFO:teuthology.orchestra.run.smithi141.stdout: "req_setxattr_latency": { 2023-11-16T17:00:27.575 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.575 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.575 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.575 INFO:teuthology.orchestra.run.smithi141.stdout: "req_symlink_latency": { 2023-11-16T17:00:27.576 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.576 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.576 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.576 INFO:teuthology.orchestra.run.smithi141.stdout: "req_unlink_latency": { 2023-11-16T17:00:27.576 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.576 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.576 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.577 INFO:teuthology.orchestra.run.smithi141.stdout: "cap_revoke_eviction": 0, 2023-11-16T17:00:27.577 INFO:teuthology.orchestra.run.smithi141.stdout: "cap_acquisition_throttle": 0, 2023-11-16T17:00:27.577 INFO:teuthology.orchestra.run.smithi141.stdout: "req_getvxattr_latency": { 2023-11-16T17:00:27.577 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.577 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.577 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.578 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:27.578 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.578 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_sessions": { 2023-11-16T17:00:27.578 INFO:teuthology.orchestra.run.smithi141.stdout: "session_count": 2, 2023-11-16T17:00:27.578 INFO:teuthology.orchestra.run.smithi141.stdout: "session_add": 5, 2023-11-16T17:00:27.579 INFO:teuthology.orchestra.run.smithi141.stdout: "session_remove": 3, 2023-11-16T17:00:27.579 INFO:teuthology.orchestra.run.smithi141.stdout: "sessions_open": 4, 2023-11-16T17:00:27.579 INFO:teuthology.orchestra.run.smithi141.stdout: "sessions_stale": 0, 2023-11-16T17:00:27.579 INFO:teuthology.orchestra.run.smithi141.stdout: "total_load": 0, 2023-11-16T17:00:27.579 INFO:teuthology.orchestra.run.smithi141.stdout: "average_load": 0, 2023-11-16T17:00:27.579 INFO:teuthology.orchestra.run.smithi141.stdout: "avg_session_uptime": 4874 2023-11-16T17:00:27.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.579 INFO:teuthology.orchestra.run.smithi141.stdout: "mempool": { 2023-11-16T17:00:27.580 INFO:teuthology.orchestra.run.smithi141.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:00:27.580 INFO:teuthology.orchestra.run.smithi141.stdout: "bloom_filter_items": 0, 2023-11-16T17:00:27.580 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:00:27.580 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:00:27.580 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:00:27.580 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:00:27.580 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:00:27.580 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:00:27.581 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:00:27.581 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:00:27.581 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:00:27.581 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:00:27.581 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:00:27.581 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:00:27.581 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:00:27.581 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_extent_items": 0, 2023-11-16T17:00:27.582 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:00:27.582 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_blob_items": 0, 2023-11-16T17:00:27.582 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:00:27.582 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:00:27.582 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:00:27.582 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:00:27.582 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:00:27.582 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:00:27.583 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:00:27.583 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_txc_items": 0, 2023-11-16T17:00:27.583 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:00:27.583 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:00:27.583 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:00:27.583 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_writing_items": 0, 2023-11-16T17:00:27.583 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_bytes": 0, 2023-11-16T17:00:27.583 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_items": 0, 2023-11-16T17:00:27.584 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:00:27.584 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:00:27.584 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:00:27.584 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:00:27.584 INFO:teuthology.orchestra.run.smithi141.stdout: "buffer_anon_bytes": 230617, 2023-11-16T17:00:27.584 INFO:teuthology.orchestra.run.smithi141.stdout: "buffer_anon_items": 74, 2023-11-16T17:00:27.584 INFO:teuthology.orchestra.run.smithi141.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:00:27.585 INFO:teuthology.orchestra.run.smithi141.stdout: "buffer_meta_items": 0, 2023-11-16T17:00:27.585 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_bytes": 0, 2023-11-16T17:00:27.585 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_items": 0, 2023-11-16T17:00:27.585 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:00:27.585 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_mapbl_items": 0, 2023-11-16T17:00:27.585 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:00:27.585 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_pglog_items": 0, 2023-11-16T17:00:27.585 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_bytes": 9944, 2023-11-16T17:00:27.585 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_items": 192, 2023-11-16T17:00:27.586 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:00:27.586 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:00:27.586 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap_bytes": 0, 2023-11-16T17:00:27.586 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap_items": 0, 2023-11-16T17:00:27.586 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_co_bytes": 11911336, 2023-11-16T17:00:27.586 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_co_items": 67369, 2023-11-16T17:00:27.586 INFO:teuthology.orchestra.run.smithi141.stdout: "unittest_1_bytes": 0, 2023-11-16T17:00:27.586 INFO:teuthology.orchestra.run.smithi141.stdout: "unittest_1_items": 0, 2023-11-16T17:00:27.587 INFO:teuthology.orchestra.run.smithi141.stdout: "unittest_2_bytes": 0, 2023-11-16T17:00:27.587 INFO:teuthology.orchestra.run.smithi141.stdout: "unittest_2_items": 0 2023-11-16T17:00:27.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.587 INFO:teuthology.orchestra.run.smithi141.stdout: "objecter": { 2023-11-16T17:00:27.587 INFO:teuthology.orchestra.run.smithi141.stdout: "op_active": 0, 2023-11-16T17:00:27.587 INFO:teuthology.orchestra.run.smithi141.stdout: "op_laggy": 0, 2023-11-16T17:00:27.587 INFO:teuthology.orchestra.run.smithi141.stdout: "op_send": 2661, 2023-11-16T17:00:27.587 INFO:teuthology.orchestra.run.smithi141.stdout: "op_send_bytes": 21, 2023-11-16T17:00:27.588 INFO:teuthology.orchestra.run.smithi141.stdout: "op_resend": 3, 2023-11-16T17:00:27.588 INFO:teuthology.orchestra.run.smithi141.stdout: "op_reply": 2661, 2023-11-16T17:00:27.588 INFO:teuthology.orchestra.run.smithi141.stdout: "op_latency": { 2023-11-16T17:00:27.588 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 2661, 2023-11-16T17:00:27.588 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 3.708921556, 2023-11-16T17:00:27.588 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.001393807 2023-11-16T17:00:27.588 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.588 INFO:teuthology.orchestra.run.smithi141.stdout: "op_inflight": 0, 2023-11-16T17:00:27.589 INFO:teuthology.orchestra.run.smithi141.stdout: "oplen_avg": { 2023-11-16T17:00:27.589 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 2661, 2023-11-16T17:00:27.589 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 2662 2023-11-16T17:00:27.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.589 INFO:teuthology.orchestra.run.smithi141.stdout: "op": 2661, 2023-11-16T17:00:27.589 INFO:teuthology.orchestra.run.smithi141.stdout: "op_r": 2661, 2023-11-16T17:00:27.589 INFO:teuthology.orchestra.run.smithi141.stdout: "op_w": 0, 2023-11-16T17:00:27.589 INFO:teuthology.orchestra.run.smithi141.stdout: "op_rmw": 0, 2023-11-16T17:00:27.590 INFO:teuthology.orchestra.run.smithi141.stdout: "op_pg": 0, 2023-11-16T17:00:27.590 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_stat": 1700, 2023-11-16T17:00:27.590 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_create": 0, 2023-11-16T17:00:27.590 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_read": 959, 2023-11-16T17:00:27.590 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_write": 0, 2023-11-16T17:00:27.590 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_writefull": 0, 2023-11-16T17:00:27.590 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_writesame": 0, 2023-11-16T17:00:27.590 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_append": 0, 2023-11-16T17:00:27.591 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_zero": 0, 2023-11-16T17:00:27.591 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_truncate": 0, 2023-11-16T17:00:27.591 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_delete": 0, 2023-11-16T17:00:27.591 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_mapext": 0, 2023-11-16T17:00:27.591 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_sparse_read": 0, 2023-11-16T17:00:27.591 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_clonerange": 0, 2023-11-16T17:00:27.591 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_getxattr": 1, 2023-11-16T17:00:27.591 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_setxattr": 0, 2023-11-16T17:00:27.592 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_cmpxattr": 0, 2023-11-16T17:00:27.592 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_rmxattr": 0, 2023-11-16T17:00:27.592 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_resetxattrs": 0, 2023-11-16T17:00:27.592 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_call": 0, 2023-11-16T17:00:27.592 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_watch": 0, 2023-11-16T17:00:27.592 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_notify": 0, 2023-11-16T17:00:27.592 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_src_cmpxattr": 0, 2023-11-16T17:00:27.592 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_pgls": 0, 2023-11-16T17:00:27.593 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_pgls_filter": 0, 2023-11-16T17:00:27.593 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_other": 0, 2023-11-16T17:00:27.593 INFO:teuthology.orchestra.run.smithi141.stdout: "linger_active": 0, 2023-11-16T17:00:27.593 INFO:teuthology.orchestra.run.smithi141.stdout: "linger_send": 0, 2023-11-16T17:00:27.593 INFO:teuthology.orchestra.run.smithi141.stdout: "linger_resend": 0, 2023-11-16T17:00:27.593 INFO:teuthology.orchestra.run.smithi141.stdout: "linger_ping": 0, 2023-11-16T17:00:27.593 INFO:teuthology.orchestra.run.smithi141.stdout: "poolop_active": 0, 2023-11-16T17:00:27.593 INFO:teuthology.orchestra.run.smithi141.stdout: "poolop_send": 0, 2023-11-16T17:00:27.594 INFO:teuthology.orchestra.run.smithi141.stdout: "poolop_resend": 0, 2023-11-16T17:00:27.594 INFO:teuthology.orchestra.run.smithi141.stdout: "poolstat_active": 0, 2023-11-16T17:00:27.594 INFO:teuthology.orchestra.run.smithi141.stdout: "poolstat_send": 0, 2023-11-16T17:00:27.594 INFO:teuthology.orchestra.run.smithi141.stdout: "poolstat_resend": 0, 2023-11-16T17:00:27.594 INFO:teuthology.orchestra.run.smithi141.stdout: "statfs_active": 0, 2023-11-16T17:00:27.594 INFO:teuthology.orchestra.run.smithi141.stdout: "statfs_send": 0, 2023-11-16T17:00:27.594 INFO:teuthology.orchestra.run.smithi141.stdout: "statfs_resend": 0, 2023-11-16T17:00:27.594 INFO:teuthology.orchestra.run.smithi141.stdout: "command_active": 0, 2023-11-16T17:00:27.594 INFO:teuthology.orchestra.run.smithi141.stdout: "command_send": 0, 2023-11-16T17:00:27.595 INFO:teuthology.orchestra.run.smithi141.stdout: "command_resend": 0, 2023-11-16T17:00:27.595 INFO:teuthology.orchestra.run.smithi141.stdout: "map_epoch": 104, 2023-11-16T17:00:27.595 INFO:teuthology.orchestra.run.smithi141.stdout: "map_full": 0, 2023-11-16T17:00:27.595 INFO:teuthology.orchestra.run.smithi141.stdout: "map_inc": 13, 2023-11-16T17:00:27.595 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_sessions": 10, 2023-11-16T17:00:27.595 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_session_open": 10, 2023-11-16T17:00:27.595 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_session_close": 0, 2023-11-16T17:00:27.595 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_laggy": 0, 2023-11-16T17:00:27.595 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_wr": 0, 2023-11-16T17:00:27.596 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_rd": 2, 2023-11-16T17:00:27.596 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_del": 0 2023-11-16T17:00:27.596 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.596 INFO:teuthology.orchestra.run.smithi141.stdout: "oft": { 2023-11-16T17:00:27.596 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_total_objs": 0, 2023-11-16T17:00:27.596 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_total_kv_pairs": 0, 2023-11-16T17:00:27.596 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_total_updates": 0, 2023-11-16T17:00:27.596 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_total_removes": 0 2023-11-16T17:00:27.597 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.597 INFO:teuthology.orchestra.run.smithi141.stdout: "osd-slow-ops": { 2023-11-16T17:00:27.597 INFO:teuthology.orchestra.run.smithi141.stdout: "slow_ops_count": 0 2023-11-16T17:00:27.597 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.597 INFO:teuthology.orchestra.run.smithi141.stdout: "purge_queue": { 2023-11-16T17:00:27.597 INFO:teuthology.orchestra.run.smithi141.stdout: "pq_executing_ops": 0, 2023-11-16T17:00:27.597 INFO:teuthology.orchestra.run.smithi141.stdout: "pq_executing_ops_high_water": 0, 2023-11-16T17:00:27.597 INFO:teuthology.orchestra.run.smithi141.stdout: "pq_executing": 0, 2023-11-16T17:00:27.598 INFO:teuthology.orchestra.run.smithi141.stdout: "pq_executing_high_water": 0, 2023-11-16T17:00:27.598 INFO:teuthology.orchestra.run.smithi141.stdout: "pq_executed": 0, 2023-11-16T17:00:27.598 INFO:teuthology.orchestra.run.smithi141.stdout: "pq_item_in_journal": 0 2023-11-16T17:00:27.598 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.598 INFO:teuthology.orchestra.run.smithi141.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:00:27.598 INFO:teuthology.orchestra.run.smithi141.stdout: "val": 0, 2023-11-16T17:00:27.598 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 104857600, 2023-11-16T17:00:27.598 INFO:teuthology.orchestra.run.smithi141.stdout: "get_started": 0, 2023-11-16T17:00:27.599 INFO:teuthology.orchestra.run.smithi141.stdout: "get": 2924, 2023-11-16T17:00:27.599 INFO:teuthology.orchestra.run.smithi141.stdout: "get_sum": 263021075, 2023-11-16T17:00:27.599 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:27.599 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_success": 2924, 2023-11-16T17:00:27.599 INFO:teuthology.orchestra.run.smithi141.stdout: "take": 0, 2023-11-16T17:00:27.599 INFO:teuthology.orchestra.run.smithi141.stdout: "take_sum": 0, 2023-11-16T17:00:27.599 INFO:teuthology.orchestra.run.smithi141.stdout: "put": 2924, 2023-11-16T17:00:27.599 INFO:teuthology.orchestra.run.smithi141.stdout: "put_sum": 263021075, 2023-11-16T17:00:27.600 INFO:teuthology.orchestra.run.smithi141.stdout: "wait": { 2023-11-16T17:00:27.600 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.600 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.600 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.600 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:27.600 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.600 INFO:teuthology.orchestra.run.smithi141.stdout: "throttle-objecter_bytes": { 2023-11-16T17:00:27.600 INFO:teuthology.orchestra.run.smithi141.stdout: "val": 0, 2023-11-16T17:00:27.601 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 104857600, 2023-11-16T17:00:27.601 INFO:teuthology.orchestra.run.smithi141.stdout: "get_started": 0, 2023-11-16T17:00:27.601 INFO:teuthology.orchestra.run.smithi141.stdout: "get": 0, 2023-11-16T17:00:27.601 INFO:teuthology.orchestra.run.smithi141.stdout: "get_sum": 0, 2023-11-16T17:00:27.601 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:27.601 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_success": 0, 2023-11-16T17:00:27.601 INFO:teuthology.orchestra.run.smithi141.stdout: "take": 2661, 2023-11-16T17:00:27.601 INFO:teuthology.orchestra.run.smithi141.stdout: "take_sum": 261038733, 2023-11-16T17:00:27.601 INFO:teuthology.orchestra.run.smithi141.stdout: "put": 2661, 2023-11-16T17:00:27.602 INFO:teuthology.orchestra.run.smithi141.stdout: "put_sum": 261038733, 2023-11-16T17:00:27.602 INFO:teuthology.orchestra.run.smithi141.stdout: "wait": { 2023-11-16T17:00:27.602 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.602 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.602 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.602 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:27.602 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.602 INFO:teuthology.orchestra.run.smithi141.stdout: "throttle-objecter_ops": { 2023-11-16T17:00:27.603 INFO:teuthology.orchestra.run.smithi141.stdout: "val": 0, 2023-11-16T17:00:27.603 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1024, 2023-11-16T17:00:27.603 INFO:teuthology.orchestra.run.smithi141.stdout: "get_started": 0, 2023-11-16T17:00:27.603 INFO:teuthology.orchestra.run.smithi141.stdout: "get": 0, 2023-11-16T17:00:27.603 INFO:teuthology.orchestra.run.smithi141.stdout: "get_sum": 0, 2023-11-16T17:00:27.603 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:27.603 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_success": 0, 2023-11-16T17:00:27.603 INFO:teuthology.orchestra.run.smithi141.stdout: "take": 2661, 2023-11-16T17:00:27.604 INFO:teuthology.orchestra.run.smithi141.stdout: "take_sum": 2661, 2023-11-16T17:00:27.604 INFO:teuthology.orchestra.run.smithi141.stdout: "put": 2661, 2023-11-16T17:00:27.604 INFO:teuthology.orchestra.run.smithi141.stdout: "put_sum": 2661, 2023-11-16T17:00:27.604 INFO:teuthology.orchestra.run.smithi141.stdout: "wait": { 2023-11-16T17:00:27.604 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.604 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.604 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.604 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:27.604 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.605 INFO:teuthology.orchestra.run.smithi141.stdout: "throttle-write_buf_throttle": { 2023-11-16T17:00:27.605 INFO:teuthology.orchestra.run.smithi141.stdout: "val": 0, 2023-11-16T17:00:27.605 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 3758096384, 2023-11-16T17:00:27.605 INFO:teuthology.orchestra.run.smithi141.stdout: "get_started": 0, 2023-11-16T17:00:27.605 INFO:teuthology.orchestra.run.smithi141.stdout: "get": 0, 2023-11-16T17:00:27.605 INFO:teuthology.orchestra.run.smithi141.stdout: "get_sum": 0, 2023-11-16T17:00:27.605 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:27.605 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_success": 0, 2023-11-16T17:00:27.606 INFO:teuthology.orchestra.run.smithi141.stdout: "take": 0, 2023-11-16T17:00:27.606 INFO:teuthology.orchestra.run.smithi141.stdout: "take_sum": 0, 2023-11-16T17:00:27.606 INFO:teuthology.orchestra.run.smithi141.stdout: "put": 0, 2023-11-16T17:00:27.606 INFO:teuthology.orchestra.run.smithi141.stdout: "put_sum": 0, 2023-11-16T17:00:27.606 INFO:teuthology.orchestra.run.smithi141.stdout: "wait": { 2023-11-16T17:00:27.606 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.606 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.606 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.607 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:27.607 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:27.607 INFO:teuthology.orchestra.run.smithi141.stdout: "throttle-write_buf_throttle-0x5593fc9aa460": { 2023-11-16T17:00:27.607 INFO:teuthology.orchestra.run.smithi141.stdout: "val": 0, 2023-11-16T17:00:27.607 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 3758096384, 2023-11-16T17:00:27.607 INFO:teuthology.orchestra.run.smithi141.stdout: "get_started": 0, 2023-11-16T17:00:27.607 INFO:teuthology.orchestra.run.smithi141.stdout: "get": 0, 2023-11-16T17:00:27.607 INFO:teuthology.orchestra.run.smithi141.stdout: "get_sum": 0, 2023-11-16T17:00:27.608 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:27.608 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_success": 0, 2023-11-16T17:00:27.608 INFO:teuthology.orchestra.run.smithi141.stdout: "take": 0, 2023-11-16T17:00:27.608 INFO:teuthology.orchestra.run.smithi141.stdout: "take_sum": 0, 2023-11-16T17:00:27.608 INFO:teuthology.orchestra.run.smithi141.stdout: "put": 0, 2023-11-16T17:00:27.608 INFO:teuthology.orchestra.run.smithi141.stdout: "put_sum": 0, 2023-11-16T17:00:27.608 INFO:teuthology.orchestra.run.smithi141.stdout: "wait": { 2023-11-16T17:00:27.608 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:27.608 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:27.609 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:27.609 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:27.609 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:27.609 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-11-16T17:00:27.610 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2023-11-16T17:00:27.610 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2023-11-16T17:00:27.610 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2023-11-16T17:00:27.610 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2023-11-16T17:00:27.610 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2023-11-16T17:00:27.610 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2023-11-16T17:00:27.611 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2023-11-16T17:00:27.611 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2023-11-16T17:00:27.611 DEBUG:tasks.check_counter:Getting stats from h 2023-11-16T17:00:27.611 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.h perf dump 2023-11-16T17:00:27.611 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:27 smithi141 ceph-mon[131184]: pgmap v328: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 4 op/s 2023-11-16T17:00:28.384 INFO:teuthology.orchestra.run.smithi141.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T17:00:28.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:28 smithi002 ceph-mon[128712]: pgmap v329: 129 pgs: 20 active+clean+snaptrim, 61 active+clean+snaptrim_wait, 48 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-11-16T17:00:28.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:28 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:28.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:28 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:28.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:28 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:28.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:28 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:28.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:28 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:28.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:28 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:28.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:28 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:28.907 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:28 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:28.992 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:28 smithi141 ceph-mon[131184]: pgmap v329: 129 pgs: 20 active+clean+snaptrim, 61 active+clean+snaptrim_wait, 48 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-11-16T17:00:28.993 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:28 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:28.993 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:28 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:28.993 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:28 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:28.993 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:28 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:28.993 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:28 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:28.993 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:28 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:28.993 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:28 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:28.994 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:28 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:29.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:28 smithi155 ceph-mon[131607]: pgmap v329: 129 pgs: 20 active+clean+snaptrim, 61 active+clean+snaptrim_wait, 48 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-11-16T17:00:29.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:28 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:29.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:28 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:29.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:28 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:29.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:28 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:29.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:28 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:29.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:28 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:29.120 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:28 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:29.120 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:28 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:29.784 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-11-16T17:00:29.784 INFO:teuthology.orchestra.run.smithi141.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:00:29.784 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_messages": 23768, 2023-11-16T17:00:29.785 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages": 25029, 2023-11-16T17:00:29.785 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_bytes": 72294031, 2023-11-16T17:00:29.785 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_bytes": 89504938, 2023-11-16T17:00:29.785 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_created_connections": 10, 2023-11-16T17:00:29.785 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_active_connections": 4, 2023-11-16T17:00:29.785 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_total_time": 1.673056150, 2023-11-16T17:00:29.786 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_send_time": 0.672413560, 2023-11-16T17:00:29.786 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_recv_time": 0.630419043, 2023-11-16T17:00:29.786 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_fast_dispatch_time": 0.410802952, 2023-11-16T17:00:29.786 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:29.786 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 25030, 2023-11-16T17:00:29.787 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 4.054667541, 2023-11-16T17:00:29.787 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000161992 2023-11-16T17:00:29.787 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.787 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:29.787 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 19027, 2023-11-16T17:00:29.787 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.003256981, 2023-11-16T17:00:29.788 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000171 2023-11-16T17:00:29.788 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.788 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_encrypted_bytes": 3152, 2023-11-16T17:00:29.788 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_encrypted_bytes": 305008 2023-11-16T17:00:29.788 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.789 INFO:teuthology.orchestra.run.smithi141.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:00:29.789 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_messages": 90351, 2023-11-16T17:00:29.789 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages": 152041, 2023-11-16T17:00:29.789 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_bytes": 31182157, 2023-11-16T17:00:29.789 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_bytes": 179804073, 2023-11-16T17:00:29.789 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_created_connections": 16, 2023-11-16T17:00:29.790 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_active_connections": 9, 2023-11-16T17:00:29.790 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_total_time": 5.301780336, 2023-11-16T17:00:29.790 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_send_time": 2.290876632, 2023-11-16T17:00:29.790 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_recv_time": 7.261972441, 2023-11-16T17:00:29.790 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_fast_dispatch_time": 0.456158677, 2023-11-16T17:00:29.791 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:29.791 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 152036, 2023-11-16T17:00:29.791 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 15.386911696, 2023-11-16T17:00:29.791 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000101205 2023-11-16T17:00:29.791 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.791 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:29.792 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 177610, 2023-11-16T17:00:29.792 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.033728938, 2023-11-16T17:00:29.792 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000189 2023-11-16T17:00:29.792 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.792 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_encrypted_bytes": 3680, 2023-11-16T17:00:29.792 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_encrypted_bytes": 135552 2023-11-16T17:00:29.793 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.793 INFO:teuthology.orchestra.run.smithi141.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:00:29.793 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_messages": 21344, 2023-11-16T17:00:29.793 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages": 24874, 2023-11-16T17:00:29.793 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_bytes": 31767412, 2023-11-16T17:00:29.794 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_bytes": 93191177, 2023-11-16T17:00:29.794 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_created_connections": 10, 2023-11-16T17:00:29.794 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_active_connections": 7, 2023-11-16T17:00:29.794 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_total_time": 1.488410129, 2023-11-16T17:00:29.794 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_send_time": 0.580949228, 2023-11-16T17:00:29.794 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_recv_time": 0.648990087, 2023-11-16T17:00:29.795 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_fast_dispatch_time": 0.319200751, 2023-11-16T17:00:29.795 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:29.795 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 24873, 2023-11-16T17:00:29.795 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 5.117077191, 2023-11-16T17:00:29.795 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000205728 2023-11-16T17:00:29.795 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.796 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:29.796 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 21692, 2023-11-16T17:00:29.796 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.003993213, 2023-11-16T17:00:29.796 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000184 2023-11-16T17:00:29.796 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.796 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_encrypted_bytes": 1529728, 2023-11-16T17:00:29.797 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_encrypted_bytes": 133360 2023-11-16T17:00:29.797 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.797 INFO:teuthology.orchestra.run.smithi141.stdout: "cct": { 2023-11-16T17:00:29.797 INFO:teuthology.orchestra.run.smithi141.stdout: "total_workers": 1, 2023-11-16T17:00:29.797 INFO:teuthology.orchestra.run.smithi141.stdout: "unhealthy_workers": 0 2023-11-16T17:00:29.797 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.798 INFO:teuthology.orchestra.run.smithi141.stdout: "finisher-MDSRank": { 2023-11-16T17:00:29.798 INFO:teuthology.orchestra.run.smithi141.stdout: "queue_len": 0, 2023-11-16T17:00:29.798 INFO:teuthology.orchestra.run.smithi141.stdout: "complete_latency": { 2023-11-16T17:00:29.798 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 3769, 2023-11-16T17:00:29.798 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 54.028321807, 2023-11-16T17:00:29.799 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.014334922 2023-11-16T17:00:29.799 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:29.799 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.799 INFO:teuthology.orchestra.run.smithi141.stdout: "finisher-PurgeQueue": { 2023-11-16T17:00:29.799 INFO:teuthology.orchestra.run.smithi141.stdout: "queue_len": 0, 2023-11-16T17:00:29.799 INFO:teuthology.orchestra.run.smithi141.stdout: "complete_latency": { 2023-11-16T17:00:29.800 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 28598, 2023-11-16T17:00:29.800 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 11.176744262, 2023-11-16T17:00:29.800 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000390822 2023-11-16T17:00:29.800 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:29.800 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.800 INFO:teuthology.orchestra.run.smithi141.stdout: "mds": { 2023-11-16T17:00:29.801 INFO:teuthology.orchestra.run.smithi141.stdout: "request": 50154, 2023-11-16T17:00:29.801 INFO:teuthology.orchestra.run.smithi141.stdout: "reply": 45145, 2023-11-16T17:00:29.801 INFO:teuthology.orchestra.run.smithi141.stdout: "reply_latency": { 2023-11-16T17:00:29.801 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 45145, 2023-11-16T17:00:29.801 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 129.136504297, 2023-11-16T17:00:29.801 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.002860482 2023-11-16T17:00:29.802 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.802 INFO:teuthology.orchestra.run.smithi141.stdout: "slow_reply": 0, 2023-11-16T17:00:29.802 INFO:teuthology.orchestra.run.smithi141.stdout: "forward": 5009, 2023-11-16T17:00:29.802 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_fetch_complete": 153, 2023-11-16T17:00:29.802 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_fetch_keys": 0, 2023-11-16T17:00:29.802 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_commit": 747, 2023-11-16T17:00:29.803 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_split": 92, 2023-11-16T17:00:29.803 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_merge": 0, 2023-11-16T17:00:29.803 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes": 19, 2023-11-16T17:00:29.803 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes_top": 2, 2023-11-16T17:00:29.803 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes_bottom": 7, 2023-11-16T17:00:29.803 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes_pin_tail": 10, 2023-11-16T17:00:29.804 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes_pinned": 15, 2023-11-16T17:00:29.804 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes_expired": 71266, 2023-11-16T17:00:29.804 INFO:teuthology.orchestra.run.smithi141.stdout: "inodes_with_caps": 2, 2023-11-16T17:00:29.804 INFO:teuthology.orchestra.run.smithi141.stdout: "caps": 2, 2023-11-16T17:00:29.805 INFO:teuthology.orchestra.run.smithi141.stdout: "subtrees": 6, 2023-11-16T17:00:29.805 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse": 68587, 2023-11-16T17:00:29.805 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse_hit": 63188, 2023-11-16T17:00:29.805 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse_forward": 4878, 2023-11-16T17:00:29.805 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse_discover": 10, 2023-11-16T17:00:29.806 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse_dir_fetch": 153, 2023-11-16T17:00:29.806 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse_remote_ino": 0, 2023-11-16T17:00:29.806 INFO:teuthology.orchestra.run.smithi141.stdout: "traverse_lock": 0, 2023-11-16T17:00:29.806 INFO:teuthology.orchestra.run.smithi141.stdout: "load_cent": 0, 2023-11-16T17:00:29.806 INFO:teuthology.orchestra.run.smithi141.stdout: "q": 0, 2023-11-16T17:00:29.806 INFO:teuthology.orchestra.run.smithi141.stdout: "exported": 1339, 2023-11-16T17:00:29.807 INFO:teuthology.orchestra.run.smithi141.stdout: "exported_inodes": 66686, 2023-11-16T17:00:29.807 INFO:teuthology.orchestra.run.smithi141.stdout: "imported": 696, 2023-11-16T17:00:29.807 INFO:teuthology.orchestra.run.smithi141.stdout: "imported_inodes": 0, 2023-11-16T17:00:29.807 INFO:teuthology.orchestra.run.smithi141.stdout: "openino_dir_fetch": 0, 2023-11-16T17:00:29.807 INFO:teuthology.orchestra.run.smithi141.stdout: "openino_backtrace_fetch": 0, 2023-11-16T17:00:29.807 INFO:teuthology.orchestra.run.smithi141.stdout: "openino_peer_discover": 0, 2023-11-16T17:00:29.807 INFO:teuthology.orchestra.run.smithi141.stdout: "root_rfiles": 4, 2023-11-16T17:00:29.808 INFO:teuthology.orchestra.run.smithi141.stdout: "root_rbytes": 246, 2023-11-16T17:00:29.808 INFO:teuthology.orchestra.run.smithi141.stdout: "root_rsnaps": 0, 2023-11-16T17:00:29.808 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_backtrace_fetch": 0, 2023-11-16T17:00:29.808 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_set_tag": 0, 2023-11-16T17:00:29.808 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_backtrace_repaired": 0, 2023-11-16T17:00:29.808 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_inotable_repaired": 0, 2023-11-16T17:00:29.809 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_dir_inodes": 0, 2023-11-16T17:00:29.809 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_dir_base_inodes": 0, 2023-11-16T17:00:29.809 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_dirfrag_rstats": 0, 2023-11-16T17:00:29.809 INFO:teuthology.orchestra.run.smithi141.stdout: "scrub_file_inodes": 0, 2023-11-16T17:00:29.809 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_inode_file_caps": 0, 2023-11-16T17:00:29.810 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_cap_op_revoke": 598, 2023-11-16T17:00:29.810 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_cap_op_grant": 28384, 2023-11-16T17:00:29.810 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_cap_op_trunc": 0, 2023-11-16T17:00:29.810 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-11-16T17:00:29.810 INFO:teuthology.orchestra.run.smithi141.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-16T17:00:29.810 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_client_caps": 26384, 2023-11-16T17:00:29.811 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_client_caps_dirty": 25757, 2023-11-16T17:00:29.811 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_client_cap_release": 312, 2023-11-16T17:00:29.811 INFO:teuthology.orchestra.run.smithi141.stdout: "process_request_cap_release": 34480 2023-11-16T17:00:29.811 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.811 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_cache": { 2023-11-16T17:00:29.811 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_update": 912, 2023-11-16T17:00:29.811 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_update_receipt": 3419, 2023-11-16T17:00:29.812 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_try_discover": 1712, 2023-11-16T17:00:29.812 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_send_discover": 1720, 2023-11-16T17:00:29.812 INFO:teuthology.orchestra.run.smithi141.stdout: "dir_handle_discover": 923, 2023-11-16T17:00:29.812 INFO:teuthology.orchestra.run.smithi141.stdout: "num_strays": 0, 2023-11-16T17:00:29.812 INFO:teuthology.orchestra.run.smithi141.stdout: "num_strays_delayed": 0, 2023-11-16T17:00:29.812 INFO:teuthology.orchestra.run.smithi141.stdout: "num_strays_enqueuing": 0, 2023-11-16T17:00:29.813 INFO:teuthology.orchestra.run.smithi141.stdout: "strays_created": 16773, 2023-11-16T17:00:29.813 INFO:teuthology.orchestra.run.smithi141.stdout: "strays_enqueued": 16773, 2023-11-16T17:00:29.813 INFO:teuthology.orchestra.run.smithi141.stdout: "strays_reintegrated": 0, 2023-11-16T17:00:29.813 INFO:teuthology.orchestra.run.smithi141.stdout: "strays_migrated": 0, 2023-11-16T17:00:29.813 INFO:teuthology.orchestra.run.smithi141.stdout: "num_recovering_processing": 0, 2023-11-16T17:00:29.813 INFO:teuthology.orchestra.run.smithi141.stdout: "num_recovering_enqueued": 0, 2023-11-16T17:00:29.814 INFO:teuthology.orchestra.run.smithi141.stdout: "num_recovering_prioritized": 0, 2023-11-16T17:00:29.814 INFO:teuthology.orchestra.run.smithi141.stdout: "recovery_started": 0, 2023-11-16T17:00:29.814 INFO:teuthology.orchestra.run.smithi141.stdout: "recovery_completed": 0, 2023-11-16T17:00:29.814 INFO:teuthology.orchestra.run.smithi141.stdout: "ireq_enqueue_scrub": 0, 2023-11-16T17:00:29.814 INFO:teuthology.orchestra.run.smithi141.stdout: "ireq_exportdir": 1404, 2023-11-16T17:00:29.814 INFO:teuthology.orchestra.run.smithi141.stdout: "ireq_flush": 0, 2023-11-16T17:00:29.815 INFO:teuthology.orchestra.run.smithi141.stdout: "ireq_fragmentdir": 92, 2023-11-16T17:00:29.815 INFO:teuthology.orchestra.run.smithi141.stdout: "ireq_fragstats": 0, 2023-11-16T17:00:29.815 INFO:teuthology.orchestra.run.smithi141.stdout: "ireq_inodestats": 0 2023-11-16T17:00:29.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.815 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_log": { 2023-11-16T17:00:29.815 INFO:teuthology.orchestra.run.smithi141.stdout: "evadd": 92995, 2023-11-16T17:00:29.815 INFO:teuthology.orchestra.run.smithi141.stdout: "evex": 0, 2023-11-16T17:00:29.816 INFO:teuthology.orchestra.run.smithi141.stdout: "evtrm": 0, 2023-11-16T17:00:29.816 INFO:teuthology.orchestra.run.smithi141.stdout: "ev": 92995, 2023-11-16T17:00:29.816 INFO:teuthology.orchestra.run.smithi141.stdout: "evexg": 0, 2023-11-16T17:00:29.816 INFO:teuthology.orchestra.run.smithi141.stdout: "evexd": 0, 2023-11-16T17:00:29.816 INFO:teuthology.orchestra.run.smithi141.stdout: "segadd": 125, 2023-11-16T17:00:29.816 INFO:teuthology.orchestra.run.smithi141.stdout: "segex": 0, 2023-11-16T17:00:29.817 INFO:teuthology.orchestra.run.smithi141.stdout: "segtrm": 0, 2023-11-16T17:00:29.817 INFO:teuthology.orchestra.run.smithi141.stdout: "seg": 125, 2023-11-16T17:00:29.817 INFO:teuthology.orchestra.run.smithi141.stdout: "segexg": 0, 2023-11-16T17:00:29.817 INFO:teuthology.orchestra.run.smithi141.stdout: "segexd": 0, 2023-11-16T17:00:29.817 INFO:teuthology.orchestra.run.smithi141.stdout: "expos": 4194304, 2023-11-16T17:00:29.817 INFO:teuthology.orchestra.run.smithi141.stdout: "wrpos": 263950820, 2023-11-16T17:00:29.818 INFO:teuthology.orchestra.run.smithi141.stdout: "rdpos": 0, 2023-11-16T17:00:29.818 INFO:teuthology.orchestra.run.smithi141.stdout: "jlat": { 2023-11-16T17:00:29.818 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 3136, 2023-11-16T17:00:29.818 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 170.935758867, 2023-11-16T17:00:29.818 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.054507576 2023-11-16T17:00:29.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.819 INFO:teuthology.orchestra.run.smithi141.stdout: "replayed": 0 2023-11-16T17:00:29.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.819 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_mem": { 2023-11-16T17:00:29.819 INFO:teuthology.orchestra.run.smithi141.stdout: "ino": 24, 2023-11-16T17:00:29.819 INFO:teuthology.orchestra.run.smithi141.stdout: "ino+": 61230, 2023-11-16T17:00:29.819 INFO:teuthology.orchestra.run.smithi141.stdout: "ino-": 61206, 2023-11-16T17:00:29.820 INFO:teuthology.orchestra.run.smithi141.stdout: "dir": 22, 2023-11-16T17:00:29.820 INFO:teuthology.orchestra.run.smithi141.stdout: "dir+": 2713, 2023-11-16T17:00:29.820 INFO:teuthology.orchestra.run.smithi141.stdout: "dir-": 2691, 2023-11-16T17:00:29.820 INFO:teuthology.orchestra.run.smithi141.stdout: "dn": 19, 2023-11-16T17:00:29.820 INFO:teuthology.orchestra.run.smithi141.stdout: "dn+": 88058, 2023-11-16T17:00:29.820 INFO:teuthology.orchestra.run.smithi141.stdout: "dn-": 88039, 2023-11-16T17:00:29.820 INFO:teuthology.orchestra.run.smithi141.stdout: "cap": 2, 2023-11-16T17:00:29.821 INFO:teuthology.orchestra.run.smithi141.stdout: "cap+": 35957, 2023-11-16T17:00:29.821 INFO:teuthology.orchestra.run.smithi141.stdout: "cap-": 35955, 2023-11-16T17:00:29.821 INFO:teuthology.orchestra.run.smithi141.stdout: "rss": 180644, 2023-11-16T17:00:29.821 INFO:teuthology.orchestra.run.smithi141.stdout: "heap": 198964 2023-11-16T17:00:29.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.821 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_server": { 2023-11-16T17:00:29.822 INFO:teuthology.orchestra.run.smithi141.stdout: "dispatch_client_request": 51029, 2023-11-16T17:00:29.822 INFO:teuthology.orchestra.run.smithi141.stdout: "dispatch_server_request": 1, 2023-11-16T17:00:29.822 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_client_request": 50154, 2023-11-16T17:00:29.822 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_client_session": 30, 2023-11-16T17:00:29.822 INFO:teuthology.orchestra.run.smithi141.stdout: "handle_peer_request": 313, 2023-11-16T17:00:29.822 INFO:teuthology.orchestra.run.smithi141.stdout: "req_create_latency": { 2023-11-16T17:00:29.823 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 11333, 2023-11-16T17:00:29.823 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 25.406570717, 2023-11-16T17:00:29.823 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.002241822 2023-11-16T17:00:29.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.823 INFO:teuthology.orchestra.run.smithi141.stdout: "req_getattr_latency": { 2023-11-16T17:00:29.823 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 252, 2023-11-16T17:00:29.823 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.371584053, 2023-11-16T17:00:29.824 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.001474539 2023-11-16T17:00:29.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.824 INFO:teuthology.orchestra.run.smithi141.stdout: "req_getfilelock_latency": { 2023-11-16T17:00:29.824 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.824 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.824 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.825 INFO:teuthology.orchestra.run.smithi141.stdout: "req_link_latency": { 2023-11-16T17:00:29.825 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.825 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.825 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.825 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lookup_latency": { 2023-11-16T17:00:29.826 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 15955, 2023-11-16T17:00:29.826 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 31.702651668, 2023-11-16T17:00:29.826 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.001987004 2023-11-16T17:00:29.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.826 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lookuphash_latency": { 2023-11-16T17:00:29.826 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.827 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.827 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.827 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lookupino_latency": { 2023-11-16T17:00:29.827 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 1, 2023-11-16T17:00:29.827 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000104209, 2023-11-16T17:00:29.827 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000104209 2023-11-16T17:00:29.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.828 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lookupname_latency": { 2023-11-16T17:00:29.828 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.828 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.828 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.829 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lookupparent_latency": { 2023-11-16T17:00:29.829 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.829 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.829 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.829 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.829 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lookupsnap_latency": { 2023-11-16T17:00:29.830 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.830 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.830 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.830 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.830 INFO:teuthology.orchestra.run.smithi141.stdout: "req_lssnap_latency": { 2023-11-16T17:00:29.830 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.831 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.831 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.831 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.831 INFO:teuthology.orchestra.run.smithi141.stdout: "req_mkdir_latency": { 2023-11-16T17:00:29.831 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.831 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.832 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.832 INFO:teuthology.orchestra.run.smithi141.stdout: "req_mknod_latency": { 2023-11-16T17:00:29.832 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.832 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.832 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.833 INFO:teuthology.orchestra.run.smithi141.stdout: "req_mksnap_latency": { 2023-11-16T17:00:29.833 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.833 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.833 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.833 INFO:teuthology.orchestra.run.smithi141.stdout: "req_open_latency": { 2023-11-16T17:00:29.834 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.834 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.834 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.834 INFO:teuthology.orchestra.run.smithi141.stdout: "req_readdir_latency": { 2023-11-16T17:00:29.834 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 831, 2023-11-16T17:00:29.835 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 1.440083574, 2023-11-16T17:00:29.835 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.001732952 2023-11-16T17:00:29.835 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.835 INFO:teuthology.orchestra.run.smithi141.stdout: "req_rename_latency": { 2023-11-16T17:00:29.835 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.835 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.835 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.836 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.836 INFO:teuthology.orchestra.run.smithi141.stdout: "req_renamesnap_latency": { 2023-11-16T17:00:29.836 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.836 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.836 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.836 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.837 INFO:teuthology.orchestra.run.smithi141.stdout: "req_snapdiff_latency": { 2023-11-16T17:00:29.837 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.837 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.837 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.837 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.837 INFO:teuthology.orchestra.run.smithi141.stdout: "req_rmdir_latency": { 2023-11-16T17:00:29.837 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.838 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.838 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.838 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.838 INFO:teuthology.orchestra.run.smithi141.stdout: "req_rmsnap_latency": { 2023-11-16T17:00:29.838 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.838 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.839 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.839 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.839 INFO:teuthology.orchestra.run.smithi141.stdout: "req_rmxattr_latency": { 2023-11-16T17:00:29.839 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.839 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.839 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.839 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.840 INFO:teuthology.orchestra.run.smithi141.stdout: "req_setattr_latency": { 2023-11-16T17:00:29.840 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.840 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.840 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.840 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.840 INFO:teuthology.orchestra.run.smithi141.stdout: "req_setdirlayout_latency": { 2023-11-16T17:00:29.841 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.841 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.841 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.841 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.841 INFO:teuthology.orchestra.run.smithi141.stdout: "req_setfilelock_latency": { 2023-11-16T17:00:29.841 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.841 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.842 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.842 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.842 INFO:teuthology.orchestra.run.smithi141.stdout: "req_setlayout_latency": { 2023-11-16T17:00:29.842 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.842 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.842 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.843 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.843 INFO:teuthology.orchestra.run.smithi141.stdout: "req_setxattr_latency": { 2023-11-16T17:00:29.843 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.843 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.843 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.843 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.843 INFO:teuthology.orchestra.run.smithi141.stdout: "req_symlink_latency": { 2023-11-16T17:00:29.844 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.844 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.844 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.844 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.844 INFO:teuthology.orchestra.run.smithi141.stdout: "req_unlink_latency": { 2023-11-16T17:00:29.844 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 16773, 2023-11-16T17:00:29.845 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 70.215510076, 2023-11-16T17:00:29.845 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.004186222 2023-11-16T17:00:29.845 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.845 INFO:teuthology.orchestra.run.smithi141.stdout: "cap_revoke_eviction": 0, 2023-11-16T17:00:29.845 INFO:teuthology.orchestra.run.smithi141.stdout: "cap_acquisition_throttle": 0, 2023-11-16T17:00:29.845 INFO:teuthology.orchestra.run.smithi141.stdout: "req_getvxattr_latency": { 2023-11-16T17:00:29.845 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.846 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.846 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.846 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:29.846 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.846 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_sessions": { 2023-11-16T17:00:29.846 INFO:teuthology.orchestra.run.smithi141.stdout: "session_count": 2, 2023-11-16T17:00:29.846 INFO:teuthology.orchestra.run.smithi141.stdout: "session_add": 5, 2023-11-16T17:00:29.847 INFO:teuthology.orchestra.run.smithi141.stdout: "session_remove": 3, 2023-11-16T17:00:29.847 INFO:teuthology.orchestra.run.smithi141.stdout: "sessions_open": 2, 2023-11-16T17:00:29.847 INFO:teuthology.orchestra.run.smithi141.stdout: "sessions_stale": 0, 2023-11-16T17:00:29.847 INFO:teuthology.orchestra.run.smithi141.stdout: "total_load": 20099, 2023-11-16T17:00:29.847 INFO:teuthology.orchestra.run.smithi141.stdout: "average_load": 10049, 2023-11-16T17:00:29.847 INFO:teuthology.orchestra.run.smithi141.stdout: "avg_session_uptime": 637 2023-11-16T17:00:29.847 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.848 INFO:teuthology.orchestra.run.smithi141.stdout: "mempool": { 2023-11-16T17:00:29.848 INFO:teuthology.orchestra.run.smithi141.stdout: "bloom_filter_bytes": 120, 2023-11-16T17:00:29.848 INFO:teuthology.orchestra.run.smithi141.stdout: "bloom_filter_items": 120, 2023-11-16T17:00:29.848 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:00:29.848 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:00:29.848 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:00:29.848 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:00:29.849 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:00:29.849 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:00:29.849 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:00:29.849 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:00:29.849 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:00:29.849 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:00:29.850 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:00:29.850 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:00:29.850 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:00:29.850 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_extent_items": 0, 2023-11-16T17:00:29.850 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:00:29.850 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_blob_items": 0, 2023-11-16T17:00:29.851 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:00:29.851 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:00:29.851 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:00:29.851 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:00:29.851 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:00:29.851 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:00:29.851 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:00:29.852 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_txc_items": 0, 2023-11-16T17:00:29.852 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:00:29.852 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:00:29.852 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:00:29.852 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_writing_items": 0, 2023-11-16T17:00:29.852 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_bytes": 0, 2023-11-16T17:00:29.853 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_items": 0, 2023-11-16T17:00:29.853 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:00:29.853 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:00:29.853 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:00:29.853 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:00:29.853 INFO:teuthology.orchestra.run.smithi141.stdout: "buffer_anon_bytes": 293697, 2023-11-16T17:00:29.853 INFO:teuthology.orchestra.run.smithi141.stdout: "buffer_anon_items": 147, 2023-11-16T17:00:29.854 INFO:teuthology.orchestra.run.smithi141.stdout: "buffer_meta_bytes": 1760, 2023-11-16T17:00:29.854 INFO:teuthology.orchestra.run.smithi141.stdout: "buffer_meta_items": 20, 2023-11-16T17:00:29.854 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_bytes": 0, 2023-11-16T17:00:29.854 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_items": 0, 2023-11-16T17:00:29.854 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:00:29.854 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_mapbl_items": 0, 2023-11-16T17:00:29.855 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:00:29.855 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_pglog_items": 0, 2023-11-16T17:00:29.855 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_bytes": 9944, 2023-11-16T17:00:29.855 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_items": 192, 2023-11-16T17:00:29.855 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:00:29.855 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:00:29.856 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap_bytes": 0, 2023-11-16T17:00:29.856 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap_items": 0, 2023-11-16T17:00:29.856 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_co_bytes": 147017, 2023-11-16T17:00:29.856 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_co_items": 719, 2023-11-16T17:00:29.856 INFO:teuthology.orchestra.run.smithi141.stdout: "unittest_1_bytes": 0, 2023-11-16T17:00:29.856 INFO:teuthology.orchestra.run.smithi141.stdout: "unittest_1_items": 0, 2023-11-16T17:00:29.856 INFO:teuthology.orchestra.run.smithi141.stdout: "unittest_2_bytes": 0, 2023-11-16T17:00:29.857 INFO:teuthology.orchestra.run.smithi141.stdout: "unittest_2_items": 0 2023-11-16T17:00:29.857 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.857 INFO:teuthology.orchestra.run.smithi141.stdout: "objecter": { 2023-11-16T17:00:29.857 INFO:teuthology.orchestra.run.smithi141.stdout: "op_active": 0, 2023-11-16T17:00:29.857 INFO:teuthology.orchestra.run.smithi141.stdout: "op_laggy": 0, 2023-11-16T17:00:29.857 INFO:teuthology.orchestra.run.smithi141.stdout: "op_send": 38440, 2023-11-16T17:00:29.857 INFO:teuthology.orchestra.run.smithi141.stdout: "op_send_bytes": 270540676, 2023-11-16T17:00:29.858 INFO:teuthology.orchestra.run.smithi141.stdout: "op_resend": 29, 2023-11-16T17:00:29.858 INFO:teuthology.orchestra.run.smithi141.stdout: "op_reply": 38440, 2023-11-16T17:00:29.858 INFO:teuthology.orchestra.run.smithi141.stdout: "op_latency": { 2023-11-16T17:00:29.858 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 38440, 2023-11-16T17:00:29.858 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 441.782588256, 2023-11-16T17:00:29.858 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.011492783 2023-11-16T17:00:29.859 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.859 INFO:teuthology.orchestra.run.smithi141.stdout: "op_inflight": 0, 2023-11-16T17:00:29.859 INFO:teuthology.orchestra.run.smithi141.stdout: "oplen_avg": { 2023-11-16T17:00:29.859 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 38440, 2023-11-16T17:00:29.859 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 39420 2023-11-16T17:00:29.859 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.859 INFO:teuthology.orchestra.run.smithi141.stdout: "op": 38440, 2023-11-16T17:00:29.860 INFO:teuthology.orchestra.run.smithi141.stdout: "op_r": 417, 2023-11-16T17:00:29.860 INFO:teuthology.orchestra.run.smithi141.stdout: "op_w": 38023, 2023-11-16T17:00:29.860 INFO:teuthology.orchestra.run.smithi141.stdout: "op_rmw": 0, 2023-11-16T17:00:29.860 INFO:teuthology.orchestra.run.smithi141.stdout: "op_pg": 0, 2023-11-16T17:00:29.860 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_stat": 0, 2023-11-16T17:00:29.860 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_create": 10, 2023-11-16T17:00:29.861 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_read": 264, 2023-11-16T17:00:29.861 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_write": 3400, 2023-11-16T17:00:29.861 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_writefull": 111, 2023-11-16T17:00:29.861 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_writesame": 0, 2023-11-16T17:00:29.861 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_append": 0, 2023-11-16T17:00:29.861 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_zero": 0, 2023-11-16T17:00:29.862 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_truncate": 0, 2023-11-16T17:00:29.862 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_delete": 33711, 2023-11-16T17:00:29.862 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_mapext": 0, 2023-11-16T17:00:29.862 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_sparse_read": 0, 2023-11-16T17:00:29.862 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_clonerange": 0, 2023-11-16T17:00:29.862 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_getxattr": 0, 2023-11-16T17:00:29.862 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_setxattr": 20, 2023-11-16T17:00:29.863 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_cmpxattr": 0, 2023-11-16T17:00:29.863 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_rmxattr": 0, 2023-11-16T17:00:29.863 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_resetxattrs": 0, 2023-11-16T17:00:29.863 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_call": 0, 2023-11-16T17:00:29.863 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_watch": 0, 2023-11-16T17:00:29.863 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_notify": 0, 2023-11-16T17:00:29.864 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_src_cmpxattr": 0, 2023-11-16T17:00:29.864 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_pgls": 0, 2023-11-16T17:00:29.864 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_pgls_filter": 0, 2023-11-16T17:00:29.864 INFO:teuthology.orchestra.run.smithi141.stdout: "osdop_other": 0, 2023-11-16T17:00:29.864 INFO:teuthology.orchestra.run.smithi141.stdout: "linger_active": 0, 2023-11-16T17:00:29.864 INFO:teuthology.orchestra.run.smithi141.stdout: "linger_send": 0, 2023-11-16T17:00:29.864 INFO:teuthology.orchestra.run.smithi141.stdout: "linger_resend": 0, 2023-11-16T17:00:29.865 INFO:teuthology.orchestra.run.smithi141.stdout: "linger_ping": 0, 2023-11-16T17:00:29.865 INFO:teuthology.orchestra.run.smithi141.stdout: "poolop_active": 0, 2023-11-16T17:00:29.865 INFO:teuthology.orchestra.run.smithi141.stdout: "poolop_send": 0, 2023-11-16T17:00:29.865 INFO:teuthology.orchestra.run.smithi141.stdout: "poolop_resend": 0, 2023-11-16T17:00:29.865 INFO:teuthology.orchestra.run.smithi141.stdout: "poolstat_active": 0, 2023-11-16T17:00:29.865 INFO:teuthology.orchestra.run.smithi141.stdout: "poolstat_send": 0, 2023-11-16T17:00:29.865 INFO:teuthology.orchestra.run.smithi141.stdout: "poolstat_resend": 0, 2023-11-16T17:00:29.866 INFO:teuthology.orchestra.run.smithi141.stdout: "statfs_active": 0, 2023-11-16T17:00:29.866 INFO:teuthology.orchestra.run.smithi141.stdout: "statfs_send": 0, 2023-11-16T17:00:29.866 INFO:teuthology.orchestra.run.smithi141.stdout: "statfs_resend": 0, 2023-11-16T17:00:29.866 INFO:teuthology.orchestra.run.smithi141.stdout: "command_active": 0, 2023-11-16T17:00:29.866 INFO:teuthology.orchestra.run.smithi141.stdout: "command_send": 0, 2023-11-16T17:00:29.866 INFO:teuthology.orchestra.run.smithi141.stdout: "command_resend": 0, 2023-11-16T17:00:29.867 INFO:teuthology.orchestra.run.smithi141.stdout: "map_epoch": 104, 2023-11-16T17:00:29.867 INFO:teuthology.orchestra.run.smithi141.stdout: "map_full": 0, 2023-11-16T17:00:29.867 INFO:teuthology.orchestra.run.smithi141.stdout: "map_inc": 13, 2023-11-16T17:00:29.867 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_sessions": 12, 2023-11-16T17:00:29.867 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_session_open": 12, 2023-11-16T17:00:29.867 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_session_close": 0, 2023-11-16T17:00:29.867 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_laggy": 0, 2023-11-16T17:00:29.867 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_wr": 1567, 2023-11-16T17:00:29.868 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_rd": 306, 2023-11-16T17:00:29.868 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_del": 31 2023-11-16T17:00:29.868 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.868 INFO:teuthology.orchestra.run.smithi141.stdout: "oft": { 2023-11-16T17:00:29.868 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_total_objs": 1, 2023-11-16T17:00:29.868 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_total_kv_pairs": 0, 2023-11-16T17:00:29.869 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_total_updates": 0, 2023-11-16T17:00:29.869 INFO:teuthology.orchestra.run.smithi141.stdout: "omap_total_removes": 0 2023-11-16T17:00:29.869 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.869 INFO:teuthology.orchestra.run.smithi141.stdout: "osd-slow-ops": { 2023-11-16T17:00:29.869 INFO:teuthology.orchestra.run.smithi141.stdout: "slow_ops_count": 51651 2023-11-16T17:00:29.869 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.869 INFO:teuthology.orchestra.run.smithi141.stdout: "purge_queue": { 2023-11-16T17:00:29.870 INFO:teuthology.orchestra.run.smithi141.stdout: "pq_executing_ops": 0, 2023-11-16T17:00:29.870 INFO:teuthology.orchestra.run.smithi141.stdout: "pq_executing_ops_high_water": 16, 2023-11-16T17:00:29.870 INFO:teuthology.orchestra.run.smithi141.stdout: "pq_executing": 0, 2023-11-16T17:00:29.870 INFO:teuthology.orchestra.run.smithi141.stdout: "pq_executing_high_water": 8, 2023-11-16T17:00:29.870 INFO:teuthology.orchestra.run.smithi141.stdout: "pq_executed": 16773, 2023-11-16T17:00:29.870 INFO:teuthology.orchestra.run.smithi141.stdout: "pq_item_in_journal": 0 2023-11-16T17:00:29.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.871 INFO:teuthology.orchestra.run.smithi141.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:00:29.871 INFO:teuthology.orchestra.run.smithi141.stdout: "val": 0, 2023-11-16T17:00:29.871 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 104857600, 2023-11-16T17:00:29.871 INFO:teuthology.orchestra.run.smithi141.stdout: "get_started": 0, 2023-11-16T17:00:29.871 INFO:teuthology.orchestra.run.smithi141.stdout: "get": 135459, 2023-11-16T17:00:29.871 INFO:teuthology.orchestra.run.smithi141.stdout: "get_sum": 123033534, 2023-11-16T17:00:29.871 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:29.872 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_success": 135459, 2023-11-16T17:00:29.872 INFO:teuthology.orchestra.run.smithi141.stdout: "take": 0, 2023-11-16T17:00:29.872 INFO:teuthology.orchestra.run.smithi141.stdout: "take_sum": 0, 2023-11-16T17:00:29.872 INFO:teuthology.orchestra.run.smithi141.stdout: "put": 135459, 2023-11-16T17:00:29.872 INFO:teuthology.orchestra.run.smithi141.stdout: "put_sum": 123033534, 2023-11-16T17:00:29.872 INFO:teuthology.orchestra.run.smithi141.stdout: "wait": { 2023-11-16T17:00:29.872 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.873 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.873 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.873 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:29.873 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.873 INFO:teuthology.orchestra.run.smithi141.stdout: "throttle-objecter_bytes": { 2023-11-16T17:00:29.873 INFO:teuthology.orchestra.run.smithi141.stdout: "val": 0, 2023-11-16T17:00:29.873 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 104857600, 2023-11-16T17:00:29.874 INFO:teuthology.orchestra.run.smithi141.stdout: "get_started": 0, 2023-11-16T17:00:29.874 INFO:teuthology.orchestra.run.smithi141.stdout: "get": 0, 2023-11-16T17:00:29.874 INFO:teuthology.orchestra.run.smithi141.stdout: "get_sum": 0, 2023-11-16T17:00:29.874 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:29.874 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_success": 0, 2023-11-16T17:00:29.874 INFO:teuthology.orchestra.run.smithi141.stdout: "take": 38440, 2023-11-16T17:00:29.875 INFO:teuthology.orchestra.run.smithi141.stdout: "take_sum": 272614677, 2023-11-16T17:00:29.875 INFO:teuthology.orchestra.run.smithi141.stdout: "put": 38440, 2023-11-16T17:00:29.875 INFO:teuthology.orchestra.run.smithi141.stdout: "put_sum": 272614677, 2023-11-16T17:00:29.875 INFO:teuthology.orchestra.run.smithi141.stdout: "wait": { 2023-11-16T17:00:29.875 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.875 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.875 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.876 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:29.876 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.876 INFO:teuthology.orchestra.run.smithi141.stdout: "throttle-objecter_ops": { 2023-11-16T17:00:29.876 INFO:teuthology.orchestra.run.smithi141.stdout: "val": 0, 2023-11-16T17:00:29.876 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 1024, 2023-11-16T17:00:29.876 INFO:teuthology.orchestra.run.smithi141.stdout: "get_started": 0, 2023-11-16T17:00:29.876 INFO:teuthology.orchestra.run.smithi141.stdout: "get": 0, 2023-11-16T17:00:29.877 INFO:teuthology.orchestra.run.smithi141.stdout: "get_sum": 0, 2023-11-16T17:00:29.877 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:29.877 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_success": 0, 2023-11-16T17:00:29.877 INFO:teuthology.orchestra.run.smithi141.stdout: "take": 38440, 2023-11-16T17:00:29.877 INFO:teuthology.orchestra.run.smithi141.stdout: "take_sum": 38440, 2023-11-16T17:00:29.877 INFO:teuthology.orchestra.run.smithi141.stdout: "put": 38440, 2023-11-16T17:00:29.877 INFO:teuthology.orchestra.run.smithi141.stdout: "put_sum": 38440, 2023-11-16T17:00:29.877 INFO:teuthology.orchestra.run.smithi141.stdout: "wait": { 2023-11-16T17:00:29.878 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.878 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.878 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.878 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:29.878 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.878 INFO:teuthology.orchestra.run.smithi141.stdout: "throttle-write_buf_throttle": { 2023-11-16T17:00:29.878 INFO:teuthology.orchestra.run.smithi141.stdout: "val": 0, 2023-11-16T17:00:29.879 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 3758096384, 2023-11-16T17:00:29.879 INFO:teuthology.orchestra.run.smithi141.stdout: "get_started": 0, 2023-11-16T17:00:29.879 INFO:teuthology.orchestra.run.smithi141.stdout: "get": 16773, 2023-11-16T17:00:29.879 INFO:teuthology.orchestra.run.smithi141.stdout: "get_sum": 2076449, 2023-11-16T17:00:29.879 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:29.879 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_success": 16773, 2023-11-16T17:00:29.879 INFO:teuthology.orchestra.run.smithi141.stdout: "take": 0, 2023-11-16T17:00:29.879 INFO:teuthology.orchestra.run.smithi141.stdout: "take_sum": 0, 2023-11-16T17:00:29.880 INFO:teuthology.orchestra.run.smithi141.stdout: "put": 264, 2023-11-16T17:00:29.880 INFO:teuthology.orchestra.run.smithi141.stdout: "put_sum": 2076449, 2023-11-16T17:00:29.880 INFO:teuthology.orchestra.run.smithi141.stdout: "wait": { 2023-11-16T17:00:29.880 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.880 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.880 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.880 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:29.881 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:29.881 INFO:teuthology.orchestra.run.smithi141.stdout: "throttle-write_buf_throttle-0x55c47bbf00a0": { 2023-11-16T17:00:29.881 INFO:teuthology.orchestra.run.smithi141.stdout: "val": 0, 2023-11-16T17:00:29.881 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 3758096384, 2023-11-16T17:00:29.881 INFO:teuthology.orchestra.run.smithi141.stdout: "get_started": 0, 2023-11-16T17:00:29.881 INFO:teuthology.orchestra.run.smithi141.stdout: "get": 92995, 2023-11-16T17:00:29.881 INFO:teuthology.orchestra.run.smithi141.stdout: "get_sum": 259756516, 2023-11-16T17:00:29.881 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:29.882 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_success": 92995, 2023-11-16T17:00:29.882 INFO:teuthology.orchestra.run.smithi141.stdout: "take": 0, 2023-11-16T17:00:29.882 INFO:teuthology.orchestra.run.smithi141.stdout: "take_sum": 0, 2023-11-16T17:00:29.882 INFO:teuthology.orchestra.run.smithi141.stdout: "put": 3136, 2023-11-16T17:00:29.882 INFO:teuthology.orchestra.run.smithi141.stdout: "put_sum": 259756516, 2023-11-16T17:00:29.882 INFO:teuthology.orchestra.run.smithi141.stdout: "wait": { 2023-11-16T17:00:29.882 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:29.883 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:29.883 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:29.883 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:29.883 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:29.883 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-11-16T17:00:29.884 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:29 smithi141 ceph-mon[131184]: Adjusting osd_memory_target on smithi155 to 196.2M 2023-11-16T17:00:29.884 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:29 smithi141 ceph-mon[131184]: Unable to set osd_memory_target on smithi155 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-11-16T17:00:29.884 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:29 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:29.884 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:29 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:29.884 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:29 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:29.884 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:29 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:29.885 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:29 smithi141 ceph-mon[131184]: Adjusting osd_memory_target on smithi141 to 196.2M 2023-11-16T17:00:29.885 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:29 smithi141 ceph-mon[131184]: Unable to set osd_memory_target on smithi141 to 205752422: error parsing value: Value '205752422' is below minimum 939524096 2023-11-16T17:00:30.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:29 smithi155 ceph-mon[131607]: Adjusting osd_memory_target on smithi155 to 196.2M 2023-11-16T17:00:30.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:29 smithi155 ceph-mon[131607]: Unable to set osd_memory_target on smithi155 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-11-16T17:00:30.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:29 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:30.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:29 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:30.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:29 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:30.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:29 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:30.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:29 smithi155 ceph-mon[131607]: Adjusting osd_memory_target on smithi141 to 196.2M 2023-11-16T17:00:30.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:29 smithi155 ceph-mon[131607]: Unable to set osd_memory_target on smithi141 to 205752422: error parsing value: Value '205752422' is below minimum 939524096 2023-11-16T17:00:30.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:29 smithi002 ceph-mon[128712]: Adjusting osd_memory_target on smithi155 to 196.2M 2023-11-16T17:00:30.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:29 smithi002 ceph-mon[128712]: Unable to set osd_memory_target on smithi155 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-11-16T17:00:30.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:29 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:30.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:29 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:30.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:29 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:30.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:29 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:30.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:29 smithi002 ceph-mon[128712]: Adjusting osd_memory_target on smithi141 to 196.2M 2023-11-16T17:00:30.156 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:29 smithi002 ceph-mon[128712]: Unable to set osd_memory_target on smithi141 to 205752422: error parsing value: Value '205752422' is below minimum 939524096 2023-11-16T17:00:30.197 INFO:tasks.check_counter:Daemon mds.h mds.exported=1339 2023-11-16T17:00:30.197 INFO:tasks.check_counter:Daemon mds.h mds.imported=696 2023-11-16T17:00:30.197 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=912 2023-11-16T17:00:30.197 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=3419 2023-11-16T17:00:30.197 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=0 2023-11-16T17:00:30.197 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2023-11-16T17:00:30.198 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2023-11-16T17:00:30.198 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=92 2023-11-16T17:00:30.198 DEBUG:tasks.check_counter:Getting stats from k 2023-11-16T17:00:30.198 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.k perf dump 2023-11-16T17:00:30.965 INFO:teuthology.orchestra.run.smithi141.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.b/config 2023-11-16T17:00:31.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:30 smithi141 ceph-mon[131184]: pgmap v330: 129 pgs: 20 active+clean+snaptrim, 61 active+clean+snaptrim_wait, 48 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-11-16T17:00:31.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:30 smithi155 ceph-mon[131607]: pgmap v330: 129 pgs: 20 active+clean+snaptrim, 61 active+clean+snaptrim_wait, 48 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-11-16T17:00:31.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:30 smithi002 ceph-mon[128712]: pgmap v330: 129 pgs: 20 active+clean+snaptrim, 61 active+clean+snaptrim_wait, 48 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-11-16T17:00:32.237 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-11-16T17:00:32.237 INFO:teuthology.orchestra.run.smithi141.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:00:32.237 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_messages": 234, 2023-11-16T17:00:32.237 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages": 222, 2023-11-16T17:00:32.238 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_bytes": 1426944, 2023-11-16T17:00:32.238 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_bytes": 125680, 2023-11-16T17:00:32.238 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_created_connections": 4, 2023-11-16T17:00:32.238 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_active_connections": 0, 2023-11-16T17:00:32.238 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_total_time": 0.045190354, 2023-11-16T17:00:32.239 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_send_time": 0.013857747, 2023-11-16T17:00:32.239 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_recv_time": 0.032191407, 2023-11-16T17:00:32.239 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_fast_dispatch_time": 0.006191908, 2023-11-16T17:00:32.239 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:32.239 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 222, 2023-11-16T17:00:32.240 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.016968413, 2023-11-16T17:00:32.240 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000076434 2023-11-16T17:00:32.240 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:32.240 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:32.240 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:32.241 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:32.241 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:32.241 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:32.241 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_encrypted_bytes": 1426944, 2023-11-16T17:00:32.241 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_encrypted_bytes": 125680 2023-11-16T17:00:32.241 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:32.242 INFO:teuthology.orchestra.run.smithi141.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:00:32.242 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_messages": 2, 2023-11-16T17:00:32.242 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages": 168, 2023-11-16T17:00:32.242 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_bytes": 320, 2023-11-16T17:00:32.242 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_bytes": 181296, 2023-11-16T17:00:32.243 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_created_connections": 5, 2023-11-16T17:00:32.243 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_active_connections": 3, 2023-11-16T17:00:32.243 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_total_time": 0.010466857, 2023-11-16T17:00:32.243 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_send_time": 0.006735472, 2023-11-16T17:00:32.243 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_recv_time": 0.001471406, 2023-11-16T17:00:32.243 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:32.244 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:32.244 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 165, 2023-11-16T17:00:32.244 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.018362844, 2023-11-16T17:00:32.244 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000111289 2023-11-16T17:00:32.244 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:32.244 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:32.245 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:32.245 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:32.245 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:32.245 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:32.245 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_encrypted_bytes": 320, 2023-11-16T17:00:32.246 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_encrypted_bytes": 181296 2023-11-16T17:00:32.246 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:32.246 INFO:teuthology.orchestra.run.smithi141.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:00:32.246 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_messages": 0, 2023-11-16T17:00:32.246 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages": 2, 2023-11-16T17:00:32.246 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_bytes": 0, 2023-11-16T17:00:32.247 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_bytes": 0, 2023-11-16T17:00:32.247 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_created_connections": 2, 2023-11-16T17:00:32.247 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_active_connections": 1, 2023-11-16T17:00:32.247 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_total_time": 0.000326819, 2023-11-16T17:00:32.247 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_send_time": 0.000000000, 2023-11-16T17:00:32.248 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_recv_time": 0.000265241, 2023-11-16T17:00:32.248 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:32.248 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:32.248 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:32.248 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:32.248 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:32.249 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:32.249 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:32.249 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:32.249 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:32.249 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:32.250 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:32.250 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T17:00:32.250 INFO:teuthology.orchestra.run.smithi141.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T17:00:32.250 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:32.250 INFO:teuthology.orchestra.run.smithi141.stdout: "cct": { 2023-11-16T17:00:32.250 INFO:teuthology.orchestra.run.smithi141.stdout: "total_workers": 0, 2023-11-16T17:00:32.251 INFO:teuthology.orchestra.run.smithi141.stdout: "unhealthy_workers": 0 2023-11-16T17:00:32.251 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:32.251 INFO:teuthology.orchestra.run.smithi141.stdout: "mempool": { 2023-11-16T17:00:32.251 INFO:teuthology.orchestra.run.smithi141.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:00:32.251 INFO:teuthology.orchestra.run.smithi141.stdout: "bloom_filter_items": 0, 2023-11-16T17:00:32.251 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:00:32.252 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:00:32.252 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:00:32.252 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:00:32.252 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:00:32.252 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:00:32.252 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:00:32.253 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:00:32.253 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:00:32.253 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:00:32.253 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:00:32.253 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:00:32.253 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:00:32.254 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_extent_items": 0, 2023-11-16T17:00:32.254 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:00:32.254 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_blob_items": 0, 2023-11-16T17:00:32.254 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:00:32.254 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:00:32.255 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:00:32.255 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:00:32.255 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:00:32.255 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:00:32.255 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:00:32.255 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_txc_items": 0, 2023-11-16T17:00:32.256 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:00:32.256 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:00:32.256 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:00:32.256 INFO:teuthology.orchestra.run.smithi141.stdout: "bluestore_writing_items": 0, 2023-11-16T17:00:32.256 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_bytes": 0, 2023-11-16T17:00:32.256 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_items": 0, 2023-11-16T17:00:32.257 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:00:32.257 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:00:32.257 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:00:32.257 INFO:teuthology.orchestra.run.smithi141.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:00:32.257 INFO:teuthology.orchestra.run.smithi141.stdout: "buffer_anon_bytes": 148731, 2023-11-16T17:00:32.257 INFO:teuthology.orchestra.run.smithi141.stdout: "buffer_anon_items": 31, 2023-11-16T17:00:32.258 INFO:teuthology.orchestra.run.smithi141.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:00:32.258 INFO:teuthology.orchestra.run.smithi141.stdout: "buffer_meta_items": 0, 2023-11-16T17:00:32.258 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_bytes": 0, 2023-11-16T17:00:32.258 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_items": 0, 2023-11-16T17:00:32.258 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:00:32.258 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_mapbl_items": 0, 2023-11-16T17:00:32.259 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:00:32.259 INFO:teuthology.orchestra.run.smithi141.stdout: "osd_pglog_items": 0, 2023-11-16T17:00:32.259 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_bytes": 0, 2023-11-16T17:00:32.259 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_items": 0, 2023-11-16T17:00:32.259 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:00:32.260 INFO:teuthology.orchestra.run.smithi141.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:00:32.260 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap_bytes": 0, 2023-11-16T17:00:32.260 INFO:teuthology.orchestra.run.smithi141.stdout: "pgmap_items": 0, 2023-11-16T17:00:32.260 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_co_bytes": 928, 2023-11-16T17:00:32.260 INFO:teuthology.orchestra.run.smithi141.stdout: "mds_co_items": 2, 2023-11-16T17:00:32.260 INFO:teuthology.orchestra.run.smithi141.stdout: "unittest_1_bytes": 0, 2023-11-16T17:00:32.261 INFO:teuthology.orchestra.run.smithi141.stdout: "unittest_1_items": 0, 2023-11-16T17:00:32.261 INFO:teuthology.orchestra.run.smithi141.stdout: "unittest_2_bytes": 0, 2023-11-16T17:00:32.261 INFO:teuthology.orchestra.run.smithi141.stdout: "unittest_2_items": 0 2023-11-16T17:00:32.261 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-11-16T17:00:32.261 INFO:teuthology.orchestra.run.smithi141.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:00:32.261 INFO:teuthology.orchestra.run.smithi141.stdout: "val": 0, 2023-11-16T17:00:32.262 INFO:teuthology.orchestra.run.smithi141.stdout: "max": 104857600, 2023-11-16T17:00:32.262 INFO:teuthology.orchestra.run.smithi141.stdout: "get_started": 0, 2023-11-16T17:00:32.262 INFO:teuthology.orchestra.run.smithi141.stdout: "get": 232, 2023-11-16T17:00:32.262 INFO:teuthology.orchestra.run.smithi141.stdout: "get_sum": 1391473, 2023-11-16T17:00:32.262 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:32.262 INFO:teuthology.orchestra.run.smithi141.stdout: "get_or_fail_success": 232, 2023-11-16T17:00:32.263 INFO:teuthology.orchestra.run.smithi141.stdout: "take": 0, 2023-11-16T17:00:32.263 INFO:teuthology.orchestra.run.smithi141.stdout: "take_sum": 0, 2023-11-16T17:00:32.263 INFO:teuthology.orchestra.run.smithi141.stdout: "put": 232, 2023-11-16T17:00:32.263 INFO:teuthology.orchestra.run.smithi141.stdout: "put_sum": 1391473, 2023-11-16T17:00:32.263 INFO:teuthology.orchestra.run.smithi141.stdout: "wait": { 2023-11-16T17:00:32.263 INFO:teuthology.orchestra.run.smithi141.stdout: "avgcount": 0, 2023-11-16T17:00:32.264 INFO:teuthology.orchestra.run.smithi141.stdout: "sum": 0.000000000, 2023-11-16T17:00:32.264 INFO:teuthology.orchestra.run.smithi141.stdout: "avgtime": 0.000000000 2023-11-16T17:00:32.264 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:32.264 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-11-16T17:00:32.264 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-11-16T17:00:32.543 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2023-11-16T17:00:32.544 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2023-11-16T17:00:32.544 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2023-11-16T17:00:32.544 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2023-11-16T17:00:32.544 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2023-11-16T17:00:32.544 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2023-11-16T17:00:32.544 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2023-11-16T17:00:32.544 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2023-11-16T17:00:32.544 DEBUG:tasks.check_counter:Getting stats from c 2023-11-16T17:00:32.545 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.c perf dump 2023-11-16T17:00:32.867 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:32 smithi155 ceph-mon[131607]: pgmap v331: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 607 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2023-11-16T17:00:33.014 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:32 smithi141 ceph-mon[131184]: pgmap v331: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 607 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2023-11-16T17:00:33.155 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:32 smithi002 ceph-mon[128712]: pgmap v331: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 607 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2023-11-16T17:00:33.306 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T17:00:34.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:34 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:34.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:34 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:34.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:34 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:34.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:34 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:34.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:34 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:34.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:34 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:34.550 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2023-11-16T17:00:34.550 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:00:34.550 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 953, 2023-11-16T17:00:34.550 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 955, 2023-11-16T17:00:34.551 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 105064015, 2023-11-16T17:00:34.551 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 296393, 2023-11-16T17:00:34.551 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 8, 2023-11-16T17:00:34.551 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 4, 2023-11-16T17:00:34.552 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.144619023, 2023-11-16T17:00:34.552 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.024993733, 2023-11-16T17:00:34.552 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.077807338, 2023-11-16T17:00:34.552 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.032359552, 2023-11-16T17:00:34.552 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:34.553 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 953, 2023-11-16T17:00:34.553 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.057383181, 2023-11-16T17:00:34.553 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000060213 2023-11-16T17:00:34.553 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.553 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:34.554 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.554 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.554 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.554 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.554 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T17:00:34.555 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T17:00:34.555 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.555 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:00:34.555 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 806, 2023-11-16T17:00:34.555 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 791, 2023-11-16T17:00:34.556 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 88152500, 2023-11-16T17:00:34.556 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 282477, 2023-11-16T17:00:34.556 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 6, 2023-11-16T17:00:34.556 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 5, 2023-11-16T17:00:34.556 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.112548154, 2023-11-16T17:00:34.557 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.028387742, 2023-11-16T17:00:34.557 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.061028207, 2023-11-16T17:00:34.557 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.020367036, 2023-11-16T17:00:34.557 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:34.557 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 791, 2023-11-16T17:00:34.558 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.051951405, 2023-11-16T17:00:34.558 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000065678 2023-11-16T17:00:34.558 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.558 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:34.558 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.559 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.559 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.559 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.559 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 390736, 2023-11-16T17:00:34.560 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 89280 2023-11-16T17:00:34.560 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.560 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:00:34.560 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 663, 2023-11-16T17:00:34.560 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 777, 2023-11-16T17:00:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 84915976, 2023-11-16T17:00:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 484523, 2023-11-16T17:00:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 6, 2023-11-16T17:00:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 5, 2023-11-16T17:00:34.561 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.100507251, 2023-11-16T17:00:34.562 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.023294459, 2023-11-16T17:00:34.562 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.049553196, 2023-11-16T17:00:34.562 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.021413557, 2023-11-16T17:00:34.562 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:34.562 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 775, 2023-11-16T17:00:34.563 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.056306223, 2023-11-16T17:00:34.563 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000072653 2023-11-16T17:00:34.563 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.563 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:34.563 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.564 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.564 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.564 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.564 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 160, 2023-11-16T17:00:34.564 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 278640 2023-11-16T17:00:34.564 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.565 INFO:teuthology.orchestra.run.smithi155.stdout: "cct": { 2023-11-16T17:00:34.565 INFO:teuthology.orchestra.run.smithi155.stdout: "total_workers": 1, 2023-11-16T17:00:34.565 INFO:teuthology.orchestra.run.smithi155.stdout: "unhealthy_workers": 0 2023-11-16T17:00:34.565 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.565 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-MDSRank": { 2023-11-16T17:00:34.566 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2023-11-16T17:00:34.566 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2023-11-16T17:00:34.566 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 3444, 2023-11-16T17:00:34.566 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.799344161, 2023-11-16T17:00:34.566 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000232097 2023-11-16T17:00:34.567 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:34.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.567 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-PurgeQueue": { 2023-11-16T17:00:34.567 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2023-11-16T17:00:34.567 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2023-11-16T17:00:34.568 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.568 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.568 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.568 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:34.568 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.569 INFO:teuthology.orchestra.run.smithi155.stdout: "mds": { 2023-11-16T17:00:34.569 INFO:teuthology.orchestra.run.smithi155.stdout: "request": 0, 2023-11-16T17:00:34.569 INFO:teuthology.orchestra.run.smithi155.stdout: "reply": 0, 2023-11-16T17:00:34.569 INFO:teuthology.orchestra.run.smithi155.stdout: "reply_latency": { 2023-11-16T17:00:34.569 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.569 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.570 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.570 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.570 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_reply": 0, 2023-11-16T17:00:34.570 INFO:teuthology.orchestra.run.smithi155.stdout: "forward": 0, 2023-11-16T17:00:34.571 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_complete": 0, 2023-11-16T17:00:34.571 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_keys": 0, 2023-11-16T17:00:34.571 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_commit": 0, 2023-11-16T17:00:34.571 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_split": 0, 2023-11-16T17:00:34.571 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_merge": 0, 2023-11-16T17:00:34.572 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes": 66164, 2023-11-16T17:00:34.572 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_top": 46305, 2023-11-16T17:00:34.572 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_bottom": 19858, 2023-11-16T17:00:34.572 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pin_tail": 1, 2023-11-16T17:00:34.572 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pinned": 13, 2023-11-16T17:00:34.573 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_expired": 243, 2023-11-16T17:00:34.573 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_with_caps": 0, 2023-11-16T17:00:34.573 INFO:teuthology.orchestra.run.smithi155.stdout: "caps": 0, 2023-11-16T17:00:34.573 INFO:teuthology.orchestra.run.smithi155.stdout: "subtrees": 3, 2023-11-16T17:00:34.573 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse": 0, 2023-11-16T17:00:34.573 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_hit": 0, 2023-11-16T17:00:34.574 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_forward": 0, 2023-11-16T17:00:34.574 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_discover": 0, 2023-11-16T17:00:34.574 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_dir_fetch": 0, 2023-11-16T17:00:34.574 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_remote_ino": 0, 2023-11-16T17:00:34.574 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_lock": 0, 2023-11-16T17:00:34.575 INFO:teuthology.orchestra.run.smithi155.stdout: "load_cent": 0, 2023-11-16T17:00:34.575 INFO:teuthology.orchestra.run.smithi155.stdout: "q": 0, 2023-11-16T17:00:34.575 INFO:teuthology.orchestra.run.smithi155.stdout: "exported": 0, 2023-11-16T17:00:34.575 INFO:teuthology.orchestra.run.smithi155.stdout: "exported_inodes": 0, 2023-11-16T17:00:34.575 INFO:teuthology.orchestra.run.smithi155.stdout: "imported": 0, 2023-11-16T17:00:34.576 INFO:teuthology.orchestra.run.smithi155.stdout: "imported_inodes": 0, 2023-11-16T17:00:34.576 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_dir_fetch": 0, 2023-11-16T17:00:34.576 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_backtrace_fetch": 0, 2023-11-16T17:00:34.576 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_peer_discover": 0, 2023-11-16T17:00:34.576 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rfiles": 4, 2023-11-16T17:00:34.576 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rbytes": 246, 2023-11-16T17:00:34.577 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rsnaps": 6, 2023-11-16T17:00:34.577 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_fetch": 0, 2023-11-16T17:00:34.577 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_set_tag": 0, 2023-11-16T17:00:34.577 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_repaired": 0, 2023-11-16T17:00:34.577 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_inotable_repaired": 0, 2023-11-16T17:00:34.578 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_inodes": 0, 2023-11-16T17:00:34.578 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_base_inodes": 0, 2023-11-16T17:00:34.578 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dirfrag_rstats": 0, 2023-11-16T17:00:34.578 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_file_inodes": 0, 2023-11-16T17:00:34.578 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_inode_file_caps": 0, 2023-11-16T17:00:34.579 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_revoke": 0, 2023-11-16T17:00:34.579 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_grant": 0, 2023-11-16T17:00:34.579 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_trunc": 0, 2023-11-16T17:00:34.579 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-11-16T17:00:34.579 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-16T17:00:34.579 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps": 0, 2023-11-16T17:00:34.580 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps_dirty": 0, 2023-11-16T17:00:34.580 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_cap_release": 0, 2023-11-16T17:00:34.580 INFO:teuthology.orchestra.run.smithi155.stdout: "process_request_cap_release": 0 2023-11-16T17:00:34.580 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.580 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_cache": { 2023-11-16T17:00:34.581 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update": 0, 2023-11-16T17:00:34.581 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update_receipt": 0, 2023-11-16T17:00:34.581 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_try_discover": 0, 2023-11-16T17:00:34.581 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_send_discover": 0, 2023-11-16T17:00:34.581 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_handle_discover": 0, 2023-11-16T17:00:34.582 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays": 0, 2023-11-16T17:00:34.582 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_delayed": 0, 2023-11-16T17:00:34.582 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_enqueuing": 0, 2023-11-16T17:00:34.582 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_created": 0, 2023-11-16T17:00:34.582 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_enqueued": 0, 2023-11-16T17:00:34.583 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_reintegrated": 0, 2023-11-16T17:00:34.583 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_migrated": 0, 2023-11-16T17:00:34.583 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_processing": 0, 2023-11-16T17:00:34.583 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_enqueued": 0, 2023-11-16T17:00:34.583 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_prioritized": 0, 2023-11-16T17:00:34.583 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_started": 0, 2023-11-16T17:00:34.584 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_completed": 0, 2023-11-16T17:00:34.584 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_enqueue_scrub": 0, 2023-11-16T17:00:34.584 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_exportdir": 0, 2023-11-16T17:00:34.584 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_flush": 0, 2023-11-16T17:00:34.584 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragmentdir": 0, 2023-11-16T17:00:34.585 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragstats": 0, 2023-11-16T17:00:34.585 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_inodestats": 0 2023-11-16T17:00:34.585 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.585 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_log": { 2023-11-16T17:00:34.585 INFO:teuthology.orchestra.run.smithi155.stdout: "evadd": 0, 2023-11-16T17:00:34.585 INFO:teuthology.orchestra.run.smithi155.stdout: "evex": 0, 2023-11-16T17:00:34.586 INFO:teuthology.orchestra.run.smithi155.stdout: "evtrm": 0, 2023-11-16T17:00:34.586 INFO:teuthology.orchestra.run.smithi155.stdout: "ev": 0, 2023-11-16T17:00:34.586 INFO:teuthology.orchestra.run.smithi155.stdout: "evexg": 0, 2023-11-16T17:00:34.586 INFO:teuthology.orchestra.run.smithi155.stdout: "evexd": 0, 2023-11-16T17:00:34.586 INFO:teuthology.orchestra.run.smithi155.stdout: "segadd": 0, 2023-11-16T17:00:34.586 INFO:teuthology.orchestra.run.smithi155.stdout: "segex": 0, 2023-11-16T17:00:34.587 INFO:teuthology.orchestra.run.smithi155.stdout: "segtrm": 0, 2023-11-16T17:00:34.587 INFO:teuthology.orchestra.run.smithi155.stdout: "seg": 2, 2023-11-16T17:00:34.587 INFO:teuthology.orchestra.run.smithi155.stdout: "segexg": 0, 2023-11-16T17:00:34.587 INFO:teuthology.orchestra.run.smithi155.stdout: "segexd": 0, 2023-11-16T17:00:34.587 INFO:teuthology.orchestra.run.smithi155.stdout: "expos": 280457945, 2023-11-16T17:00:34.588 INFO:teuthology.orchestra.run.smithi155.stdout: "wrpos": 0, 2023-11-16T17:00:34.588 INFO:teuthology.orchestra.run.smithi155.stdout: "rdpos": 280516679, 2023-11-16T17:00:34.588 INFO:teuthology.orchestra.run.smithi155.stdout: "jlat": { 2023-11-16T17:00:34.588 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.588 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.588 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.589 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.589 INFO:teuthology.orchestra.run.smithi155.stdout: "replayed": 84033 2023-11-16T17:00:34.589 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.589 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_mem": { 2023-11-16T17:00:34.589 INFO:teuthology.orchestra.run.smithi155.stdout: "ino": 30919, 2023-11-16T17:00:34.589 INFO:teuthology.orchestra.run.smithi155.stdout: "ino+": 54279, 2023-11-16T17:00:34.590 INFO:teuthology.orchestra.run.smithi155.stdout: "ino-": 23360, 2023-11-16T17:00:34.590 INFO:teuthology.orchestra.run.smithi155.stdout: "dir": 1039, 2023-11-16T17:00:34.590 INFO:teuthology.orchestra.run.smithi155.stdout: "dir+": 1301, 2023-11-16T17:00:34.590 INFO:teuthology.orchestra.run.smithi155.stdout: "dir-": 262, 2023-11-16T17:00:34.590 INFO:teuthology.orchestra.run.smithi155.stdout: "dn": 66164, 2023-11-16T17:00:34.591 INFO:teuthology.orchestra.run.smithi155.stdout: "dn+": 75752, 2023-11-16T17:00:34.591 INFO:teuthology.orchestra.run.smithi155.stdout: "dn-": 9588, 2023-11-16T17:00:34.591 INFO:teuthology.orchestra.run.smithi155.stdout: "cap": 0, 2023-11-16T17:00:34.591 INFO:teuthology.orchestra.run.smithi155.stdout: "cap+": 0, 2023-11-16T17:00:34.591 INFO:teuthology.orchestra.run.smithi155.stdout: "cap-": 0, 2023-11-16T17:00:34.591 INFO:teuthology.orchestra.run.smithi155.stdout: "rss": 195368, 2023-11-16T17:00:34.592 INFO:teuthology.orchestra.run.smithi155.stdout: "heap": 207156 2023-11-16T17:00:34.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.592 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_server": { 2023-11-16T17:00:34.592 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_client_request": 0, 2023-11-16T17:00:34.592 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_server_request": 0, 2023-11-16T17:00:34.592 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_request": 0, 2023-11-16T17:00:34.593 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_session": 0, 2023-11-16T17:00:34.593 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_peer_request": 0, 2023-11-16T17:00:34.593 INFO:teuthology.orchestra.run.smithi155.stdout: "req_create_latency": { 2023-11-16T17:00:34.593 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.593 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.593 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.594 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.594 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getattr_latency": { 2023-11-16T17:00:34.594 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.594 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.594 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.595 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getfilelock_latency": { 2023-11-16T17:00:34.595 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.595 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.595 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.596 INFO:teuthology.orchestra.run.smithi155.stdout: "req_link_latency": { 2023-11-16T17:00:34.596 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.596 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.596 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.596 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.596 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookup_latency": { 2023-11-16T17:00:34.597 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.597 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.597 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.597 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookuphash_latency": { 2023-11-16T17:00:34.597 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.598 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.598 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.598 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupino_latency": { 2023-11-16T17:00:34.598 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.598 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.599 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.599 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupname_latency": { 2023-11-16T17:00:34.599 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.599 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.600 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.600 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupparent_latency": { 2023-11-16T17:00:34.600 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.600 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.600 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.601 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupsnap_latency": { 2023-11-16T17:00:34.601 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.601 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.601 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.601 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lssnap_latency": { 2023-11-16T17:00:34.602 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.602 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.602 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.602 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mkdir_latency": { 2023-11-16T17:00:34.603 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.603 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.603 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.603 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mknod_latency": { 2023-11-16T17:00:34.603 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.604 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.604 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.604 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mksnap_latency": { 2023-11-16T17:00:34.604 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.604 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.605 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.605 INFO:teuthology.orchestra.run.smithi155.stdout: "req_open_latency": { 2023-11-16T17:00:34.605 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.605 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.605 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.606 INFO:teuthology.orchestra.run.smithi155.stdout: "req_readdir_latency": { 2023-11-16T17:00:34.606 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.606 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.606 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.607 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rename_latency": { 2023-11-16T17:00:34.607 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.607 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.607 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.608 INFO:teuthology.orchestra.run.smithi155.stdout: "req_renamesnap_latency": { 2023-11-16T17:00:34.608 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.608 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.608 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.608 INFO:teuthology.orchestra.run.smithi155.stdout: "req_snapdiff_latency": { 2023-11-16T17:00:34.609 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.609 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.609 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.609 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmdir_latency": { 2023-11-16T17:00:34.609 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.610 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.610 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.610 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmsnap_latency": { 2023-11-16T17:00:34.610 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.610 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.611 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.611 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmxattr_latency": { 2023-11-16T17:00:34.611 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.611 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.611 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.612 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setattr_latency": { 2023-11-16T17:00:34.612 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.612 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.612 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.613 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setdirlayout_latency": { 2023-11-16T17:00:34.613 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.613 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.613 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.613 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setfilelock_latency": { 2023-11-16T17:00:34.613 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.614 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.614 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.614 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.614 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setlayout_latency": { 2023-11-16T17:00:34.614 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.614 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.615 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.615 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setxattr_latency": { 2023-11-16T17:00:34.615 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.615 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.615 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.616 INFO:teuthology.orchestra.run.smithi155.stdout: "req_symlink_latency": { 2023-11-16T17:00:34.616 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.616 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.616 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.617 INFO:teuthology.orchestra.run.smithi155.stdout: "req_unlink_latency": { 2023-11-16T17:00:34.617 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.617 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.617 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.617 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_revoke_eviction": 0, 2023-11-16T17:00:34.618 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_acquisition_throttle": 0, 2023-11-16T17:00:34.618 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getvxattr_latency": { 2023-11-16T17:00:34.618 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.618 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.618 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.618 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:34.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.619 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_sessions": { 2023-11-16T17:00:34.619 INFO:teuthology.orchestra.run.smithi155.stdout: "session_count": 2, 2023-11-16T17:00:34.619 INFO:teuthology.orchestra.run.smithi155.stdout: "session_add": 4, 2023-11-16T17:00:34.619 INFO:teuthology.orchestra.run.smithi155.stdout: "session_remove": 2, 2023-11-16T17:00:34.619 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_open": 0, 2023-11-16T17:00:34.620 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_stale": 0, 2023-11-16T17:00:34.620 INFO:teuthology.orchestra.run.smithi155.stdout: "total_load": 0, 2023-11-16T17:00:34.620 INFO:teuthology.orchestra.run.smithi155.stdout: "average_load": 0, 2023-11-16T17:00:34.620 INFO:teuthology.orchestra.run.smithi155.stdout: "avg_session_uptime": 4820 2023-11-16T17:00:34.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.620 INFO:teuthology.orchestra.run.smithi155.stdout: "mempool": { 2023-11-16T17:00:34.620 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:00:34.621 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_items": 0, 2023-11-16T17:00:34.621 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:00:34.621 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:00:34.621 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:00:34.621 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:00:34.621 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:00:34.622 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:00:34.622 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:00:34.622 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:00:34.622 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:00:34.622 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:00:34.622 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:00:34.623 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:00:34.623 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:00:34.623 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_items": 0, 2023-11-16T17:00:34.623 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:00:34.623 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_items": 0, 2023-11-16T17:00:34.623 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:00:34.624 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:00:34.624 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:00:34.624 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:00:34.624 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:00:34.624 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:00:34.624 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:00:34.624 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_items": 0, 2023-11-16T17:00:34.625 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:00:34.625 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:00:34.625 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:00:34.625 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_items": 0, 2023-11-16T17:00:34.625 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_bytes": 0, 2023-11-16T17:00:34.625 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_items": 0, 2023-11-16T17:00:34.625 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:00:34.626 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:00:34.626 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:00:34.626 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:00:34.626 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_bytes": 1288894, 2023-11-16T17:00:34.626 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_items": 30987, 2023-11-16T17:00:34.626 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:00:34.627 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_items": 0, 2023-11-16T17:00:34.627 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_bytes": 0, 2023-11-16T17:00:34.627 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_items": 0, 2023-11-16T17:00:34.627 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:00:34.627 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_items": 0, 2023-11-16T17:00:34.627 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:00:34.628 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_items": 0, 2023-11-16T17:00:34.628 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_bytes": 9944, 2023-11-16T17:00:34.628 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_items": 192, 2023-11-16T17:00:34.628 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:00:34.628 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:00:34.628 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_bytes": 0, 2023-11-16T17:00:34.628 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_items": 0, 2023-11-16T17:00:34.629 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_bytes": 117520398, 2023-11-16T17:00:34.629 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_items": 970811, 2023-11-16T17:00:34.629 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_bytes": 0, 2023-11-16T17:00:34.629 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_items": 0, 2023-11-16T17:00:34.629 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_bytes": 0, 2023-11-16T17:00:34.629 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_items": 0 2023-11-16T17:00:34.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.630 INFO:teuthology.orchestra.run.smithi155.stdout: "objecter": { 2023-11-16T17:00:34.630 INFO:teuthology.orchestra.run.smithi155.stdout: "op_active": 0, 2023-11-16T17:00:34.630 INFO:teuthology.orchestra.run.smithi155.stdout: "op_laggy": 0, 2023-11-16T17:00:34.630 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send": 2236, 2023-11-16T17:00:34.630 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send_bytes": 26, 2023-11-16T17:00:34.631 INFO:teuthology.orchestra.run.smithi155.stdout: "op_resend": 4, 2023-11-16T17:00:34.631 INFO:teuthology.orchestra.run.smithi155.stdout: "op_reply": 2236, 2023-11-16T17:00:34.631 INFO:teuthology.orchestra.run.smithi155.stdout: "op_latency": { 2023-11-16T17:00:34.631 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2236, 2023-11-16T17:00:34.631 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 2.897941066, 2023-11-16T17:00:34.631 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.001296038 2023-11-16T17:00:34.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.632 INFO:teuthology.orchestra.run.smithi155.stdout: "op_inflight": 0, 2023-11-16T17:00:34.632 INFO:teuthology.orchestra.run.smithi155.stdout: "oplen_avg": { 2023-11-16T17:00:34.632 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2236, 2023-11-16T17:00:34.632 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 2237 2023-11-16T17:00:34.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.632 INFO:teuthology.orchestra.run.smithi155.stdout: "op": 2236, 2023-11-16T17:00:34.633 INFO:teuthology.orchestra.run.smithi155.stdout: "op_r": 2236, 2023-11-16T17:00:34.633 INFO:teuthology.orchestra.run.smithi155.stdout: "op_w": 0, 2023-11-16T17:00:34.633 INFO:teuthology.orchestra.run.smithi155.stdout: "op_rmw": 0, 2023-11-16T17:00:34.633 INFO:teuthology.orchestra.run.smithi155.stdout: "op_pg": 0, 2023-11-16T17:00:34.633 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_stat": 1424, 2023-11-16T17:00:34.633 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_create": 0, 2023-11-16T17:00:34.634 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_read": 809, 2023-11-16T17:00:34.634 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_write": 0, 2023-11-16T17:00:34.634 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writefull": 0, 2023-11-16T17:00:34.634 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writesame": 0, 2023-11-16T17:00:34.634 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_append": 0, 2023-11-16T17:00:34.634 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_zero": 0, 2023-11-16T17:00:34.635 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_truncate": 0, 2023-11-16T17:00:34.635 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_delete": 0, 2023-11-16T17:00:34.635 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_mapext": 0, 2023-11-16T17:00:34.635 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_sparse_read": 0, 2023-11-16T17:00:34.635 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_clonerange": 0, 2023-11-16T17:00:34.635 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_getxattr": 2, 2023-11-16T17:00:34.635 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_setxattr": 0, 2023-11-16T17:00:34.636 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_cmpxattr": 0, 2023-11-16T17:00:34.636 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_rmxattr": 0, 2023-11-16T17:00:34.636 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_resetxattrs": 0, 2023-11-16T17:00:34.636 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_call": 0, 2023-11-16T17:00:34.636 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_watch": 0, 2023-11-16T17:00:34.636 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_notify": 0, 2023-11-16T17:00:34.637 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_src_cmpxattr": 0, 2023-11-16T17:00:34.637 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls": 0, 2023-11-16T17:00:34.637 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls_filter": 0, 2023-11-16T17:00:34.637 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_other": 0, 2023-11-16T17:00:34.637 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_active": 0, 2023-11-16T17:00:34.637 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_send": 0, 2023-11-16T17:00:34.637 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_resend": 0, 2023-11-16T17:00:34.638 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_ping": 0, 2023-11-16T17:00:34.638 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_active": 0, 2023-11-16T17:00:34.638 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_send": 0, 2023-11-16T17:00:34.638 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_resend": 0, 2023-11-16T17:00:34.638 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_active": 0, 2023-11-16T17:00:34.638 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_send": 0, 2023-11-16T17:00:34.639 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_resend": 0, 2023-11-16T17:00:34.639 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_active": 0, 2023-11-16T17:00:34.639 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_send": 0, 2023-11-16T17:00:34.639 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_resend": 0, 2023-11-16T17:00:34.639 INFO:teuthology.orchestra.run.smithi155.stdout: "command_active": 0, 2023-11-16T17:00:34.639 INFO:teuthology.orchestra.run.smithi155.stdout: "command_send": 0, 2023-11-16T17:00:34.640 INFO:teuthology.orchestra.run.smithi155.stdout: "command_resend": 0, 2023-11-16T17:00:34.640 INFO:teuthology.orchestra.run.smithi155.stdout: "map_epoch": 104, 2023-11-16T17:00:34.640 INFO:teuthology.orchestra.run.smithi155.stdout: "map_full": 0, 2023-11-16T17:00:34.640 INFO:teuthology.orchestra.run.smithi155.stdout: "map_inc": 9, 2023-11-16T17:00:34.640 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_sessions": 11, 2023-11-16T17:00:34.640 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_open": 11, 2023-11-16T17:00:34.640 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_close": 0, 2023-11-16T17:00:34.641 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_laggy": 0, 2023-11-16T17:00:34.641 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_wr": 0, 2023-11-16T17:00:34.641 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_rd": 2, 2023-11-16T17:00:34.641 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_del": 0 2023-11-16T17:00:34.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.641 INFO:teuthology.orchestra.run.smithi155.stdout: "oft": { 2023-11-16T17:00:34.642 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_objs": 0, 2023-11-16T17:00:34.642 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_kv_pairs": 0, 2023-11-16T17:00:34.642 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_updates": 0, 2023-11-16T17:00:34.642 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_removes": 0 2023-11-16T17:00:34.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.642 INFO:teuthology.orchestra.run.smithi155.stdout: "osd-slow-ops": { 2023-11-16T17:00:34.643 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_ops_count": 0 2023-11-16T17:00:34.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.643 INFO:teuthology.orchestra.run.smithi155.stdout: "purge_queue": { 2023-11-16T17:00:34.643 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops": 0, 2023-11-16T17:00:34.643 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops_high_water": 0, 2023-11-16T17:00:34.643 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing": 0, 2023-11-16T17:00:34.643 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_high_water": 0, 2023-11-16T17:00:34.644 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executed": 0, 2023-11-16T17:00:34.644 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_item_in_journal": 0 2023-11-16T17:00:34.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.644 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:00:34.644 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2023-11-16T17:00:34.644 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2023-11-16T17:00:34.645 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2023-11-16T17:00:34.645 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 2418, 2023-11-16T17:00:34.645 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 277902682, 2023-11-16T17:00:34.645 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:34.645 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 2418, 2023-11-16T17:00:34.645 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2023-11-16T17:00:34.645 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2023-11-16T17:00:34.646 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 2418, 2023-11-16T17:00:34.646 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 277902682, 2023-11-16T17:00:34.646 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2023-11-16T17:00:34.646 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.646 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.646 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.647 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:34.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.647 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_bytes": { 2023-11-16T17:00:34.647 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2023-11-16T17:00:34.647 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2023-11-16T17:00:34.647 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2023-11-16T17:00:34.647 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2023-11-16T17:00:34.648 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2023-11-16T17:00:34.648 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:34.648 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2023-11-16T17:00:34.648 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 2236, 2023-11-16T17:00:34.648 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 277123282, 2023-11-16T17:00:34.648 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 2236, 2023-11-16T17:00:34.648 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 277123282, 2023-11-16T17:00:34.649 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2023-11-16T17:00:34.649 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.649 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.649 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.649 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:34.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.650 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_ops": { 2023-11-16T17:00:34.650 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2023-11-16T17:00:34.650 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 1024, 2023-11-16T17:00:34.650 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2023-11-16T17:00:34.650 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2023-11-16T17:00:34.650 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2023-11-16T17:00:34.651 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:34.651 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2023-11-16T17:00:34.651 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 2236, 2023-11-16T17:00:34.651 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 2236, 2023-11-16T17:00:34.651 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 2236, 2023-11-16T17:00:34.651 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 2236, 2023-11-16T17:00:34.651 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2023-11-16T17:00:34.652 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.652 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.652 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.652 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:34.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.652 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle": { 2023-11-16T17:00:34.652 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2023-11-16T17:00:34.653 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2023-11-16T17:00:34.653 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2023-11-16T17:00:34.653 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2023-11-16T17:00:34.653 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2023-11-16T17:00:34.653 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:34.653 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2023-11-16T17:00:34.653 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2023-11-16T17:00:34.654 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2023-11-16T17:00:34.654 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 0, 2023-11-16T17:00:34.654 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 0, 2023-11-16T17:00:34.654 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2023-11-16T17:00:34.654 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.654 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.655 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.655 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:34.655 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:34.655 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle-0x561bfa438460": { 2023-11-16T17:00:34.655 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2023-11-16T17:00:34.655 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2023-11-16T17:00:34.655 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2023-11-16T17:00:34.656 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2023-11-16T17:00:34.656 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2023-11-16T17:00:34.656 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:34.656 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2023-11-16T17:00:34.656 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2023-11-16T17:00:34.656 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2023-11-16T17:00:34.657 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 0, 2023-11-16T17:00:34.657 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 0, 2023-11-16T17:00:34.657 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2023-11-16T17:00:34.657 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:34.657 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:34.657 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:34.657 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:34.658 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:34.658 INFO:teuthology.orchestra.run.smithi155.stdout:} 2023-11-16T17:00:34.900 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2023-11-16T17:00:34.900 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2023-11-16T17:00:34.900 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2023-11-16T17:00:34.901 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2023-11-16T17:00:34.901 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2023-11-16T17:00:34.901 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2023-11-16T17:00:34.901 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2023-11-16T17:00:34.901 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2023-11-16T17:00:34.902 DEBUG:tasks.check_counter:Getting stats from f 2023-11-16T17:00:34.902 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.f perf dump 2023-11-16T17:00:35.117 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:35 smithi155 ceph-mon[131607]: pgmap v332: 129 pgs: 13 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 80 active+clean; 607 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T17:00:35.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:35 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:35.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:35 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:35.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:35 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:35.118 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:35 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:35.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:35 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T17:00:35.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:35 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T17:00:35.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:35 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:35.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:35 smithi002 ceph-mon[128712]: pgmap v332: 129 pgs: 13 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 80 active+clean; 607 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T17:00:35.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:35 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:35.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:35 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:35.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:35 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:35.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:35 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:35.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:35 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T17:00:35.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:35 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T17:00:35.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:35 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:35.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:35 smithi141 ceph-mon[131184]: pgmap v332: 129 pgs: 13 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 80 active+clean; 607 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T17:00:35.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:35 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:35.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:35 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:35.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:35 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:35.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:35 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-11-16T17:00:35.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:35 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-16T17:00:35.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:35 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-16T17:00:35.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:35 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' 2023-11-16T17:00:35.688 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T17:00:36.216 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:36 smithi155 ceph-mon[131607]: Adjusting osd_memory_target on smithi002 to 223.0M 2023-11-16T17:00:36.217 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:36 smithi155 ceph-mon[131607]: Unable to set osd_memory_target on smithi002 to 233877504: error parsing value: Value '233877504' is below minimum 939524096 2023-11-16T17:00:36.217 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:36 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T17:00:36.217 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:36 smithi155 ceph-mon[131607]: Service mds.all has no related daemon already placed. 2023-11-16T17:00:36.217 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:36 smithi155 ceph-mon[131607]: Service mgr has no related daemon already placed. 2023-11-16T17:00:36.217 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:36 smithi155 ceph-mon[131607]: Service mon has no related daemon already placed. 2023-11-16T17:00:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:36 smithi002 ceph-mon[128712]: Adjusting osd_memory_target on smithi002 to 223.0M 2023-11-16T17:00:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:36 smithi002 ceph-mon[128712]: Unable to set osd_memory_target on smithi002 to 233877504: error parsing value: Value '233877504' is below minimum 939524096 2023-11-16T17:00:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:36 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T17:00:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:36 smithi002 ceph-mon[128712]: Service mds.all has no related daemon already placed. 2023-11-16T17:00:36.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:36 smithi002 ceph-mon[128712]: Service mgr has no related daemon already placed. 2023-11-16T17:00:36.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:36 smithi002 ceph-mon[128712]: Service mon has no related daemon already placed. 2023-11-16T17:00:36.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:36 smithi141 ceph-mon[131184]: Adjusting osd_memory_target on smithi002 to 223.0M 2023-11-16T17:00:36.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:36 smithi141 ceph-mon[131184]: Unable to set osd_memory_target on smithi002 to 233877504: error parsing value: Value '233877504' is below minimum 939524096 2023-11-16T17:00:36.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:36 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T17:00:36.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:36 smithi141 ceph-mon[131184]: Service mds.all has no related daemon already placed. 2023-11-16T17:00:36.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:36 smithi141 ceph-mon[131184]: Service mgr has no related daemon already placed. 2023-11-16T17:00:36.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:36 smithi141 ceph-mon[131184]: Service mon has no related daemon already placed. 2023-11-16T17:00:36.922 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2023-11-16T17:00:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:00:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 2267, 2023-11-16T17:00:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 2267, 2023-11-16T17:00:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 152356599, 2023-11-16T17:00:36.923 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 704248, 2023-11-16T17:00:36.924 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 8, 2023-11-16T17:00:36.924 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 4, 2023-11-16T17:00:36.924 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.284976660, 2023-11-16T17:00:36.924 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.056784262, 2023-11-16T17:00:36.924 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.118710959, 2023-11-16T17:00:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.092929204, 2023-11-16T17:00:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2266, 2023-11-16T17:00:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.105405117, 2023-11-16T17:00:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000046515 2023-11-16T17:00:36.925 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.926 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:36.926 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.926 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.926 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.927 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-11-16T17:00:36.927 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 448 2023-11-16T17:00:36.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.927 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:00:36.928 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 540, 2023-11-16T17:00:36.928 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 513, 2023-11-16T17:00:36.928 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 91004088, 2023-11-16T17:00:36.928 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 213377, 2023-11-16T17:00:36.928 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 6, 2023-11-16T17:00:36.928 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 5, 2023-11-16T17:00:36.929 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.113563141, 2023-11-16T17:00:36.929 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.018669625, 2023-11-16T17:00:36.929 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.088342121, 2023-11-16T17:00:36.929 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.012957118, 2023-11-16T17:00:36.929 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:36.930 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 512, 2023-11-16T17:00:36.930 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.029208851, 2023-11-16T17:00:36.930 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000057048 2023-11-16T17:00:36.930 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.930 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:36.931 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.931 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.931 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.931 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.931 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 1518656, 2023-11-16T17:00:36.931 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 127168 2023-11-16T17:00:36.932 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.932 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:00:36.932 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 453, 2023-11-16T17:00:36.932 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 617, 2023-11-16T17:00:36.932 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 139349969, 2023-11-16T17:00:36.933 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 573157, 2023-11-16T17:00:36.933 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 8, 2023-11-16T17:00:36.933 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 6, 2023-11-16T17:00:36.933 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.133574825, 2023-11-16T17:00:36.933 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.014175964, 2023-11-16T17:00:36.933 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.075723436, 2023-11-16T17:00:36.934 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.038888751, 2023-11-16T17:00:36.934 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:36.934 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 614, 2023-11-16T17:00:36.934 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.042141790, 2023-11-16T17:00:36.934 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000068634 2023-11-16T17:00:36.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.935 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:36.935 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.935 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.935 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 320, 2023-11-16T17:00:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 432896 2023-11-16T17:00:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: "cct": { 2023-11-16T17:00:36.936 INFO:teuthology.orchestra.run.smithi155.stdout: "total_workers": 1, 2023-11-16T17:00:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: "unhealthy_workers": 0 2023-11-16T17:00:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-MDSRank": { 2023-11-16T17:00:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2023-11-16T17:00:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2023-11-16T17:00:36.937 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 4590, 2023-11-16T17:00:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.135139375, 2023-11-16T17:00:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000029442 2023-11-16T17:00:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-PurgeQueue": { 2023-11-16T17:00:36.938 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2023-11-16T17:00:36.939 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2023-11-16T17:00:36.939 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.939 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.939 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.939 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: "mds": { 2023-11-16T17:00:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: "request": 0, 2023-11-16T17:00:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: "reply": 0, 2023-11-16T17:00:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: "reply_latency": { 2023-11-16T17:00:36.940 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_reply": 0, 2023-11-16T17:00:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: "forward": 0, 2023-11-16T17:00:36.941 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_complete": 0, 2023-11-16T17:00:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_keys": 0, 2023-11-16T17:00:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_commit": 0, 2023-11-16T17:00:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_split": 0, 2023-11-16T17:00:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_merge": 0, 2023-11-16T17:00:36.942 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes": 90557, 2023-11-16T17:00:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_top": 537, 2023-11-16T17:00:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_bottom": 69669, 2023-11-16T17:00:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pin_tail": 20351, 2023-11-16T17:00:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pinned": 89789, 2023-11-16T17:00:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_expired": 14, 2023-11-16T17:00:36.943 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_with_caps": 0, 2023-11-16T17:00:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: "caps": 0, 2023-11-16T17:00:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: "subtrees": 3, 2023-11-16T17:00:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse": 0, 2023-11-16T17:00:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_hit": 0, 2023-11-16T17:00:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_forward": 0, 2023-11-16T17:00:36.944 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_discover": 0, 2023-11-16T17:00:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_dir_fetch": 0, 2023-11-16T17:00:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_remote_ino": 0, 2023-11-16T17:00:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_lock": 0, 2023-11-16T17:00:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: "load_cent": 0, 2023-11-16T17:00:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: "q": 0, 2023-11-16T17:00:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: "exported": 0, 2023-11-16T17:00:36.945 INFO:teuthology.orchestra.run.smithi155.stdout: "exported_inodes": 0, 2023-11-16T17:00:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: "imported": 0, 2023-11-16T17:00:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: "imported_inodes": 0, 2023-11-16T17:00:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_dir_fetch": 0, 2023-11-16T17:00:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_backtrace_fetch": 0, 2023-11-16T17:00:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_peer_discover": 0, 2023-11-16T17:00:36.946 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rfiles": 0, 2023-11-16T17:00:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rbytes": 0, 2023-11-16T17:00:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rsnaps": 0, 2023-11-16T17:00:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_fetch": 0, 2023-11-16T17:00:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_set_tag": 0, 2023-11-16T17:00:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_repaired": 0, 2023-11-16T17:00:36.948 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_inotable_repaired": 0, 2023-11-16T17:00:36.948 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_inodes": 0, 2023-11-16T17:00:36.948 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_base_inodes": 0, 2023-11-16T17:00:36.948 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dirfrag_rstats": 0, 2023-11-16T17:00:36.948 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_file_inodes": 0, 2023-11-16T17:00:36.948 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_inode_file_caps": 0, 2023-11-16T17:00:36.949 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_revoke": 0, 2023-11-16T17:00:36.949 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_grant": 0, 2023-11-16T17:00:36.949 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_trunc": 0, 2023-11-16T17:00:36.949 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-11-16T17:00:36.949 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-16T17:00:36.949 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps": 0, 2023-11-16T17:00:36.950 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps_dirty": 0, 2023-11-16T17:00:36.950 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_cap_release": 0, 2023-11-16T17:00:36.950 INFO:teuthology.orchestra.run.smithi155.stdout: "process_request_cap_release": 0 2023-11-16T17:00:36.950 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.950 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_cache": { 2023-11-16T17:00:36.950 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update": 0, 2023-11-16T17:00:36.951 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update_receipt": 0, 2023-11-16T17:00:36.951 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_try_discover": 0, 2023-11-16T17:00:36.951 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_send_discover": 0, 2023-11-16T17:00:36.951 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_handle_discover": 0, 2023-11-16T17:00:36.951 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays": 0, 2023-11-16T17:00:36.951 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_delayed": 0, 2023-11-16T17:00:36.951 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_enqueuing": 0, 2023-11-16T17:00:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_created": 0, 2023-11-16T17:00:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_enqueued": 0, 2023-11-16T17:00:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_reintegrated": 0, 2023-11-16T17:00:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_migrated": 0, 2023-11-16T17:00:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_processing": 0, 2023-11-16T17:00:36.952 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_enqueued": 0, 2023-11-16T17:00:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_prioritized": 0, 2023-11-16T17:00:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_started": 0, 2023-11-16T17:00:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_completed": 0, 2023-11-16T17:00:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_enqueue_scrub": 0, 2023-11-16T17:00:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_exportdir": 0, 2023-11-16T17:00:36.953 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_flush": 0, 2023-11-16T17:00:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragmentdir": 0, 2023-11-16T17:00:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragstats": 0, 2023-11-16T17:00:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_inodestats": 0 2023-11-16T17:00:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_log": { 2023-11-16T17:00:36.954 INFO:teuthology.orchestra.run.smithi155.stdout: "evadd": 0, 2023-11-16T17:00:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: "evex": 0, 2023-11-16T17:00:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: "evtrm": 0, 2023-11-16T17:00:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: "ev": 0, 2023-11-16T17:00:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: "evexg": 0, 2023-11-16T17:00:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: "evexd": 0, 2023-11-16T17:00:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: "segadd": 0, 2023-11-16T17:00:36.955 INFO:teuthology.orchestra.run.smithi155.stdout: "segex": 0, 2023-11-16T17:00:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: "segtrm": 0, 2023-11-16T17:00:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: "seg": 129, 2023-11-16T17:00:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: "segexg": 0, 2023-11-16T17:00:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: "segexd": 0, 2023-11-16T17:00:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: "expos": 19047953, 2023-11-16T17:00:36.956 INFO:teuthology.orchestra.run.smithi155.stdout: "wrpos": 0, 2023-11-16T17:00:36.957 INFO:teuthology.orchestra.run.smithi155.stdout: "rdpos": 382082344, 2023-11-16T17:00:36.957 INFO:teuthology.orchestra.run.smithi155.stdout: "jlat": { 2023-11-16T17:00:36.957 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.957 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.957 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.957 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.957 INFO:teuthology.orchestra.run.smithi155.stdout: "replayed": 91184 2023-11-16T17:00:36.958 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.958 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_mem": { 2023-11-16T17:00:36.958 INFO:teuthology.orchestra.run.smithi155.stdout: "ino": 42920, 2023-11-16T17:00:36.958 INFO:teuthology.orchestra.run.smithi155.stdout: "ino+": 78926, 2023-11-16T17:00:36.958 INFO:teuthology.orchestra.run.smithi155.stdout: "ino-": 36006, 2023-11-16T17:00:36.958 INFO:teuthology.orchestra.run.smithi155.stdout: "dir": 1553, 2023-11-16T17:00:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: "dir+": 1913, 2023-11-16T17:00:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: "dir-": 360, 2023-11-16T17:00:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: "dn": 90557, 2023-11-16T17:00:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: "dn+": 117475, 2023-11-16T17:00:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: "dn-": 26918, 2023-11-16T17:00:36.959 INFO:teuthology.orchestra.run.smithi155.stdout: "cap": 0, 2023-11-16T17:00:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: "cap+": 0, 2023-11-16T17:00:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: "cap-": 0, 2023-11-16T17:00:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: "rss": 248720, 2023-11-16T17:00:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: "heap": 207156 2023-11-16T17:00:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.960 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_server": { 2023-11-16T17:00:36.961 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_client_request": 0, 2023-11-16T17:00:36.961 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_server_request": 0, 2023-11-16T17:00:36.961 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_request": 0, 2023-11-16T17:00:36.961 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_session": 0, 2023-11-16T17:00:36.961 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_peer_request": 0, 2023-11-16T17:00:36.961 INFO:teuthology.orchestra.run.smithi155.stdout: "req_create_latency": { 2023-11-16T17:00:36.961 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.962 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.962 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.962 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getattr_latency": { 2023-11-16T17:00:36.962 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.962 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.963 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.963 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getfilelock_latency": { 2023-11-16T17:00:36.963 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.963 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.963 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: "req_link_latency": { 2023-11-16T17:00:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.964 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookup_latency": { 2023-11-16T17:00:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookuphash_latency": { 2023-11-16T17:00:36.965 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupino_latency": { 2023-11-16T17:00:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.966 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupname_latency": { 2023-11-16T17:00:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupparent_latency": { 2023-11-16T17:00:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupsnap_latency": { 2023-11-16T17:00:36.968 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.969 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.969 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.969 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.969 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lssnap_latency": { 2023-11-16T17:00:36.969 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.969 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.969 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.970 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.970 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mkdir_latency": { 2023-11-16T17:00:36.970 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.970 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.970 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.970 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.971 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mknod_latency": { 2023-11-16T17:00:36.971 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.971 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.971 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.971 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.971 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mksnap_latency": { 2023-11-16T17:00:36.971 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.972 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.972 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.972 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.972 INFO:teuthology.orchestra.run.smithi155.stdout: "req_open_latency": { 2023-11-16T17:00:36.972 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.972 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.973 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.973 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.973 INFO:teuthology.orchestra.run.smithi155.stdout: "req_readdir_latency": { 2023-11-16T17:00:36.973 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.973 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.973 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.973 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.974 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rename_latency": { 2023-11-16T17:00:36.974 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.974 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.974 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.974 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.974 INFO:teuthology.orchestra.run.smithi155.stdout: "req_renamesnap_latency": { 2023-11-16T17:00:36.974 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.975 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.975 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.975 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.975 INFO:teuthology.orchestra.run.smithi155.stdout: "req_snapdiff_latency": { 2023-11-16T17:00:36.975 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.975 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.976 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.976 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.976 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmdir_latency": { 2023-11-16T17:00:36.976 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.976 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.976 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.977 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.977 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmsnap_latency": { 2023-11-16T17:00:36.977 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.977 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.977 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.977 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.977 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmxattr_latency": { 2023-11-16T17:00:36.978 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.978 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.978 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.978 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.978 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setattr_latency": { 2023-11-16T17:00:36.978 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.979 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.979 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.979 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.979 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setdirlayout_latency": { 2023-11-16T17:00:36.979 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.979 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.980 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.980 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.980 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setfilelock_latency": { 2023-11-16T17:00:36.980 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.980 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.980 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.981 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.981 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setlayout_latency": { 2023-11-16T17:00:36.981 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.981 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.981 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.981 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.982 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setxattr_latency": { 2023-11-16T17:00:36.982 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.982 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.982 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.982 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.982 INFO:teuthology.orchestra.run.smithi155.stdout: "req_symlink_latency": { 2023-11-16T17:00:36.982 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.983 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.983 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.983 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.983 INFO:teuthology.orchestra.run.smithi155.stdout: "req_unlink_latency": { 2023-11-16T17:00:36.983 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.983 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.983 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.984 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.984 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_revoke_eviction": 0, 2023-11-16T17:00:36.984 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_acquisition_throttle": 0, 2023-11-16T17:00:36.984 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getvxattr_latency": { 2023-11-16T17:00:36.984 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:36.984 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:36.985 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:36.985 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:36.985 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.985 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_sessions": { 2023-11-16T17:00:36.985 INFO:teuthology.orchestra.run.smithi155.stdout: "session_count": 2, 2023-11-16T17:00:36.985 INFO:teuthology.orchestra.run.smithi155.stdout: "session_add": 5, 2023-11-16T17:00:36.985 INFO:teuthology.orchestra.run.smithi155.stdout: "session_remove": 3, 2023-11-16T17:00:36.986 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_open": 4, 2023-11-16T17:00:36.986 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_stale": 0, 2023-11-16T17:00:36.986 INFO:teuthology.orchestra.run.smithi155.stdout: "total_load": 0, 2023-11-16T17:00:36.986 INFO:teuthology.orchestra.run.smithi155.stdout: "average_load": 0, 2023-11-16T17:00:36.986 INFO:teuthology.orchestra.run.smithi155.stdout: "avg_session_uptime": 4741 2023-11-16T17:00:36.986 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.986 INFO:teuthology.orchestra.run.smithi155.stdout: "mempool": { 2023-11-16T17:00:36.987 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:00:36.987 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_items": 0, 2023-11-16T17:00:36.987 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:00:36.987 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:00:36.987 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:00:36.987 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:00:36.988 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:00:36.988 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:00:36.988 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:00:36.988 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:00:36.988 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:00:36.988 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:00:36.989 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:00:36.989 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:00:36.989 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:00:36.989 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_items": 0, 2023-11-16T17:00:36.989 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:00:36.989 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_items": 0, 2023-11-16T17:00:36.989 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:00:36.990 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:00:36.990 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:00:36.990 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:00:36.990 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:00:36.990 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:00:36.990 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:00:36.990 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_items": 0, 2023-11-16T17:00:36.991 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:00:36.991 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:00:36.991 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:00:36.991 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_items": 0, 2023-11-16T17:00:36.991 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_bytes": 0, 2023-11-16T17:00:36.991 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_items": 0, 2023-11-16T17:00:36.991 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:00:36.992 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:00:36.992 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:00:36.992 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:00:36.992 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_bytes": 1699125, 2023-11-16T17:00:36.992 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_items": 43010, 2023-11-16T17:00:36.992 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:00:36.992 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_items": 0, 2023-11-16T17:00:36.993 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_bytes": 0, 2023-11-16T17:00:36.993 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_items": 0, 2023-11-16T17:00:36.993 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:00:36.993 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_items": 0, 2023-11-16T17:00:36.993 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:00:36.993 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_items": 0, 2023-11-16T17:00:36.993 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_bytes": 9944, 2023-11-16T17:00:36.994 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_items": 192, 2023-11-16T17:00:36.994 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:00:36.994 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:00:36.994 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_bytes": 0, 2023-11-16T17:00:36.994 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_items": 0, 2023-11-16T17:00:36.994 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_bytes": 162384441, 2023-11-16T17:00:36.994 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_items": 1339675, 2023-11-16T17:00:36.995 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_bytes": 0, 2023-11-16T17:00:36.995 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_items": 0, 2023-11-16T17:00:36.995 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_bytes": 0, 2023-11-16T17:00:36.995 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_items": 0 2023-11-16T17:00:36.995 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.995 INFO:teuthology.orchestra.run.smithi155.stdout: "objecter": { 2023-11-16T17:00:36.995 INFO:teuthology.orchestra.run.smithi155.stdout: "op_active": 0, 2023-11-16T17:00:36.995 INFO:teuthology.orchestra.run.smithi155.stdout: "op_laggy": 0, 2023-11-16T17:00:36.996 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send": 2991, 2023-11-16T17:00:36.996 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send_bytes": 21, 2023-11-16T17:00:36.996 INFO:teuthology.orchestra.run.smithi155.stdout: "op_resend": 3, 2023-11-16T17:00:36.996 INFO:teuthology.orchestra.run.smithi155.stdout: "op_reply": 2991, 2023-11-16T17:00:36.996 INFO:teuthology.orchestra.run.smithi155.stdout: "op_latency": { 2023-11-16T17:00:36.996 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2991, 2023-11-16T17:00:36.996 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 7.631846417, 2023-11-16T17:00:36.997 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.002551603 2023-11-16T17:00:36.997 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.997 INFO:teuthology.orchestra.run.smithi155.stdout: "op_inflight": 0, 2023-11-16T17:00:36.997 INFO:teuthology.orchestra.run.smithi155.stdout: "oplen_avg": { 2023-11-16T17:00:36.997 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2991, 2023-11-16T17:00:36.997 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 2992 2023-11-16T17:00:36.997 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:36.998 INFO:teuthology.orchestra.run.smithi155.stdout: "op": 2991, 2023-11-16T17:00:36.998 INFO:teuthology.orchestra.run.smithi155.stdout: "op_r": 2991, 2023-11-16T17:00:36.998 INFO:teuthology.orchestra.run.smithi155.stdout: "op_w": 0, 2023-11-16T17:00:36.998 INFO:teuthology.orchestra.run.smithi155.stdout: "op_rmw": 0, 2023-11-16T17:00:36.998 INFO:teuthology.orchestra.run.smithi155.stdout: "op_pg": 0, 2023-11-16T17:00:36.998 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_stat": 1943, 2023-11-16T17:00:36.998 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_create": 0, 2023-11-16T17:00:36.999 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_read": 1046, 2023-11-16T17:00:36.999 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_write": 0, 2023-11-16T17:00:36.999 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writefull": 0, 2023-11-16T17:00:36.999 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writesame": 0, 2023-11-16T17:00:36.999 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_append": 0, 2023-11-16T17:00:36.999 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_zero": 0, 2023-11-16T17:00:36.999 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_truncate": 0, 2023-11-16T17:00:37.000 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_delete": 0, 2023-11-16T17:00:37.000 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_mapext": 0, 2023-11-16T17:00:37.000 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_sparse_read": 0, 2023-11-16T17:00:37.000 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_clonerange": 0, 2023-11-16T17:00:37.000 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_getxattr": 1, 2023-11-16T17:00:37.000 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_setxattr": 0, 2023-11-16T17:00:37.000 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_cmpxattr": 0, 2023-11-16T17:00:37.001 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_rmxattr": 0, 2023-11-16T17:00:37.001 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_resetxattrs": 0, 2023-11-16T17:00:37.001 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_call": 0, 2023-11-16T17:00:37.001 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_watch": 0, 2023-11-16T17:00:37.001 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_notify": 0, 2023-11-16T17:00:37.001 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_src_cmpxattr": 0, 2023-11-16T17:00:37.001 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls": 0, 2023-11-16T17:00:37.001 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls_filter": 0, 2023-11-16T17:00:37.002 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_other": 0, 2023-11-16T17:00:37.002 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_active": 0, 2023-11-16T17:00:37.002 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_send": 0, 2023-11-16T17:00:37.002 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_resend": 0, 2023-11-16T17:00:37.002 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_ping": 0, 2023-11-16T17:00:37.002 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_active": 0, 2023-11-16T17:00:37.002 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_send": 0, 2023-11-16T17:00:37.003 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_resend": 0, 2023-11-16T17:00:37.003 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_active": 0, 2023-11-16T17:00:37.003 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_send": 0, 2023-11-16T17:00:37.003 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_resend": 0, 2023-11-16T17:00:37.003 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_active": 0, 2023-11-16T17:00:37.003 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_send": 0, 2023-11-16T17:00:37.003 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_resend": 0, 2023-11-16T17:00:37.004 INFO:teuthology.orchestra.run.smithi155.stdout: "command_active": 0, 2023-11-16T17:00:37.004 INFO:teuthology.orchestra.run.smithi155.stdout: "command_send": 0, 2023-11-16T17:00:37.004 INFO:teuthology.orchestra.run.smithi155.stdout: "command_resend": 0, 2023-11-16T17:00:37.004 INFO:teuthology.orchestra.run.smithi155.stdout: "map_epoch": 104, 2023-11-16T17:00:37.004 INFO:teuthology.orchestra.run.smithi155.stdout: "map_full": 0, 2023-11-16T17:00:37.004 INFO:teuthology.orchestra.run.smithi155.stdout: "map_inc": 13, 2023-11-16T17:00:37.004 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_sessions": 11, 2023-11-16T17:00:37.004 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_open": 11, 2023-11-16T17:00:37.005 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_close": 0, 2023-11-16T17:00:37.005 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_laggy": 0, 2023-11-16T17:00:37.005 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_wr": 0, 2023-11-16T17:00:37.005 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_rd": 2, 2023-11-16T17:00:37.005 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_del": 0 2023-11-16T17:00:37.005 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:37.005 INFO:teuthology.orchestra.run.smithi155.stdout: "oft": { 2023-11-16T17:00:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_objs": 0, 2023-11-16T17:00:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_kv_pairs": 0, 2023-11-16T17:00:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_updates": 0, 2023-11-16T17:00:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_removes": 0 2023-11-16T17:00:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: "osd-slow-ops": { 2023-11-16T17:00:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_ops_count": 0 2023-11-16T17:00:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: "purge_queue": { 2023-11-16T17:00:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops": 0, 2023-11-16T17:00:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops_high_water": 0, 2023-11-16T17:00:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing": 0, 2023-11-16T17:00:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_high_water": 0, 2023-11-16T17:00:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executed": 0, 2023-11-16T17:00:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_item_in_journal": 0 2023-11-16T17:00:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:00:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2023-11-16T17:00:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2023-11-16T17:00:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2023-11-16T17:00:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 3256, 2023-11-16T17:00:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 382401196, 2023-11-16T17:00:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 3256, 2023-11-16T17:00:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2023-11-16T17:00:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2023-11-16T17:00:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 3256, 2023-11-16T17:00:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 382401196, 2023-11-16T17:00:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2023-11-16T17:00:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_bytes": { 2023-11-16T17:00:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2023-11-16T17:00:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2023-11-16T17:00:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2023-11-16T17:00:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2023-11-16T17:00:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2023-11-16T17:00:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2023-11-16T17:00:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 2991, 2023-11-16T17:00:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 380362864, 2023-11-16T17:00:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 2991, 2023-11-16T17:00:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 380362864, 2023-11-16T17:00:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2023-11-16T17:00:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:37.013 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_ops": { 2023-11-16T17:00:37.013 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2023-11-16T17:00:37.013 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 1024, 2023-11-16T17:00:37.013 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2023-11-16T17:00:37.013 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2023-11-16T17:00:37.013 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2023-11-16T17:00:37.013 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2023-11-16T17:00:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 2991, 2023-11-16T17:00:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 2991, 2023-11-16T17:00:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 2991, 2023-11-16T17:00:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 2991, 2023-11-16T17:00:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2023-11-16T17:00:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle": { 2023-11-16T17:00:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2023-11-16T17:00:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2023-11-16T17:00:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2023-11-16T17:00:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2023-11-16T17:00:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2023-11-16T17:00:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2023-11-16T17:00:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2023-11-16T17:00:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2023-11-16T17:00:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 0, 2023-11-16T17:00:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 0, 2023-11-16T17:00:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2023-11-16T17:00:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle-0x5560730de460": { 2023-11-16T17:00:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2023-11-16T17:00:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2023-11-16T17:00:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2023-11-16T17:00:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2023-11-16T17:00:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2023-11-16T17:00:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2023-11-16T17:00:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2023-11-16T17:00:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2023-11-16T17:00:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 0, 2023-11-16T17:00:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 0, 2023-11-16T17:00:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2023-11-16T17:00:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:37.020 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:37.020 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:37.020 INFO:teuthology.orchestra.run.smithi155.stdout:} 2023-11-16T17:00:37.270 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2023-11-16T17:00:37.270 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2023-11-16T17:00:37.270 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2023-11-16T17:00:37.270 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2023-11-16T17:00:37.270 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2023-11-16T17:00:37.271 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2023-11-16T17:00:37.271 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2023-11-16T17:00:37.271 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2023-11-16T17:00:37.271 DEBUG:tasks.check_counter:Getting stats from i 2023-11-16T17:00:37.271 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.i perf dump 2023-11-16T17:00:37.299 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:37 smithi155 ceph-mon[131607]: pgmap v333: 129 pgs: 13 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 80 active+clean; 607 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T17:00:37.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:37 smithi002 ceph-mon[128712]: pgmap v333: 129 pgs: 13 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 80 active+clean; 607 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T17:00:37.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:37 smithi141 ceph-mon[131184]: pgmap v333: 129 pgs: 13 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 80 active+clean; 607 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T17:00:37.960 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T17:00:39.176 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2023-11-16T17:00:39.176 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:00:39.177 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 118685, 2023-11-16T17:00:39.177 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 197569, 2023-11-16T17:00:39.177 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 100733138, 2023-11-16T17:00:39.177 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 187054945, 2023-11-16T17:00:39.177 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 14, 2023-11-16T17:00:39.177 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 4, 2023-11-16T17:00:39.177 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 6.996169813, 2023-11-16T17:00:39.177 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 2.968346983, 2023-11-16T17:00:39.178 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 14.869265538, 2023-11-16T17:00:39.178 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.519877813, 2023-11-16T17:00:39.178 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:39.178 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 197569, 2023-11-16T17:00:39.178 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 14.040870779, 2023-11-16T17:00:39.178 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000071068 2023-11-16T17:00:39.178 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.178 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:39.179 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 249836, 2023-11-16T17:00:39.179 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.041162609, 2023-11-16T17:00:39.179 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000164 2023-11-16T17:00:39.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.179 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 1537440, 2023-11-16T17:00:39.179 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 144592 2023-11-16T17:00:39.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.180 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:00:39.180 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 18601, 2023-11-16T17:00:39.180 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 18643, 2023-11-16T17:00:39.180 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 9465630, 2023-11-16T17:00:39.180 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 160057842, 2023-11-16T17:00:39.180 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 13, 2023-11-16T17:00:39.180 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 6, 2023-11-16T17:00:39.180 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 1.830416173, 2023-11-16T17:00:39.181 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.661819782, 2023-11-16T17:00:39.181 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.286639991, 2023-11-16T17:00:39.181 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.803799017, 2023-11-16T17:00:39.181 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:39.181 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 18640, 2023-11-16T17:00:39.181 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 2.259308129, 2023-11-16T17:00:39.181 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000121207 2023-11-16T17:00:39.181 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.182 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:39.182 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 27, 2023-11-16T17:00:39.182 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000004165, 2023-11-16T17:00:39.182 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000154 2023-11-16T17:00:39.182 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.182 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 2896, 2023-11-16T17:00:39.182 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 131584 2023-11-16T17:00:39.182 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.183 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:00:39.183 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 27484, 2023-11-16T17:00:39.183 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 24061, 2023-11-16T17:00:39.183 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 44331574, 2023-11-16T17:00:39.183 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 126394908, 2023-11-16T17:00:39.183 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 8, 2023-11-16T17:00:39.183 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 6, 2023-11-16T17:00:39.183 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 2.018170692, 2023-11-16T17:00:39.184 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.679417403, 2023-11-16T17:00:39.184 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.761119208, 2023-11-16T17:00:39.184 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.699969424, 2023-11-16T17:00:39.184 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:39.184 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 24059, 2023-11-16T17:00:39.184 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 3.112338515, 2023-11-16T17:00:39.184 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000129362 2023-11-16T17:00:39.184 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.185 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:39.185 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 22249, 2023-11-16T17:00:39.185 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.003162548, 2023-11-16T17:00:39.185 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000142 2023-11-16T17:00:39.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.185 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 160, 2023-11-16T17:00:39.185 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 308016 2023-11-16T17:00:39.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.186 INFO:teuthology.orchestra.run.smithi155.stdout: "cct": { 2023-11-16T17:00:39.186 INFO:teuthology.orchestra.run.smithi155.stdout: "total_workers": 1, 2023-11-16T17:00:39.186 INFO:teuthology.orchestra.run.smithi155.stdout: "unhealthy_workers": 0 2023-11-16T17:00:39.186 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.186 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-MDSRank": { 2023-11-16T17:00:39.186 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2023-11-16T17:00:39.186 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2023-11-16T17:00:39.186 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 3968, 2023-11-16T17:00:39.187 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 71.051849001, 2023-11-16T17:00:39.187 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.017906211 2023-11-16T17:00:39.187 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:39.187 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.187 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-PurgeQueue": { 2023-11-16T17:00:39.187 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2023-11-16T17:00:39.187 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2023-11-16T17:00:39.187 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 31267, 2023-11-16T17:00:39.188 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 11.053125824, 2023-11-16T17:00:39.188 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000353507 2023-11-16T17:00:39.188 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:39.188 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.188 INFO:teuthology.orchestra.run.smithi155.stdout: "mds": { 2023-11-16T17:00:39.188 INFO:teuthology.orchestra.run.smithi155.stdout: "request": 69034, 2023-11-16T17:00:39.188 INFO:teuthology.orchestra.run.smithi155.stdout: "reply": 58950, 2023-11-16T17:00:39.188 INFO:teuthology.orchestra.run.smithi155.stdout: "reply_latency": { 2023-11-16T17:00:39.189 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 58950, 2023-11-16T17:00:39.189 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 459.468791662, 2023-11-16T17:00:39.189 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.007794211 2023-11-16T17:00:39.189 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.189 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_reply": 0, 2023-11-16T17:00:39.189 INFO:teuthology.orchestra.run.smithi155.stdout: "forward": 10084, 2023-11-16T17:00:39.189 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_complete": 1023, 2023-11-16T17:00:39.190 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_keys": 0, 2023-11-16T17:00:39.190 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_commit": 1089, 2023-11-16T17:00:39.190 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_split": 132, 2023-11-16T17:00:39.190 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_merge": 8, 2023-11-16T17:00:39.190 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes": 73009, 2023-11-16T17:00:39.190 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_top": 1624, 2023-11-16T17:00:39.190 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_bottom": 7008, 2023-11-16T17:00:39.191 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pin_tail": 64377, 2023-11-16T17:00:39.191 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pinned": 70689, 2023-11-16T17:00:39.191 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_expired": 23261, 2023-11-16T17:00:39.191 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_with_caps": 5, 2023-11-16T17:00:39.191 INFO:teuthology.orchestra.run.smithi155.stdout: "caps": 5, 2023-11-16T17:00:39.191 INFO:teuthology.orchestra.run.smithi155.stdout: "subtrees": 5, 2023-11-16T17:00:39.191 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse": 87865, 2023-11-16T17:00:39.191 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_hit": 77463, 2023-11-16T17:00:39.191 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_forward": 5355, 2023-11-16T17:00:39.192 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_discover": 23, 2023-11-16T17:00:39.192 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_dir_fetch": 0, 2023-11-16T17:00:39.192 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_remote_ino": 0, 2023-11-16T17:00:39.192 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_lock": 0, 2023-11-16T17:00:39.192 INFO:teuthology.orchestra.run.smithi155.stdout: "load_cent": 0, 2023-11-16T17:00:39.192 INFO:teuthology.orchestra.run.smithi155.stdout: "q": 0, 2023-11-16T17:00:39.192 INFO:teuthology.orchestra.run.smithi155.stdout: "exported": 453, 2023-11-16T17:00:39.193 INFO:teuthology.orchestra.run.smithi155.stdout: "exported_inodes": 26888, 2023-11-16T17:00:39.193 INFO:teuthology.orchestra.run.smithi155.stdout: "imported": 1077, 2023-11-16T17:00:39.193 INFO:teuthology.orchestra.run.smithi155.stdout: "imported_inodes": 0, 2023-11-16T17:00:39.193 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_dir_fetch": 0, 2023-11-16T17:00:39.193 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_backtrace_fetch": 0, 2023-11-16T17:00:39.193 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_peer_discover": 0, 2023-11-16T17:00:39.193 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rfiles": 4, 2023-11-16T17:00:39.193 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rbytes": 246, 2023-11-16T17:00:39.193 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rsnaps": 0, 2023-11-16T17:00:39.194 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_fetch": 0, 2023-11-16T17:00:39.194 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_set_tag": 0, 2023-11-16T17:00:39.194 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_repaired": 0, 2023-11-16T17:00:39.194 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_inotable_repaired": 0, 2023-11-16T17:00:39.194 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_inodes": 0, 2023-11-16T17:00:39.194 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_base_inodes": 0, 2023-11-16T17:00:39.194 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dirfrag_rstats": 0, 2023-11-16T17:00:39.195 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_file_inodes": 0, 2023-11-16T17:00:39.195 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_inode_file_caps": 12, 2023-11-16T17:00:39.195 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_revoke": 72, 2023-11-16T17:00:39.195 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_grant": 38709, 2023-11-16T17:00:39.195 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_trunc": 1, 2023-11-16T17:00:39.195 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-11-16T17:00:39.195 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flush_ack": 0, 2023-11-16T17:00:39.195 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps": 32330, 2023-11-16T17:00:39.196 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps_dirty": 32209, 2023-11-16T17:00:39.196 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_cap_release": 291, 2023-11-16T17:00:39.196 INFO:teuthology.orchestra.run.smithi155.stdout: "process_request_cap_release": 38190 2023-11-16T17:00:39.196 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.196 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_cache": { 2023-11-16T17:00:39.196 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update": 1718, 2023-11-16T17:00:39.196 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update_receipt": 2866, 2023-11-16T17:00:39.196 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_try_discover": 1437, 2023-11-16T17:00:39.196 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_send_discover": 1851, 2023-11-16T17:00:39.197 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_handle_discover": 2247, 2023-11-16T17:00:39.197 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays": 9, 2023-11-16T17:00:39.197 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_delayed": 0, 2023-11-16T17:00:39.197 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_enqueuing": 0, 2023-11-16T17:00:39.197 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_created": 17644, 2023-11-16T17:00:39.197 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_enqueued": 17635, 2023-11-16T17:00:39.197 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_reintegrated": 0, 2023-11-16T17:00:39.198 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_migrated": 0, 2023-11-16T17:00:39.198 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_processing": 0, 2023-11-16T17:00:39.198 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_enqueued": 0, 2023-11-16T17:00:39.198 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_prioritized": 0, 2023-11-16T17:00:39.198 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_started": 0, 2023-11-16T17:00:39.198 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_completed": 0, 2023-11-16T17:00:39.198 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_enqueue_scrub": 0, 2023-11-16T17:00:39.199 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_exportdir": 455, 2023-11-16T17:00:39.199 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_flush": 0, 2023-11-16T17:00:39.199 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragmentdir": 145, 2023-11-16T17:00:39.199 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragstats": 0, 2023-11-16T17:00:39.199 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_inodestats": 0 2023-11-16T17:00:39.199 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.199 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_log": { 2023-11-16T17:00:39.199 INFO:teuthology.orchestra.run.smithi155.stdout: "evadd": 91184, 2023-11-16T17:00:39.200 INFO:teuthology.orchestra.run.smithi155.stdout: "evex": 6766, 2023-11-16T17:00:39.200 INFO:teuthology.orchestra.run.smithi155.stdout: "evtrm": 6766, 2023-11-16T17:00:39.200 INFO:teuthology.orchestra.run.smithi155.stdout: "ev": 84418, 2023-11-16T17:00:39.200 INFO:teuthology.orchestra.run.smithi155.stdout: "evexg": 0, 2023-11-16T17:00:39.200 INFO:teuthology.orchestra.run.smithi155.stdout: "evexd": 1024, 2023-11-16T17:00:39.200 INFO:teuthology.orchestra.run.smithi155.stdout: "segadd": 135, 2023-11-16T17:00:39.200 INFO:teuthology.orchestra.run.smithi155.stdout: "segex": 7, 2023-11-16T17:00:39.200 INFO:teuthology.orchestra.run.smithi155.stdout: "segtrm": 7, 2023-11-16T17:00:39.201 INFO:teuthology.orchestra.run.smithi155.stdout: "seg": 129, 2023-11-16T17:00:39.201 INFO:teuthology.orchestra.run.smithi155.stdout: "segexg": 0, 2023-11-16T17:00:39.201 INFO:teuthology.orchestra.run.smithi155.stdout: "segexd": 1, 2023-11-16T17:00:39.201 INFO:teuthology.orchestra.run.smithi155.stdout: "expos": 19047953, 2023-11-16T17:00:39.201 INFO:teuthology.orchestra.run.smithi155.stdout: "wrpos": 382082918, 2023-11-16T17:00:39.201 INFO:teuthology.orchestra.run.smithi155.stdout: "rdpos": 0, 2023-11-16T17:00:39.201 INFO:teuthology.orchestra.run.smithi155.stdout: "jlat": { 2023-11-16T17:00:39.201 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2353, 2023-11-16T17:00:39.202 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 226.715562258, 2023-11-16T17:00:39.202 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.096351705 2023-11-16T17:00:39.202 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.202 INFO:teuthology.orchestra.run.smithi155.stdout: "replayed": 0 2023-11-16T17:00:39.202 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.202 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_mem": { 2023-11-16T17:00:39.202 INFO:teuthology.orchestra.run.smithi155.stdout: "ino": 43001, 2023-11-16T17:00:39.202 INFO:teuthology.orchestra.run.smithi155.stdout: "ino+": 83446, 2023-11-16T17:00:39.203 INFO:teuthology.orchestra.run.smithi155.stdout: "ino-": 40445, 2023-11-16T17:00:39.203 INFO:teuthology.orchestra.run.smithi155.stdout: "dir": 1552, 2023-11-16T17:00:39.203 INFO:teuthology.orchestra.run.smithi155.stdout: "dir+": 3429, 2023-11-16T17:00:39.203 INFO:teuthology.orchestra.run.smithi155.stdout: "dir-": 1877, 2023-11-16T17:00:39.203 INFO:teuthology.orchestra.run.smithi155.stdout: "dn": 73009, 2023-11-16T17:00:39.203 INFO:teuthology.orchestra.run.smithi155.stdout: "dn+": 113913, 2023-11-16T17:00:39.203 INFO:teuthology.orchestra.run.smithi155.stdout: "dn-": 40904, 2023-11-16T17:00:39.203 INFO:teuthology.orchestra.run.smithi155.stdout: "cap": 5, 2023-11-16T17:00:39.204 INFO:teuthology.orchestra.run.smithi155.stdout: "cap+": 40149, 2023-11-16T17:00:39.204 INFO:teuthology.orchestra.run.smithi155.stdout: "cap-": 40144, 2023-11-16T17:00:39.204 INFO:teuthology.orchestra.run.smithi155.stdout: "rss": 316728, 2023-11-16T17:00:39.204 INFO:teuthology.orchestra.run.smithi155.stdout: "heap": 198964 2023-11-16T17:00:39.204 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.204 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_server": { 2023-11-16T17:00:39.204 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_client_request": 70293, 2023-11-16T17:00:39.204 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_server_request": 286, 2023-11-16T17:00:39.205 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_request": 69034, 2023-11-16T17:00:39.205 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_session": 48, 2023-11-16T17:00:39.205 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_peer_request": 576, 2023-11-16T17:00:39.205 INFO:teuthology.orchestra.run.smithi155.stdout: "req_create_latency": { 2023-11-16T17:00:39.205 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 20948, 2023-11-16T17:00:39.205 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 362.541619222, 2023-11-16T17:00:39.205 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.017306741 2023-11-16T17:00:39.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.206 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getattr_latency": { 2023-11-16T17:00:39.206 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 75, 2023-11-16T17:00:39.206 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.441081630, 2023-11-16T17:00:39.206 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.005881088 2023-11-16T17:00:39.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.206 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getfilelock_latency": { 2023-11-16T17:00:39.207 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.207 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.207 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.207 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.207 INFO:teuthology.orchestra.run.smithi155.stdout: "req_link_latency": { 2023-11-16T17:00:39.207 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.207 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.207 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.208 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.208 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookup_latency": { 2023-11-16T17:00:39.208 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 17176, 2023-11-16T17:00:39.208 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 27.364310490, 2023-11-16T17:00:39.208 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.001593171 2023-11-16T17:00:39.208 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.208 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookuphash_latency": { 2023-11-16T17:00:39.208 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.209 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.209 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.209 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.209 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupino_latency": { 2023-11-16T17:00:39.209 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.209 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.209 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.210 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.210 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupname_latency": { 2023-11-16T17:00:39.210 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.210 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.210 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.210 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.210 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupparent_latency": { 2023-11-16T17:00:39.210 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.210 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.211 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.211 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.211 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupsnap_latency": { 2023-11-16T17:00:39.211 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.211 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.211 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.211 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.212 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lssnap_latency": { 2023-11-16T17:00:39.212 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.212 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.212 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.212 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.212 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mkdir_latency": { 2023-11-16T17:00:39.212 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 119, 2023-11-16T17:00:39.212 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.116310998, 2023-11-16T17:00:39.213 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000977403 2023-11-16T17:00:39.213 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.213 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mknod_latency": { 2023-11-16T17:00:39.213 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.213 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.213 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.213 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.214 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mksnap_latency": { 2023-11-16T17:00:39.214 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.214 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.214 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.214 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.214 INFO:teuthology.orchestra.run.smithi155.stdout: "req_open_latency": { 2023-11-16T17:00:39.214 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.214 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.215 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.215 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.215 INFO:teuthology.orchestra.run.smithi155.stdout: "req_readdir_latency": { 2023-11-16T17:00:39.215 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2951, 2023-11-16T17:00:39.215 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 5.572873943, 2023-11-16T17:00:39.215 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.001888469 2023-11-16T17:00:39.215 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.216 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rename_latency": { 2023-11-16T17:00:39.216 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.216 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.216 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.216 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.216 INFO:teuthology.orchestra.run.smithi155.stdout: "req_renamesnap_latency": { 2023-11-16T17:00:39.216 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.217 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.217 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.217 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.217 INFO:teuthology.orchestra.run.smithi155.stdout: "req_snapdiff_latency": { 2023-11-16T17:00:39.217 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.217 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.217 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.217 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.218 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmdir_latency": { 2023-11-16T17:00:39.218 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 114, 2023-11-16T17:00:39.218 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.478857490, 2023-11-16T17:00:39.218 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.004200504 2023-11-16T17:00:39.218 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.218 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmsnap_latency": { 2023-11-16T17:00:39.218 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.218 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.219 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.219 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.219 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmxattr_latency": { 2023-11-16T17:00:39.219 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 3, 2023-11-16T17:00:39.219 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.002928340, 2023-11-16T17:00:39.219 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000976113 2023-11-16T17:00:39.219 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.220 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setattr_latency": { 2023-11-16T17:00:39.220 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 3, 2023-11-16T17:00:39.220 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.010393266, 2023-11-16T17:00:39.220 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.003464422 2023-11-16T17:00:39.220 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.220 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setdirlayout_latency": { 2023-11-16T17:00:39.220 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.220 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.221 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.221 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.221 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setfilelock_latency": { 2023-11-16T17:00:39.221 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 22, 2023-11-16T17:00:39.221 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 10.015247764, 2023-11-16T17:00:39.221 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.455238534 2023-11-16T17:00:39.221 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.221 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setlayout_latency": { 2023-11-16T17:00:39.222 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.222 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.222 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.222 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.222 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setxattr_latency": { 2023-11-16T17:00:39.222 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 9, 2023-11-16T17:00:39.222 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.014749696, 2023-11-16T17:00:39.223 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.001638855 2023-11-16T17:00:39.223 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.223 INFO:teuthology.orchestra.run.smithi155.stdout: "req_symlink_latency": { 2023-11-16T17:00:39.223 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.223 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.223 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.223 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.223 INFO:teuthology.orchestra.run.smithi155.stdout: "req_unlink_latency": { 2023-11-16T17:00:39.223 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 17530, 2023-11-16T17:00:39.224 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 52.910418823, 2023-11-16T17:00:39.224 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.003018278 2023-11-16T17:00:39.224 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.224 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_revoke_eviction": 0, 2023-11-16T17:00:39.224 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_acquisition_throttle": 0, 2023-11-16T17:00:39.224 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getvxattr_latency": { 2023-11-16T17:00:39.224 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.224 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.225 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.225 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:39.225 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.225 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_sessions": { 2023-11-16T17:00:39.225 INFO:teuthology.orchestra.run.smithi155.stdout: "session_count": 2, 2023-11-16T17:00:39.225 INFO:teuthology.orchestra.run.smithi155.stdout: "session_add": 5, 2023-11-16T17:00:39.225 INFO:teuthology.orchestra.run.smithi155.stdout: "session_remove": 3, 2023-11-16T17:00:39.225 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_open": 2, 2023-11-16T17:00:39.226 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_stale": 0, 2023-11-16T17:00:39.226 INFO:teuthology.orchestra.run.smithi155.stdout: "total_load": 16514, 2023-11-16T17:00:39.226 INFO:teuthology.orchestra.run.smithi155.stdout: "average_load": 8257, 2023-11-16T17:00:39.226 INFO:teuthology.orchestra.run.smithi155.stdout: "avg_session_uptime": 644 2023-11-16T17:00:39.226 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.226 INFO:teuthology.orchestra.run.smithi155.stdout: "mempool": { 2023-11-16T17:00:39.226 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_bytes": 322, 2023-11-16T17:00:39.226 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_items": 322, 2023-11-16T17:00:39.227 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:00:39.227 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:00:39.227 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:00:39.227 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:00:39.227 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:00:39.227 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:00:39.227 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:00:39.227 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:00:39.228 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:00:39.228 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:00:39.228 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:00:39.228 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:00:39.228 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:00:39.228 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_items": 0, 2023-11-16T17:00:39.228 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:00:39.228 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_items": 0, 2023-11-16T17:00:39.228 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:00:39.229 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:00:39.229 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:00:39.229 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:00:39.229 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:00:39.229 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:00:39.229 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:00:39.229 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_items": 0, 2023-11-16T17:00:39.230 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:00:39.230 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:00:39.230 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:00:39.230 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_items": 0, 2023-11-16T17:00:39.230 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_bytes": 0, 2023-11-16T17:00:39.230 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_items": 0, 2023-11-16T17:00:39.230 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:00:39.230 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:00:39.230 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:00:39.231 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:00:39.231 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_bytes": 1652298, 2023-11-16T17:00:39.231 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_items": 38243, 2023-11-16T17:00:39.231 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_bytes": 792, 2023-11-16T17:00:39.231 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_items": 9, 2023-11-16T17:00:39.231 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_bytes": 0, 2023-11-16T17:00:39.231 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_items": 0, 2023-11-16T17:00:39.231 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:00:39.232 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_items": 0, 2023-11-16T17:00:39.232 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:00:39.232 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_items": 0, 2023-11-16T17:00:39.232 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_bytes": 9944, 2023-11-16T17:00:39.232 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_items": 192, 2023-11-16T17:00:39.232 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:00:39.232 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:00:39.232 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_bytes": 0, 2023-11-16T17:00:39.232 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_items": 0, 2023-11-16T17:00:39.233 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_bytes": 150889667, 2023-11-16T17:00:39.233 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_items": 1390383, 2023-11-16T17:00:39.233 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_bytes": 0, 2023-11-16T17:00:39.233 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_items": 0, 2023-11-16T17:00:39.233 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_bytes": 0, 2023-11-16T17:00:39.233 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_items": 0 2023-11-16T17:00:39.233 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.233 INFO:teuthology.orchestra.run.smithi155.stdout: "objecter": { 2023-11-16T17:00:39.233 INFO:teuthology.orchestra.run.smithi155.stdout: "op_active": 0, 2023-11-16T17:00:39.234 INFO:teuthology.orchestra.run.smithi155.stdout: "op_laggy": 0, 2023-11-16T17:00:39.234 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send": 40617, 2023-11-16T17:00:39.234 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send_bytes": 392388661, 2023-11-16T17:00:39.234 INFO:teuthology.orchestra.run.smithi155.stdout: "op_resend": 28, 2023-11-16T17:00:39.234 INFO:teuthology.orchestra.run.smithi155.stdout: "op_reply": 40617, 2023-11-16T17:00:39.234 INFO:teuthology.orchestra.run.smithi155.stdout: "op_latency": { 2023-11-16T17:00:39.234 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 40617, 2023-11-16T17:00:39.234 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 475.087207722, 2023-11-16T17:00:39.235 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.011696757 2023-11-16T17:00:39.235 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.235 INFO:teuthology.orchestra.run.smithi155.stdout: "op_inflight": 0, 2023-11-16T17:00:39.235 INFO:teuthology.orchestra.run.smithi155.stdout: "oplen_avg": { 2023-11-16T17:00:39.235 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 40617, 2023-11-16T17:00:39.235 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 42902 2023-11-16T17:00:39.235 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.235 INFO:teuthology.orchestra.run.smithi155.stdout: "op": 40617, 2023-11-16T17:00:39.235 INFO:teuthology.orchestra.run.smithi155.stdout: "op_r": 1269, 2023-11-16T17:00:39.236 INFO:teuthology.orchestra.run.smithi155.stdout: "op_w": 39348, 2023-11-16T17:00:39.236 INFO:teuthology.orchestra.run.smithi155.stdout: "op_rmw": 0, 2023-11-16T17:00:39.236 INFO:teuthology.orchestra.run.smithi155.stdout: "op_pg": 0, 2023-11-16T17:00:39.236 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_stat": 1, 2023-11-16T17:00:39.236 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_create": 40, 2023-11-16T17:00:39.236 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_read": 246, 2023-11-16T17:00:39.236 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_write": 2599, 2023-11-16T17:00:39.236 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writefull": 130, 2023-11-16T17:00:39.237 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writesame": 0, 2023-11-16T17:00:39.237 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_append": 0, 2023-11-16T17:00:39.237 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_zero": 0, 2023-11-16T17:00:39.237 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_truncate": 5, 2023-11-16T17:00:39.237 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_delete": 35413, 2023-11-16T17:00:39.237 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_mapext": 0, 2023-11-16T17:00:39.237 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_sparse_read": 0, 2023-11-16T17:00:39.237 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_clonerange": 0, 2023-11-16T17:00:39.237 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_getxattr": 0, 2023-11-16T17:00:39.238 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_setxattr": 71, 2023-11-16T17:00:39.238 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_cmpxattr": 0, 2023-11-16T17:00:39.238 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_rmxattr": 0, 2023-11-16T17:00:39.238 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_resetxattrs": 0, 2023-11-16T17:00:39.238 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_call": 0, 2023-11-16T17:00:39.238 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_watch": 0, 2023-11-16T17:00:39.238 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_notify": 0, 2023-11-16T17:00:39.238 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_src_cmpxattr": 0, 2023-11-16T17:00:39.238 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls": 0, 2023-11-16T17:00:39.239 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls_filter": 0, 2023-11-16T17:00:39.239 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_other": 1, 2023-11-16T17:00:39.239 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_active": 0, 2023-11-16T17:00:39.239 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_send": 0, 2023-11-16T17:00:39.239 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_resend": 0, 2023-11-16T17:00:39.239 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_ping": 0, 2023-11-16T17:00:39.239 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_active": 0, 2023-11-16T17:00:39.239 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_send": 0, 2023-11-16T17:00:39.240 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_resend": 0, 2023-11-16T17:00:39.240 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_active": 0, 2023-11-16T17:00:39.240 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_send": 0, 2023-11-16T17:00:39.240 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_resend": 0, 2023-11-16T17:00:39.240 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_active": 0, 2023-11-16T17:00:39.240 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_send": 0, 2023-11-16T17:00:39.240 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_resend": 0, 2023-11-16T17:00:39.240 INFO:teuthology.orchestra.run.smithi155.stdout: "command_active": 0, 2023-11-16T17:00:39.241 INFO:teuthology.orchestra.run.smithi155.stdout: "command_send": 0, 2023-11-16T17:00:39.241 INFO:teuthology.orchestra.run.smithi155.stdout: "command_resend": 0, 2023-11-16T17:00:39.241 INFO:teuthology.orchestra.run.smithi155.stdout: "map_epoch": 104, 2023-11-16T17:00:39.241 INFO:teuthology.orchestra.run.smithi155.stdout: "map_full": 0, 2023-11-16T17:00:39.241 INFO:teuthology.orchestra.run.smithi155.stdout: "map_inc": 13, 2023-11-16T17:00:39.241 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_sessions": 12, 2023-11-16T17:00:39.241 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_open": 12, 2023-11-16T17:00:39.241 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_close": 0, 2023-11-16T17:00:39.241 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_laggy": 0, 2023-11-16T17:00:39.242 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_wr": 2296, 2023-11-16T17:00:39.242 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_rd": 2046, 2023-11-16T17:00:39.242 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_del": 54 2023-11-16T17:00:39.242 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.242 INFO:teuthology.orchestra.run.smithi155.stdout: "oft": { 2023-11-16T17:00:39.242 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_objs": 1, 2023-11-16T17:00:39.242 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_kv_pairs": 0, 2023-11-16T17:00:39.242 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_updates": 0, 2023-11-16T17:00:39.243 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_removes": 0 2023-11-16T17:00:39.243 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.243 INFO:teuthology.orchestra.run.smithi155.stdout: "osd-slow-ops": { 2023-11-16T17:00:39.243 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_ops_count": 69920 2023-11-16T17:00:39.243 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.243 INFO:teuthology.orchestra.run.smithi155.stdout: "purge_queue": { 2023-11-16T17:00:39.243 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops": 0, 2023-11-16T17:00:39.243 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops_high_water": 16, 2023-11-16T17:00:39.243 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing": 0, 2023-11-16T17:00:39.244 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_high_water": 15, 2023-11-16T17:00:39.244 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executed": 17635, 2023-11-16T17:00:39.244 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_item_in_journal": 0 2023-11-16T17:00:39.244 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.244 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:00:39.244 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2023-11-16T17:00:39.244 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2023-11-16T17:00:39.244 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2023-11-16T17:00:39.244 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 164766, 2023-11-16T17:00:39.245 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 139682492, 2023-11-16T17:00:39.245 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:39.245 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 164766, 2023-11-16T17:00:39.245 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2023-11-16T17:00:39.245 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2023-11-16T17:00:39.245 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 164766, 2023-11-16T17:00:39.245 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 139682492, 2023-11-16T17:00:39.245 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2023-11-16T17:00:39.245 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.246 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.246 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.246 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:39.246 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.246 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_bytes": { 2023-11-16T17:00:39.246 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2023-11-16T17:00:39.246 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2023-11-16T17:00:39.246 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2023-11-16T17:00:39.247 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2023-11-16T17:00:39.247 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2023-11-16T17:00:39.247 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:39.247 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2023-11-16T17:00:39.247 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 40617, 2023-11-16T17:00:39.247 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 394539996, 2023-11-16T17:00:39.247 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 40617, 2023-11-16T17:00:39.247 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 394539996, 2023-11-16T17:00:39.248 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2023-11-16T17:00:39.248 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.248 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.248 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.248 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:39.248 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.248 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_ops": { 2023-11-16T17:00:39.248 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2023-11-16T17:00:39.249 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 1024, 2023-11-16T17:00:39.249 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2023-11-16T17:00:39.249 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2023-11-16T17:00:39.249 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2023-11-16T17:00:39.249 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:39.249 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2023-11-16T17:00:39.249 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 40617, 2023-11-16T17:00:39.250 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 40617, 2023-11-16T17:00:39.250 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 40617, 2023-11-16T17:00:39.250 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 40617, 2023-11-16T17:00:39.250 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2023-11-16T17:00:39.250 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.250 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.250 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.250 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:39.251 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.251 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle": { 2023-11-16T17:00:39.251 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2023-11-16T17:00:39.251 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2023-11-16T17:00:39.251 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2023-11-16T17:00:39.251 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 17635, 2023-11-16T17:00:39.251 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 2167703, 2023-11-16T17:00:39.251 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:39.252 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 17635, 2023-11-16T17:00:39.252 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2023-11-16T17:00:39.252 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2023-11-16T17:00:39.252 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 246, 2023-11-16T17:00:39.252 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 2167703, 2023-11-16T17:00:39.252 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2023-11-16T17:00:39.252 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.252 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.253 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.253 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:39.253 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:39.253 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle-0x563ae6c420a0": { 2023-11-16T17:00:39.253 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2023-11-16T17:00:39.253 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2023-11-16T17:00:39.253 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2023-11-16T17:00:39.253 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 91184, 2023-11-16T17:00:39.253 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 377888614, 2023-11-16T17:00:39.254 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:39.254 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 91184, 2023-11-16T17:00:39.254 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2023-11-16T17:00:39.254 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2023-11-16T17:00:39.254 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 2353, 2023-11-16T17:00:39.254 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 377888614, 2023-11-16T17:00:39.254 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2023-11-16T17:00:39.254 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:39.255 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:39.255 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:39.255 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:39.255 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:39.255 INFO:teuthology.orchestra.run.smithi155.stdout:} 2023-11-16T17:00:39.255 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:39 smithi155 ceph-mon[131607]: pgmap v334: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 607 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T17:00:39.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:39 smithi002 ceph-mon[128712]: pgmap v334: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 607 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T17:00:39.503 INFO:tasks.check_counter:Daemon mds.i mds.exported=453 2023-11-16T17:00:39.504 INFO:tasks.check_counter:Daemon mds.i mds.imported=1077 2023-11-16T17:00:39.504 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=1718 2023-11-16T17:00:39.504 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=2866 2023-11-16T17:00:39.504 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=0 2023-11-16T17:00:39.504 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2023-11-16T17:00:39.504 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2023-11-16T17:00:39.504 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=132 2023-11-16T17:00:39.505 DEBUG:tasks.check_counter:Getting stats from l 2023-11-16T17:00:39.505 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e4a743ad7fd1422ebd6c4dc950a21a303279207 shell --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 -- ceph daemon mds.l perf dump 2023-11-16T17:00:39.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:39 smithi141 ceph-mon[131184]: pgmap v334: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 607 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-11-16T17:00:40.184 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/mon.c/config 2023-11-16T17:00:41.322 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2023-11-16T17:00:41.323 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-0": { 2023-11-16T17:00:41.323 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 233, 2023-11-16T17:00:41.323 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 223, 2023-11-16T17:00:41.323 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 1424768, 2023-11-16T17:00:41.323 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 127008, 2023-11-16T17:00:41.323 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 4, 2023-11-16T17:00:41.323 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 0, 2023-11-16T17:00:41.323 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.045157227, 2023-11-16T17:00:41.324 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.013776081, 2023-11-16T17:00:41.324 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.031472078, 2023-11-16T17:00:41.324 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.006360689, 2023-11-16T17:00:41.324 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:41.324 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 222, 2023-11-16T17:00:41.324 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.014874212, 2023-11-16T17:00:41.324 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000067000 2023-11-16T17:00:41.324 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:41.325 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:41.325 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:41.325 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:41.325 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:41.325 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:41.325 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 1424768, 2023-11-16T17:00:41.325 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 127008 2023-11-16T17:00:41.325 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:41.326 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-1": { 2023-11-16T17:00:41.326 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 6, 2023-11-16T17:00:41.326 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 172, 2023-11-16T17:00:41.326 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 3312, 2023-11-16T17:00:41.326 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 182096, 2023-11-16T17:00:41.326 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 5, 2023-11-16T17:00:41.326 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 3, 2023-11-16T17:00:41.327 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.011803267, 2023-11-16T17:00:41.327 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.007440895, 2023-11-16T17:00:41.327 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.002107920, 2023-11-16T17:00:41.327 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:41.327 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:41.327 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 170, 2023-11-16T17:00:41.327 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.020787742, 2023-11-16T17:00:41.328 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000122280 2023-11-16T17:00:41.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:41.328 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:41.328 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:41.328 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:41.328 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:41.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:41.328 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 3312, 2023-11-16T17:00:41.329 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 182096 2023-11-16T17:00:41.329 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:41.329 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-2": { 2023-11-16T17:00:41.329 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 0, 2023-11-16T17:00:41.329 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 2, 2023-11-16T17:00:41.329 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 0, 2023-11-16T17:00:41.329 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 0, 2023-11-16T17:00:41.330 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 2, 2023-11-16T17:00:41.330 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 1, 2023-11-16T17:00:41.330 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.000392401, 2023-11-16T17:00:41.330 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.000000000, 2023-11-16T17:00:41.330 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.000330564, 2023-11-16T17:00:41.330 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-11-16T17:00:41.330 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2023-11-16T17:00:41.330 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:41.331 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:41.331 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:41.331 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:41.331 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2023-11-16T17:00:41.331 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:41.331 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:41.331 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:41.332 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:41.332 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 0, 2023-11-16T17:00:41.332 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 0 2023-11-16T17:00:41.332 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:41.332 INFO:teuthology.orchestra.run.smithi155.stdout: "cct": { 2023-11-16T17:00:41.332 INFO:teuthology.orchestra.run.smithi155.stdout: "total_workers": 0, 2023-11-16T17:00:41.332 INFO:teuthology.orchestra.run.smithi155.stdout: "unhealthy_workers": 0 2023-11-16T17:00:41.332 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:41.333 INFO:teuthology.orchestra.run.smithi155.stdout: "mempool": { 2023-11-16T17:00:41.333 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_bytes": 0, 2023-11-16T17:00:41.333 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_items": 0, 2023-11-16T17:00:41.333 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_bytes": 0, 2023-11-16T17:00:41.333 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_items": 0, 2023-11-16T17:00:41.333 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_bytes": 0, 2023-11-16T17:00:41.333 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_items": 0, 2023-11-16T17:00:41.333 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_bytes": 0, 2023-11-16T17:00:41.334 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_items": 0, 2023-11-16T17:00:41.334 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_bytes": 0, 2023-11-16T17:00:41.334 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_items": 0, 2023-11-16T17:00:41.334 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_bytes": 0, 2023-11-16T17:00:41.334 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_items": 0, 2023-11-16T17:00:41.334 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_bytes": 0, 2023-11-16T17:00:41.334 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_items": 0, 2023-11-16T17:00:41.335 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_bytes": 0, 2023-11-16T17:00:41.335 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_items": 0, 2023-11-16T17:00:41.335 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_bytes": 0, 2023-11-16T17:00:41.335 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_items": 0, 2023-11-16T17:00:41.335 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_bytes": 0, 2023-11-16T17:00:41.335 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_items": 0, 2023-11-16T17:00:41.335 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_bytes": 0, 2023-11-16T17:00:41.335 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_items": 0, 2023-11-16T17:00:41.336 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_bytes": 0, 2023-11-16T17:00:41.336 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_items": 0, 2023-11-16T17:00:41.336 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_bytes": 0, 2023-11-16T17:00:41.336 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_items": 0, 2023-11-16T17:00:41.336 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_bytes": 0, 2023-11-16T17:00:41.336 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_items": 0, 2023-11-16T17:00:41.336 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_bytes": 0, 2023-11-16T17:00:41.337 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_items": 0, 2023-11-16T17:00:41.337 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_bytes": 0, 2023-11-16T17:00:41.337 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_items": 0, 2023-11-16T17:00:41.337 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_bytes": 0, 2023-11-16T17:00:41.337 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_items": 0, 2023-11-16T17:00:41.337 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_bytes": 0, 2023-11-16T17:00:41.337 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_items": 0, 2023-11-16T17:00:41.337 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_bytes": 148754, 2023-11-16T17:00:41.338 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_items": 31, 2023-11-16T17:00:41.338 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_bytes": 0, 2023-11-16T17:00:41.338 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_items": 0, 2023-11-16T17:00:41.338 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_bytes": 0, 2023-11-16T17:00:41.338 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_items": 0, 2023-11-16T17:00:41.338 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_bytes": 0, 2023-11-16T17:00:41.338 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_items": 0, 2023-11-16T17:00:41.338 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_bytes": 0, 2023-11-16T17:00:41.339 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_items": 0, 2023-11-16T17:00:41.339 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_bytes": 0, 2023-11-16T17:00:41.339 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_items": 0, 2023-11-16T17:00:41.339 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_bytes": 0, 2023-11-16T17:00:41.339 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_items": 0, 2023-11-16T17:00:41.339 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_bytes": 0, 2023-11-16T17:00:41.340 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_items": 0, 2023-11-16T17:00:41.340 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_bytes": 928, 2023-11-16T17:00:41.340 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_items": 2, 2023-11-16T17:00:41.340 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_bytes": 0, 2023-11-16T17:00:41.340 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_items": 0, 2023-11-16T17:00:41.340 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_bytes": 0, 2023-11-16T17:00:41.340 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_items": 0 2023-11-16T17:00:41.340 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2023-11-16T17:00:41.341 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-11-16T17:00:41.341 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2023-11-16T17:00:41.341 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2023-11-16T17:00:41.341 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2023-11-16T17:00:41.341 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 235, 2023-11-16T17:00:41.341 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 1391863, 2023-11-16T17:00:41.341 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2023-11-16T17:00:41.341 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 235, 2023-11-16T17:00:41.342 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2023-11-16T17:00:41.342 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2023-11-16T17:00:41.342 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 235, 2023-11-16T17:00:41.342 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 1391863, 2023-11-16T17:00:41.342 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2023-11-16T17:00:41.342 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2023-11-16T17:00:41.342 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2023-11-16T17:00:41.343 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2023-11-16T17:00:41.343 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:41.343 INFO:teuthology.orchestra.run.smithi155.stdout: } 2023-11-16T17:00:41.343 INFO:teuthology.orchestra.run.smithi155.stdout:} 2023-11-16T17:00:41.343 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:41 smithi155 ceph-mon[131607]: pgmap v335: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 607 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T17:00:41.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:41 smithi002 ceph-mon[128712]: pgmap v335: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 607 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T17:00:41.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:41 smithi141 ceph-mon[131184]: pgmap v335: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 607 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-11-16T17:00:41.637 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2023-11-16T17:00:41.637 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2023-11-16T17:00:41.637 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2023-11-16T17:00:41.637 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2023-11-16T17:00:41.638 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2023-11-16T17:00:41.638 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2023-11-16T17:00:41.638 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2023-11-16T17:00:41.638 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2023-11-16T17:00:41.638 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-11-16T17:00:41.649 INFO:tasks.kclient:Unmounting kernel clients... 2023-11-16T17:00:41.649 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T17:00:41.649 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-16T17:00:41.716 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2023-11-16T17:00:41.716 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-16T17:00:41.763 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T17:00:42.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.014+0000 7fc52b5ff700 1 -- 172.21.15.2:0/2342711458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc524105540 msgr2=0x7fc52410d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:42.015 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.014+0000 7fc52b5ff700 1 --2- 172.21.15.2:0/2342711458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc524105540 0x7fc52410d110 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc51c00b0d0 tx=0x7fc51c00b3e0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.014+0000 7fc52b5ff700 1 -- 172.21.15.2:0/2342711458 shutdown_connections 2023-11-16T17:00:42.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.014+0000 7fc52b5ff700 1 --2- 172.21.15.2:0/2342711458 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc524105540 0x7fc52410d110 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.014+0000 7fc52b5ff700 1 --2- 172.21.15.2:0/2342711458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc524100550 0x7fc524100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.014+0000 7fc52b5ff700 1 --2- 172.21.15.2:0/2342711458 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc524100ef0 0x7fc524104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.016 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.014+0000 7fc52b5ff700 1 -- 172.21.15.2:0/2342711458 >> 172.21.15.2:0/2342711458 conn(0x7fc524077020 msgr2=0x7fc524077420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:42.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.014+0000 7fc52b5ff700 1 -- 172.21.15.2:0/2342711458 shutdown_connections 2023-11-16T17:00:42.017 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.014+0000 7fc52b5ff700 1 -- 172.21.15.2:0/2342711458 wait complete. 2023-11-16T17:00:42.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.015+0000 7fc52b5ff700 1 Processor -- start 2023-11-16T17:00:42.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.015+0000 7fc52b5ff700 1 -- start start 2023-11-16T17:00:42.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.016+0000 7fc52b5ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc524100550 0x7fc524072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:42.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.016+0000 7fc52b5ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc524100ef0 0x7fc524073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:42.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.016+0000 7fc52b5ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc524105540 0x7fc5241a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:42.020 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.016+0000 7fc52b5ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc524073b60 con 0x7fc524100ef0 2023-11-16T17:00:42.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.016+0000 7fc52b5ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5241a4c10 con 0x7fc524105540 2023-11-16T17:00:42.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.016+0000 7fc52b5ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5241a4d50 con 0x7fc524100550 2023-11-16T17:00:42.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.016+0000 7fc528b9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc524100ef0 0x7fc524073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:42.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.016+0000 7fc52939b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc524100550 0x7fc524072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:42.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.016+0000 7fc52939b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc524100550 0x7fc524072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:48310/0 (socket says 172.21.15.2:48310) 2023-11-16T17:00:42.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.016+0000 7fc52939b700 1 -- 172.21.15.2:0/1595255653 learned_addr learned my addr 172.21.15.2:0/1595255653 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T17:00:42.021 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.016+0000 7fc529b9c700 1 --2- 172.21.15.2:0/1595255653 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc524105540 0x7fc5241a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:42.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.016+0000 7fc52939b700 1 -- 172.21.15.2:0/1595255653 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc524105540 msgr2=0x7fc5241a46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:42.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.016+0000 7fc52939b700 1 --2- 172.21.15.2:0/1595255653 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc524105540 0x7fc5241a46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.016+0000 7fc52939b700 1 -- 172.21.15.2:0/1595255653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc524100ef0 msgr2=0x7fc524073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:42.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.016+0000 7fc52939b700 1 --2- 172.21.15.2:0/1595255653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc524100ef0 0x7fc524073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.016+0000 7fc52939b700 1 -- 172.21.15.2:0/1595255653 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51c009d00 con 0x7fc524100550 2023-11-16T17:00:42.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.017+0000 7fc52939b700 1 --2- 172.21.15.2:0/1595255653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc524100550 0x7fc524072df0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc52000e9a0 tx=0x7fc52000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:42.022 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.017+0000 7fc51a7fc700 1 -- 172.21.15.2:0/1595255653 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc520009790 con 0x7fc524100550 2023-11-16T17:00:42.023 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.017+0000 7fc51a7fc700 1 -- 172.21.15.2:0/1595255653 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc520010430 con 0x7fc524100550 2023-11-16T17:00:42.023 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.017+0000 7fc51a7fc700 1 -- 172.21.15.2:0/1595255653 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc520018680 con 0x7fc524100550 2023-11-16T17:00:42.023 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.017+0000 7fc52b5ff700 1 -- 172.21.15.2:0/1595255653 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc524110360 con 0x7fc524100550 2023-11-16T17:00:42.023 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.017+0000 7fc52b5ff700 1 -- 172.21.15.2:0/1595255653 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5241101d0 con 0x7fc524100550 2023-11-16T17:00:42.023 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.018+0000 7fc52b5ff700 1 -- 172.21.15.2:0/1595255653 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5241104f0 con 0x7fc524100550 2023-11-16T17:00:42.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.020+0000 7fc51a7fc700 1 -- 172.21.15.2:0/1595255653 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fc5200105a0 con 0x7fc524100550 2023-11-16T17:00:42.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.021+0000 7fc51a7fc700 1 --2- 172.21.15.2:0/1595255653 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fc5100a12c0 0x7fc5100a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:42.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.022+0000 7fc51a7fc700 1 -- 172.21.15.2:0/1595255653 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9570+0+0 (secure 0 0 0) 0x7fc520014070 con 0x7fc524100550 2023-11-16T17:00:42.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.022+0000 7fc528b9a700 1 --2- 172.21.15.2:0/1595255653 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fc5100a12c0 0x7fc5100a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:42.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.022+0000 7fc528b9a700 1 --2- 172.21.15.2:0/1595255653 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fc5100a12c0 0x7fc5100a3770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc5240745e0 tx=0x7fc514009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:42.025 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.024+0000 7fc51a7fc700 1 -- 172.21.15.2:0/1595255653 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc5200962a0 con 0x7fc524100550 2023-11-16T17:00:42.233 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.232+0000 7fc52b5ff700 1 -- 172.21.15.2:0/1595255653 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc524067a60 con 0x7fc524100550 2023-11-16T17:00:42.235 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.234+0000 7fc51a7fc700 1 -- 172.21.15.2:0/1595255653 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+25260 (secure 0 0 0) 0x7fc5200998c0 con 0x7fc524100550 2023-11-16T17:00:42.235 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T17:00:42.236 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":104,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T17:00:10.980792+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:24.458452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:25.786005+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:29.313181+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2834229980":"2023-11-17T16:49:54.626188+0000","172.21.15.2:0/3820370584":"2023-11-17T16:49:54.626188+0000","172.21.15.2:6801/1793067160":"2023-11-17T16:49:54.626188+0000","172.21.15.2:6800/1793067160":"2023-11-17T16:49:54.626188+0000","172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3784433150":"2023-11-17T16:49:54.626188+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/1824419243":"2023-11-17T16:49:54.626188+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T17:00:42.237 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.237+0000 7fc52b5ff700 1 -- 172.21.15.2:0/1595255653 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fc5100a12c0 msgr2=0x7fc5100a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.237+0000 7fc52b5ff700 1 --2- 172.21.15.2:0/1595255653 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fc5100a12c0 0x7fc5100a3770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc5240745e0 tx=0x7fc514009260 comp rx=0 tx=0).stop 2023-11-16T17:00:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.237+0000 7fc52b5ff700 1 -- 172.21.15.2:0/1595255653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc524100550 msgr2=0x7fc524072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.237+0000 7fc52b5ff700 1 --2- 172.21.15.2:0/1595255653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc524100550 0x7fc524072df0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc52000e9a0 tx=0x7fc52000ecb0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.237+0000 7fc52b5ff700 1 -- 172.21.15.2:0/1595255653 shutdown_connections 2023-11-16T17:00:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.237+0000 7fc52b5ff700 1 --2- 172.21.15.2:0/1595255653 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fc5100a12c0 0x7fc5100a3770 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.237+0000 7fc52b5ff700 1 --2- 172.21.15.2:0/1595255653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc524100550 0x7fc524072df0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.237+0000 7fc52b5ff700 1 --2- 172.21.15.2:0/1595255653 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fc524105540 0x7fc5241a46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.237+0000 7fc52b5ff700 1 --2- 172.21.15.2:0/1595255653 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc524100ef0 0x7fc524073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.237+0000 7fc52b5ff700 1 -- 172.21.15.2:0/1595255653 >> 172.21.15.2:0/1595255653 conn(0x7fc524077020 msgr2=0x7fc5240fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:42.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.237+0000 7fc52b5ff700 1 -- 172.21.15.2:0/1595255653 shutdown_connections 2023-11-16T17:00:42.239 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.237+0000 7fc52b5ff700 1 -- 172.21.15.2:0/1595255653 wait complete. 2023-11-16T17:00:42.256 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-16T16:43:04.306584+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-16T16:47:24.458452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-16T16:47:25.786005+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 104, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-11-16T16:47:29.313181+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 104, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-11-16T17:00:42.257 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-16T17:00:42.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.465+0000 7f422d3ff700 1 -- 172.21.15.2:0/504217444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4228101390 msgr2=0x7f4228101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:42.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.465+0000 7f422d3ff700 1 --2- 172.21.15.2:0/504217444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4228101390 0x7f4228101760 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4218009990 tx=0x7f4218009ca0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.465+0000 7f422d3ff700 1 -- 172.21.15.2:0/504217444 shutdown_connections 2023-11-16T17:00:42.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.465+0000 7f422d3ff700 1 --2- 172.21.15.2:0/504217444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f422810a8b0 0x7f422810ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.465+0000 7f422d3ff700 1 --2- 172.21.15.2:0/504217444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4228101390 0x7f4228101760 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.465+0000 7f422d3ff700 1 --2- 172.21.15.2:0/504217444 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4228101ca0 0x7f422810a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.465+0000 7f422d3ff700 1 -- 172.21.15.2:0/504217444 >> 172.21.15.2:0/504217444 conn(0x7f42280770b0 msgr2=0x7f42280774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:42.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.465+0000 7f422d3ff700 1 -- 172.21.15.2:0/504217444 shutdown_connections 2023-11-16T17:00:42.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.465+0000 7f422d3ff700 1 -- 172.21.15.2:0/504217444 wait complete. 2023-11-16T17:00:42.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.466+0000 7f422d3ff700 1 Processor -- start 2023-11-16T17:00:42.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.466+0000 7f422d3ff700 1 -- start start 2023-11-16T17:00:42.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.467+0000 7f422d3ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4228101390 0x7f42281963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:42.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.467+0000 7f422d3ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4228101ca0 0x7f4228196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:42.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.467+0000 7f422d3ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f422810a8b0 0x7f422819ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:42.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.467+0000 7f422d3ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422819b3d0 con 0x7f4228101390 2023-11-16T17:00:42.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.467+0000 7f422d3ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422819b540 con 0x7f422810a8b0 2023-11-16T17:00:42.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.467+0000 7f422d3ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422819b700 con 0x7f4228101ca0 2023-11-16T17:00:42.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.467+0000 7f4226ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4228101390 0x7f42281963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:42.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.467+0000 7f4226ffd700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4228101390 0x7f42281963f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37030/0 (socket says 172.21.15.2:37030) 2023-11-16T17:00:42.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.467+0000 7f421ffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4228101ca0 0x7f4228196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:42.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.467+0000 7f4226ffd700 1 -- 172.21.15.2:0/3691020775 learned_addr learned my addr 172.21.15.2:0/3691020775 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T17:00:42.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.467+0000 7f42277fe700 1 --2- 172.21.15.2:0/3691020775 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f422810a8b0 0x7f422819ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:42.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.468+0000 7f4226ffd700 1 -- 172.21.15.2:0/3691020775 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4228101ca0 msgr2=0x7f4228196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:42.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.468+0000 7f4226ffd700 1 --2- 172.21.15.2:0/3691020775 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4228101ca0 0x7f4228196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.468+0000 7f4226ffd700 1 -- 172.21.15.2:0/3691020775 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f422810a8b0 msgr2=0x7f422819ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:42.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.468+0000 7f4226ffd700 1 --2- 172.21.15.2:0/3691020775 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f422810a8b0 0x7f422819ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.468+0000 7f4226ffd700 1 -- 172.21.15.2:0/3691020775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42180095f0 con 0x7f4228101390 2023-11-16T17:00:42.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.468+0000 7f4226ffd700 1 --2- 172.21.15.2:0/3691020775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4228101390 0x7f42281963f0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4218000c00 tx=0x7f4218004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:42.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.468+0000 7f4224ff9700 1 -- 172.21.15.2:0/3691020775 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421801d070 con 0x7f4228101390 2023-11-16T17:00:42.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.468+0000 7f4224ff9700 1 -- 172.21.15.2:0/3691020775 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4218021410 con 0x7f4228101390 2023-11-16T17:00:42.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.468+0000 7f422d3ff700 1 -- 172.21.15.2:0/3691020775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4228110330 con 0x7f4228101390 2023-11-16T17:00:42.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.468+0000 7f4224ff9700 1 -- 172.21.15.2:0/3691020775 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4218017650 con 0x7f4228101390 2023-11-16T17:00:42.472 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.468+0000 7f422d3ff700 1 -- 172.21.15.2:0/3691020775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42281101a0 con 0x7f4228101390 2023-11-16T17:00:42.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.469+0000 7f422d3ff700 1 -- 172.21.15.2:0/3691020775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42281104c0 con 0x7f4228101390 2023-11-16T17:00:42.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.471+0000 7f4224ff9700 1 -- 172.21.15.2:0/3691020775 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f4218021580 con 0x7f4228101390 2023-11-16T17:00:42.473 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.473+0000 7f4224ff9700 1 --2- 172.21.15.2:0/3691020775 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f42140a12c0 0x7f42140a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:42.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.473+0000 7f4224ff9700 1 -- 172.21.15.2:0/3691020775 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9570+0+0 (secure 0 0 0) 0x7f42180ce420 con 0x7f4228101390 2023-11-16T17:00:42.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.473+0000 7f421ffff700 1 --2- 172.21.15.2:0/3691020775 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f42140a12c0 0x7f42140a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:42.474 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.473+0000 7f421ffff700 1 --2- 172.21.15.2:0/3691020775 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f42140a12c0 0x7f42140a3770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4228197a00 tx=0x7f4210009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:42.477 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.477+0000 7f4224ff9700 1 -- 172.21.15.2:0/3691020775 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4218097440 con 0x7f4228101390 2023-11-16T17:00:42.686 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.685+0000 7f422d3ff700 1 -- 172.21.15.2:0/3691020775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4228197840 con 0x7f4228101390 2023-11-16T17:00:42.687 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.686+0000 7f4224ff9700 1 -- 172.21.15.2:0/3691020775 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4228197840 con 0x7f4228101390 2023-11-16T17:00:42.687 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2023-11-16T17:00:42.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.688+0000 7f422d3ff700 1 -- 172.21.15.2:0/3691020775 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f42140a12c0 msgr2=0x7f42140a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:42.689 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.688+0000 7f422d3ff700 1 --2- 172.21.15.2:0/3691020775 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f42140a12c0 0x7f42140a3770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4228197a00 tx=0x7f4210009260 comp rx=0 tx=0).stop 2023-11-16T17:00:42.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.688+0000 7f422d3ff700 1 -- 172.21.15.2:0/3691020775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4228101390 msgr2=0x7f42281963f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:42.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.688+0000 7f422d3ff700 1 --2- 172.21.15.2:0/3691020775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4228101390 0x7f42281963f0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4218000c00 tx=0x7f4218004790 comp rx=0 tx=0).stop 2023-11-16T17:00:42.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.689+0000 7f422d3ff700 1 -- 172.21.15.2:0/3691020775 shutdown_connections 2023-11-16T17:00:42.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.689+0000 7f422d3ff700 1 --2- 172.21.15.2:0/3691020775 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f42140a12c0 0x7f42140a3770 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.689+0000 7f422d3ff700 1 --2- 172.21.15.2:0/3691020775 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f422810a8b0 0x7f422819ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.690 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.689+0000 7f422d3ff700 1 --2- 172.21.15.2:0/3691020775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4228101390 0x7f42281963f0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.689+0000 7f422d3ff700 1 --2- 172.21.15.2:0/3691020775 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4228101ca0 0x7f4228196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.689+0000 7f422d3ff700 1 -- 172.21.15.2:0/3691020775 >> 172.21.15.2:0/3691020775 conn(0x7f42280770b0 msgr2=0x7f42280fbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:42.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.689+0000 7f422d3ff700 1 -- 172.21.15.2:0/3691020775 shutdown_connections 2023-11-16T17:00:42.691 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.689+0000 7f422d3ff700 1 -- 172.21.15.2:0/3691020775 wait complete. 2023-11-16T17:00:42.708 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-16T17:00:42.920 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.919+0000 7f65583ff700 1 -- 172.21.15.2:0/2934613571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f655010a8b0 msgr2=0x7f655010ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:42.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.919+0000 7f65583ff700 1 --2- 172.21.15.2:0/2934613571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f655010a8b0 0x7f655010ccd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f654800b0a0 tx=0x7f654800b3b0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.919+0000 7f65583ff700 1 -- 172.21.15.2:0/2934613571 shutdown_connections 2023-11-16T17:00:42.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.919+0000 7f65583ff700 1 --2- 172.21.15.2:0/2934613571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f655010a8b0 0x7f655010ccd0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.919+0000 7f65583ff700 1 --2- 172.21.15.2:0/2934613571 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6550101390 0x7f6550101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.919+0000 7f65583ff700 1 --2- 172.21.15.2:0/2934613571 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6550101ca0 0x7f655010a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.919+0000 7f65583ff700 1 -- 172.21.15.2:0/2934613571 >> 172.21.15.2:0/2934613571 conn(0x7f65500770b0 msgr2=0x7f65500774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:42.921 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.919+0000 7f65583ff700 1 -- 172.21.15.2:0/2934613571 shutdown_connections 2023-11-16T17:00:42.922 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.920+0000 7f65583ff700 1 -- 172.21.15.2:0/2934613571 wait complete. 2023-11-16T17:00:42.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.920+0000 7f65583ff700 1 Processor -- start 2023-11-16T17:00:42.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.921+0000 7f65583ff700 1 -- start start 2023-11-16T17:00:42.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.921+0000 7f65583ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6550101390 0x7f6550198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:42.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.921+0000 7f65583ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6550101ca0 0x7f6550198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:42.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.921+0000 7f65583ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f655010a8b0 0x7f655019c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:42.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.921+0000 7f65583ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655019c600 con 0x7f655010a8b0 2023-11-16T17:00:42.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.921+0000 7f65583ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655019c770 con 0x7f6550101ca0 2023-11-16T17:00:42.924 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.921+0000 7f65583ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655019c930 con 0x7f6550101390 2023-11-16T17:00:42.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.921+0000 7f655619b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6550101390 0x7f6550198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:42.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.921+0000 7f655699c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f655010a8b0 0x7f655019c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:42.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.921+0000 7f655619b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6550101390 0x7f6550198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.2:48362/0 (socket says 172.21.15.2:48362) 2023-11-16T17:00:42.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.921+0000 7f655619b700 1 -- 172.21.15.2:0/1072221810 learned_addr learned my addr 172.21.15.2:0/1072221810 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T17:00:42.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.922+0000 7f655599a700 1 --2- 172.21.15.2:0/1072221810 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6550101ca0 0x7f6550198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:42.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.922+0000 7f655699c700 1 -- 172.21.15.2:0/1072221810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6550101390 msgr2=0x7f6550198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:42.925 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.922+0000 7f655699c700 1 --2- 172.21.15.2:0/1072221810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6550101390 0x7f6550198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.922+0000 7f655699c700 1 -- 172.21.15.2:0/1072221810 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6550101ca0 msgr2=0x7f6550198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:42.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.922+0000 7f655699c700 1 --2- 172.21.15.2:0/1072221810 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6550101ca0 0x7f6550198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:42.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.922+0000 7f655699c700 1 -- 172.21.15.2:0/1072221810 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6540009520 con 0x7f655010a8b0 2023-11-16T17:00:42.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.922+0000 7f655619b700 1 --2- 172.21.15.2:0/1072221810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6550101390 0x7f6550198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T17:00:42.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.922+0000 7f655699c700 1 --2- 172.21.15.2:0/1072221810 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f655010a8b0 0x7f655019c000 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6548009fd0 tx=0x7f6548007930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:42.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.922+0000 7f65477fe700 1 -- 172.21.15.2:0/1072221810 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654800e040 con 0x7f655010a8b0 2023-11-16T17:00:42.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.922+0000 7f65477fe700 1 -- 172.21.15.2:0/1072221810 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f654801d470 con 0x7f655010a8b0 2023-11-16T17:00:42.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.922+0000 7f65583ff700 1 -- 172.21.15.2:0/1072221810 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6548009d00 con 0x7f655010a8b0 2023-11-16T17:00:42.926 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.922+0000 7f65477fe700 1 -- 172.21.15.2:0/1072221810 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65480052a0 con 0x7f655010a8b0 2023-11-16T17:00:42.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.923+0000 7f65583ff700 1 -- 172.21.15.2:0/1072221810 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6550110350 con 0x7f655010a8b0 2023-11-16T17:00:42.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.924+0000 7f65583ff700 1 -- 172.21.15.2:0/1072221810 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65501101c0 con 0x7f655010a8b0 2023-11-16T17:00:42.927 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.925+0000 7f65477fe700 1 -- 172.21.15.2:0/1072221810 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f654801d5e0 con 0x7f655010a8b0 2023-11-16T17:00:42.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.927+0000 7f65477fe700 1 --2- 172.21.15.2:0/1072221810 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f653c0a1340 0x7f653c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:42.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.927+0000 7f65477fe700 1 -- 172.21.15.2:0/1072221810 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9570+0+0 (secure 0 0 0) 0x7f65480ce570 con 0x7f655010a8b0 2023-11-16T17:00:42.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.927+0000 7f655619b700 1 --2- 172.21.15.2:0/1072221810 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f653c0a1340 0x7f653c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:42.928 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.928+0000 7f655619b700 1 --2- 172.21.15.2:0/1072221810 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f653c0a1340 0x7f653c0a37f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f654c011550 tx=0x7f654c009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:42.931 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:42.931+0000 7f65477fe700 1 -- 172.21.15.2:0/1072221810 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6548097590 con 0x7f655010a8b0 2023-11-16T17:00:43.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.139+0000 7f65583ff700 1 -- 172.21.15.2:0/1072221810 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f65501104e0 con 0x7f655010a8b0 2023-11-16T17:00:43.141 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.139+0000 7f65477fe700 1 -- 172.21.15.2:0/1072221810 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v104) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6548095e60 con 0x7f655010a8b0 2023-11-16T17:00:43.141 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T17:00:43.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.142+0000 7f65583ff700 1 -- 172.21.15.2:0/1072221810 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f653c0a1340 msgr2=0x7f653c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:43.143 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.142+0000 7f65583ff700 1 --2- 172.21.15.2:0/1072221810 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f653c0a1340 0x7f653c0a37f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f654c011550 tx=0x7f654c009260 comp rx=0 tx=0).stop 2023-11-16T17:00:43.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.142+0000 7f65583ff700 1 -- 172.21.15.2:0/1072221810 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f655010a8b0 msgr2=0x7f655019c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:43.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.142+0000 7f65583ff700 1 --2- 172.21.15.2:0/1072221810 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f655010a8b0 0x7f655019c000 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6548009fd0 tx=0x7f6548007930 comp rx=0 tx=0).stop 2023-11-16T17:00:43.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.142+0000 7f65583ff700 1 -- 172.21.15.2:0/1072221810 shutdown_connections 2023-11-16T17:00:43.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.142+0000 7f65583ff700 1 --2- 172.21.15.2:0/1072221810 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f653c0a1340 0x7f653c0a37f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.142+0000 7f65583ff700 1 --2- 172.21.15.2:0/1072221810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6550101390 0x7f6550198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.142+0000 7f65583ff700 1 --2- 172.21.15.2:0/1072221810 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f655010a8b0 0x7f655019c000 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.144 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.142+0000 7f65583ff700 1 --2- 172.21.15.2:0/1072221810 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6550101ca0 0x7f6550198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.142+0000 7f65583ff700 1 -- 172.21.15.2:0/1072221810 >> 172.21.15.2:0/1072221810 conn(0x7f65500770b0 msgr2=0x7f65500fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:43.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.142+0000 7f65583ff700 1 -- 172.21.15.2:0/1072221810 shutdown_connections 2023-11-16T17:00:43.145 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.142+0000 7f65583ff700 1 -- 172.21.15.2:0/1072221810 wait complete. 2023-11-16T17:00:43.162 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-16T17:00:43.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:43 smithi155 ceph-mon[131607]: pgmap v336: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 607 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-11-16T17:00:43.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:43 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1595255653' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T17:00:43.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:43 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3691020775' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T17:00:43.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.370+0000 7f6d0f7ff700 1 -- 172.21.15.2:0/3834243462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d08101400 msgr2=0x7f6d0810d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:43.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.370+0000 7f6d0f7ff700 1 --2- 172.21.15.2:0/3834243462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d08101400 0x7f6d0810d0a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6d0000b0a0 tx=0x7f6d0000b3b0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.371+0000 7f6d0f7ff700 1 -- 172.21.15.2:0/3834243462 shutdown_connections 2023-11-16T17:00:43.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.371+0000 7f6d0f7ff700 1 --2- 172.21.15.2:0/3834243462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d081063f0 0x7f6d081067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.371+0000 7f6d0f7ff700 1 --2- 172.21.15.2:0/3834243462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d08101400 0x7f6d0810d0a0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.371+0000 7f6d0f7ff700 1 --2- 172.21.15.2:0/3834243462 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d08100a50 0x7f6d08100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.371+0000 7f6d0f7ff700 1 -- 172.21.15.2:0/3834243462 >> 172.21.15.2:0/3834243462 conn(0x7f6d08077590 msgr2=0x7f6d08077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:43.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.371+0000 7f6d0f7ff700 1 -- 172.21.15.2:0/3834243462 shutdown_connections 2023-11-16T17:00:43.373 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.371+0000 7f6d0f7ff700 1 -- 172.21.15.2:0/3834243462 wait complete. 2023-11-16T17:00:43.375 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.372+0000 7f6d0f7ff700 1 Processor -- start 2023-11-16T17:00:43.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.372+0000 7f6d0f7ff700 1 -- start start 2023-11-16T17:00:43.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.372+0000 7f6d0f7ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d08100a50 0x7f6d08072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:43.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.372+0000 7f6d0f7ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d08101400 0x7f6d08073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:43.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.372+0000 7f6d0f7ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d081063f0 0x7f6d081a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:43.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.372+0000 7f6d0f7ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d08073b60 con 0x7f6d08101400 2023-11-16T17:00:43.376 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.372+0000 7f6d0f7ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d081a4be0 con 0x7f6d08100a50 2023-11-16T17:00:43.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.372+0000 7f6d0f7ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d081a4d20 con 0x7f6d081063f0 2023-11-16T17:00:43.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.373+0000 7f6d0cd9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d08101400 0x7f6d08073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:43.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.373+0000 7f6d0cd9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d08101400 0x7f6d08073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37066/0 (socket says 172.21.15.2:37066) 2023-11-16T17:00:43.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.373+0000 7f6d0cd9a700 1 -- 172.21.15.2:0/3127355426 learned_addr learned my addr 172.21.15.2:0/3127355426 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T17:00:43.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.373+0000 7f6d0d59b700 1 --2- 172.21.15.2:0/3127355426 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d08100a50 0x7f6d08072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:43.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.373+0000 7f6d0dd9c700 1 --2- 172.21.15.2:0/3127355426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d081063f0 0x7f6d081a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:43.377 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.373+0000 7f6d0cd9a700 1 -- 172.21.15.2:0/3127355426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d081063f0 msgr2=0x7f6d081a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:43.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.373+0000 7f6d0cd9a700 1 --2- 172.21.15.2:0/3127355426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d081063f0 0x7f6d081a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.373+0000 7f6d0cd9a700 1 -- 172.21.15.2:0/3127355426 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d08100a50 msgr2=0x7f6d08072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:43.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.373+0000 7f6d0cd9a700 1 --2- 172.21.15.2:0/3127355426 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d08100a50 0x7f6d08072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.373+0000 7f6d0cd9a700 1 -- 172.21.15.2:0/3127355426 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d00009d00 con 0x7f6d08101400 2023-11-16T17:00:43.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.373+0000 7f6d0cd9a700 1 --2- 172.21.15.2:0/3127355426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d08101400 0x7f6d08073330 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800e9a0 tx=0x7f6cf800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:43.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.374+0000 7f6cfe7fc700 1 -- 172.21.15.2:0/3127355426 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8009790 con 0x7f6d08101400 2023-11-16T17:00:43.378 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.374+0000 7f6d0f7ff700 1 -- 172.21.15.2:0/3127355426 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d08110410 con 0x7f6d08101400 2023-11-16T17:00:43.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.374+0000 7f6cfe7fc700 1 -- 172.21.15.2:0/3127355426 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6cf8010430 con 0x7f6d08101400 2023-11-16T17:00:43.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.374+0000 7f6cfe7fc700 1 -- 172.21.15.2:0/3127355426 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8018680 con 0x7f6d08101400 2023-11-16T17:00:43.379 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.374+0000 7f6d0f7ff700 1 -- 172.21.15.2:0/3127355426 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d08110280 con 0x7f6d08101400 2023-11-16T17:00:43.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.375+0000 7f6d0f7ff700 1 -- 172.21.15.2:0/3127355426 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d081105a0 con 0x7f6d08101400 2023-11-16T17:00:43.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.377+0000 7f6cfe7fc700 1 -- 172.21.15.2:0/3127355426 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f6cf80105a0 con 0x7f6d08101400 2023-11-16T17:00:43.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.378+0000 7f6cfe7fc700 1 --2- 172.21.15.2:0/3127355426 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f6cf40a12c0 0x7f6cf40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:43.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.378+0000 7f6cfe7fc700 1 -- 172.21.15.2:0/3127355426 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9570+0+0 (secure 0 0 0) 0x7f6cf8014070 con 0x7f6d08101400 2023-11-16T17:00:43.380 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.379+0000 7f6d0d59b700 1 --2- 172.21.15.2:0/3127355426 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f6cf40a12c0 0x7f6cf40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:43.381 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.379+0000 7f6d0d59b700 1 --2- 172.21.15.2:0/3127355426 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f6cf40a12c0 0x7f6cf40a3770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009be0 tx=0x7f6d04009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:43.382 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.380+0000 7f6cfe7fc700 1 -- 172.21.15.2:0/3127355426 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6cf80964b0 con 0x7f6d08101400 2023-11-16T17:00:43.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:43 smithi002 ceph-mon[128712]: pgmap v336: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 607 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-11-16T17:00:43.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:43 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1595255653' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T17:00:43.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:43 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3691020775' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T17:00:43.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:43 smithi141 ceph-mon[131184]: pgmap v336: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 607 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-11-16T17:00:43.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:43 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1595255653' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T17:00:43.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:43 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3691020775' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-16T17:00:43.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.591+0000 7f6d0f7ff700 1 -- 172.21.15.2:0/3127355426 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6d081a5750 con 0x7f6d08101400 2023-11-16T17:00:43.593 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.592+0000 7f6cfe7fc700 1 -- 172.21.15.2:0/3127355426 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v104) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6cf8026070 con 0x7f6d08101400 2023-11-16T17:00:43.594 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2023-11-16T17:00:43.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.594+0000 7f6d0f7ff700 1 -- 172.21.15.2:0/3127355426 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f6cf40a12c0 msgr2=0x7f6cf40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:43.596 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.594+0000 7f6d0f7ff700 1 --2- 172.21.15.2:0/3127355426 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f6cf40a12c0 0x7f6cf40a3770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009be0 tx=0x7f6d04009260 comp rx=0 tx=0).stop 2023-11-16T17:00:43.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.594+0000 7f6d0f7ff700 1 -- 172.21.15.2:0/3127355426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d08101400 msgr2=0x7f6d08073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:43.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.594+0000 7f6d0f7ff700 1 --2- 172.21.15.2:0/3127355426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d08101400 0x7f6d08073330 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800e9a0 tx=0x7f6cf800ecb0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.594+0000 7f6d0f7ff700 1 -- 172.21.15.2:0/3127355426 shutdown_connections 2023-11-16T17:00:43.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.594+0000 7f6d0f7ff700 1 --2- 172.21.15.2:0/3127355426 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f6cf40a12c0 0x7f6cf40a3770 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.594+0000 7f6d0f7ff700 1 --2- 172.21.15.2:0/3127355426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d081063f0 0x7f6d081a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.594+0000 7f6d0f7ff700 1 --2- 172.21.15.2:0/3127355426 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6d08100a50 0x7f6d08072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.597 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.594+0000 7f6d0f7ff700 1 --2- 172.21.15.2:0/3127355426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d08101400 0x7f6d08073330 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.598 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.594+0000 7f6d0f7ff700 1 -- 172.21.15.2:0/3127355426 >> 172.21.15.2:0/3127355426 conn(0x7f6d08077590 msgr2=0x7f6d080ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:43.598 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.594+0000 7f6d0f7ff700 1 -- 172.21.15.2:0/3127355426 shutdown_connections 2023-11-16T17:00:43.598 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.594+0000 7f6d0f7ff700 1 -- 172.21.15.2:0/3127355426 wait complete. 2023-11-16T17:00:43.615 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-11-16T17:00:43.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.822+0000 7f5e2bbff700 1 -- 172.21.15.2:0/2590252948 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e24105620 msgr2=0x7f5e2410d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:43.827 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.822+0000 7f5e2bbff700 1 --2- 172.21.15.2:0/2590252948 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e24105620 0x7f5e2410d1a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00b0d0 tx=0x7f5e1c00b3e0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.822+0000 7f5e2bbff700 1 -- 172.21.15.2:0/2590252948 shutdown_connections 2023-11-16T17:00:43.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.822+0000 7f5e2bbff700 1 --2- 172.21.15.2:0/2590252948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e24100cc0 0x7f5e24101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.822+0000 7f5e2bbff700 1 --2- 172.21.15.2:0/2590252948 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e24105620 0x7f5e2410d1a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.828 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.822+0000 7f5e2bbff700 1 --2- 172.21.15.2:0/2590252948 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e24101660 0x7f5e24104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.822+0000 7f5e2bbff700 1 -- 172.21.15.2:0/2590252948 >> 172.21.15.2:0/2590252948 conn(0x7f5e24077560 msgr2=0x7f5e24077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:43.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.822+0000 7f5e2bbff700 1 -- 172.21.15.2:0/2590252948 shutdown_connections 2023-11-16T17:00:43.829 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.822+0000 7f5e2bbff700 1 -- 172.21.15.2:0/2590252948 wait complete. 2023-11-16T17:00:43.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.823+0000 7f5e2bbff700 1 Processor -- start 2023-11-16T17:00:43.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.823+0000 7f5e2bbff700 1 -- start start 2023-11-16T17:00:43.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.824+0000 7f5e2bbff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e24100cc0 0x7f5e24072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:43.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.824+0000 7f5e2bbff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e24101660 0x7f5e24073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:43.830 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.824+0000 7f5e2bbff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e24105620 0x7f5e24073870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:43.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.824+0000 7f5e2bbff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e241a8fe0 con 0x7f5e24105620 2023-11-16T17:00:43.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.824+0000 7f5e2bbff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e241a9120 con 0x7f5e24101660 2023-11-16T17:00:43.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.824+0000 7f5e2bbff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e241a9260 con 0x7f5e24100cc0 2023-11-16T17:00:43.831 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.824+0000 7f5e2a19c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e24105620 0x7f5e24073870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:43.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.824+0000 7f5e2999b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e24100cc0 0x7f5e24072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:43.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.824+0000 7f5e2a19c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e24105620 0x7f5e24073870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37076/0 (socket says 172.21.15.2:37076) 2023-11-16T17:00:43.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.824+0000 7f5e2a19c700 1 -- 172.21.15.2:0/4245607052 learned_addr learned my addr 172.21.15.2:0/4245607052 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T17:00:43.832 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.824+0000 7f5e2919a700 1 --2- 172.21.15.2:0/4245607052 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e24101660 0x7f5e24073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:43.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.824+0000 7f5e2a19c700 1 -- 172.21.15.2:0/4245607052 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e24100cc0 msgr2=0x7f5e24072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:43.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.824+0000 7f5e2a19c700 1 --2- 172.21.15.2:0/4245607052 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e24100cc0 0x7f5e24072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.824+0000 7f5e2a19c700 1 -- 172.21.15.2:0/4245607052 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e24101660 msgr2=0x7f5e24073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:43.833 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.824+0000 7f5e2a19c700 1 --2- 172.21.15.2:0/4245607052 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e24101660 0x7f5e24073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:43.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.825+0000 7f5e2a19c700 1 -- 172.21.15.2:0/4245607052 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c009d00 con 0x7f5e24105620 2023-11-16T17:00:43.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.825+0000 7f5e2a19c700 1 --2- 172.21.15.2:0/4245607052 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e24105620 0x7f5e24073870 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c007820 tx=0x7f5e1c007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:43.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.825+0000 7f5e1affd700 1 -- 172.21.15.2:0/4245607052 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c00e040 con 0x7f5e24105620 2023-11-16T17:00:43.834 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.825+0000 7f5e1affd700 1 -- 172.21.15.2:0/4245607052 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e1c00bcb0 con 0x7f5e24105620 2023-11-16T17:00:43.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.825+0000 7f5e2bbff700 1 -- 172.21.15.2:0/4245607052 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e241103e0 con 0x7f5e24105620 2023-11-16T17:00:43.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.825+0000 7f5e1affd700 1 -- 172.21.15.2:0/4245607052 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0058a0 con 0x7f5e24105620 2023-11-16T17:00:43.835 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.825+0000 7f5e2bbff700 1 -- 172.21.15.2:0/4245607052 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e24110250 con 0x7f5e24105620 2023-11-16T17:00:43.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.826+0000 7f5e2bbff700 1 -- 172.21.15.2:0/4245607052 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e24110570 con 0x7f5e24105620 2023-11-16T17:00:43.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.828+0000 7f5e1affd700 1 -- 172.21.15.2:0/4245607052 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f5e1c004a50 con 0x7f5e24105620 2023-11-16T17:00:43.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.829+0000 7f5e1affd700 1 --2- 172.21.15.2:0/4245607052 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f5e100a1340 0x7f5e100a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:43.837 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.830+0000 7f5e1affd700 1 -- 172.21.15.2:0/4245607052 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9570+0+0 (secure 0 0 0) 0x7f5e1c0ce270 con 0x7f5e24105620 2023-11-16T17:00:43.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.830+0000 7f5e2999b700 1 --2- 172.21.15.2:0/4245607052 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f5e100a1340 0x7f5e100a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:43.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.830+0000 7f5e2999b700 1 --2- 172.21.15.2:0/4245607052 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f5e100a1340 0x7f5e100a37f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5e20009c60 tx=0x7f5e20009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:43.838 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:43.833+0000 7f5e1affd700 1 -- 172.21.15.2:0/4245607052 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5e1c097290 con 0x7f5e24105620 2023-11-16T17:00:44.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.046+0000 7f5e2bbff700 1 -- 172.21.15.2:0/4245607052 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5e241a9c80 con 0x7f5e24105620 2023-11-16T17:00:44.048 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.047+0000 7f5e1affd700 1 -- 172.21.15.2:0/4245607052 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v104) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5e1c09a8b0 con 0x7f5e24105620 2023-11-16T17:00:44.049 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2023-11-16T17:00:44.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.049+0000 7f5e2bbff700 1 -- 172.21.15.2:0/4245607052 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f5e100a1340 msgr2=0x7f5e100a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:44.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.049+0000 7f5e2bbff700 1 --2- 172.21.15.2:0/4245607052 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f5e100a1340 0x7f5e100a37f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5e20009c60 tx=0x7f5e20009260 comp rx=0 tx=0).stop 2023-11-16T17:00:44.051 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.049+0000 7f5e2bbff700 1 -- 172.21.15.2:0/4245607052 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e24105620 msgr2=0x7f5e24073870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:44.052 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.049+0000 7f5e2bbff700 1 --2- 172.21.15.2:0/4245607052 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e24105620 0x7f5e24073870 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c007820 tx=0x7f5e1c007850 comp rx=0 tx=0).stop 2023-11-16T17:00:44.052 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.049+0000 7f5e2bbff700 1 -- 172.21.15.2:0/4245607052 shutdown_connections 2023-11-16T17:00:44.052 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.049+0000 7f5e2bbff700 1 --2- 172.21.15.2:0/4245607052 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f5e100a1340 0x7f5e100a37f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:44.052 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.049+0000 7f5e2bbff700 1 --2- 172.21.15.2:0/4245607052 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e24100cc0 0x7f5e24072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:44.053 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.049+0000 7f5e2bbff700 1 --2- 172.21.15.2:0/4245607052 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e24105620 0x7f5e24073870 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:44.053 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.049+0000 7f5e2bbff700 1 --2- 172.21.15.2:0/4245607052 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f5e24101660 0x7f5e24073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:44.053 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.049+0000 7f5e2bbff700 1 -- 172.21.15.2:0/4245607052 >> 172.21.15.2:0/4245607052 conn(0x7f5e24077560 msgr2=0x7f5e24103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:44.053 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.049+0000 7f5e2bbff700 1 -- 172.21.15.2:0/4245607052 shutdown_connections 2023-11-16T17:00:44.054 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.049+0000 7f5e2bbff700 1 -- 172.21.15.2:0/4245607052 wait complete. 2023-11-16T17:00:44.070 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-16T17:00:44.277 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.275+0000 7fb9903ff700 1 -- 172.21.15.2:0/1453068945 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb9881055b0 msgr2=0x7fb98810d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:44.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.275+0000 7fb9903ff700 1 --2- 172.21.15.2:0/1453068945 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb9881055b0 0x7fb98810d0f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb980009fc0 tx=0x7fb98000b330 comp rx=0 tx=0).stop 2023-11-16T17:00:44.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.275+0000 7fb9903ff700 1 -- 172.21.15.2:0/1453068945 shutdown_connections 2023-11-16T17:00:44.278 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.275+0000 7fb9903ff700 1 --2- 172.21.15.2:0/1453068945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb988100c50 0x7fb988101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:44.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.275+0000 7fb9903ff700 1 --2- 172.21.15.2:0/1453068945 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb9881055b0 0x7fb98810d0f0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:44.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.275+0000 7fb9903ff700 1 --2- 172.21.15.2:0/1453068945 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9881015f0 0x7fb988104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:44.279 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.275+0000 7fb9903ff700 1 -- 172.21.15.2:0/1453068945 >> 172.21.15.2:0/1453068945 conn(0x7fb988077550 msgr2=0x7fb988077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:44.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.275+0000 7fb9903ff700 1 -- 172.21.15.2:0/1453068945 shutdown_connections 2023-11-16T17:00:44.280 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.275+0000 7fb9903ff700 1 -- 172.21.15.2:0/1453068945 wait complete. 2023-11-16T17:00:44.283 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.277+0000 7fb9903ff700 1 Processor -- start 2023-11-16T17:00:44.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.277+0000 7fb9903ff700 1 -- start start 2023-11-16T17:00:44.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.277+0000 7fb9903ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb988100c50 0x7fb988196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:44.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.277+0000 7fb9903ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9881015f0 0x7fb988196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:44.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.277+0000 7fb9903ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9881055b0 0x7fb98819adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:44.284 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.277+0000 7fb9903ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98819b400 con 0x7fb988100c50 2023-11-16T17:00:44.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.278+0000 7fb9903ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98819b570 con 0x7fb9881055b0 2023-11-16T17:00:44.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.278+0000 7fb9903ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98819b730 con 0x7fb9881015f0 2023-11-16T17:00:44.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.278+0000 7fb98e19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb988100c50 0x7fb988196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:44.285 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.278+0000 7fb98e19b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb988100c50 0x7fb988196420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46748/0 (socket says 172.21.15.2:46748) 2023-11-16T17:00:44.286 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.278+0000 7fb98e19b700 1 -- 172.21.15.2:0/1246174894 learned_addr learned my addr 172.21.15.2:0/1246174894 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T17:00:44.286 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.278+0000 7fb98e99c700 1 --2- 172.21.15.2:0/1246174894 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9881055b0 0x7fb98819adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:44.286 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.278+0000 7fb98d99a700 1 --2- 172.21.15.2:0/1246174894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9881015f0 0x7fb988196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:44.286 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.278+0000 7fb98e19b700 1 -- 172.21.15.2:0/1246174894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9881015f0 msgr2=0x7fb988196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:44.287 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.278+0000 7fb98e19b700 1 --2- 172.21.15.2:0/1246174894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9881015f0 0x7fb988196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:44.287 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.278+0000 7fb98e19b700 1 -- 172.21.15.2:0/1246174894 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9881055b0 msgr2=0x7fb98819adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:44.287 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.278+0000 7fb98e19b700 1 --2- 172.21.15.2:0/1246174894 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9881055b0 0x7fb98819adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:44.287 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.278+0000 7fb98e19b700 1 -- 172.21.15.2:0/1246174894 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb980009c40 con 0x7fb988100c50 2023-11-16T17:00:44.287 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.278+0000 7fb98e19b700 1 --2- 172.21.15.2:0/1246174894 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb988100c50 0x7fb988196420 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb98400e9c0 tx=0x7fb98400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:44.288 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.279+0000 7fb97f7fe700 1 -- 172.21.15.2:0/1246174894 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984004a30 con 0x7fb988100c50 2023-11-16T17:00:44.288 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.279+0000 7fb9903ff700 1 -- 172.21.15.2:0/1246174894 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb988110350 con 0x7fb988100c50 2023-11-16T17:00:44.288 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.279+0000 7fb97f7fe700 1 -- 172.21.15.2:0/1246174894 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb98400cc80 con 0x7fb988100c50 2023-11-16T17:00:44.288 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.279+0000 7fb97f7fe700 1 -- 172.21.15.2:0/1246174894 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9840054e0 con 0x7fb988100c50 2023-11-16T17:00:44.288 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.279+0000 7fb9903ff700 1 -- 172.21.15.2:0/1246174894 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9881101c0 con 0x7fb988100c50 2023-11-16T17:00:44.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.280+0000 7fb9903ff700 1 -- 172.21.15.2:0/1246174894 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9881104e0 con 0x7fb988100c50 2023-11-16T17:00:44.289 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.282+0000 7fb97f7fe700 1 -- 172.21.15.2:0/1246174894 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fb984004b90 con 0x7fb988100c50 2023-11-16T17:00:44.290 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.283+0000 7fb97f7fe700 1 --2- 172.21.15.2:0/1246174894 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fb9740a12c0 0x7fb9740a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:44.290 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.283+0000 7fb97f7fe700 1 -- 172.21.15.2:0/1246174894 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9570+0+0 (secure 0 0 0) 0x7fb984014070 con 0x7fb988100c50 2023-11-16T17:00:44.290 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.284+0000 7fb98d99a700 1 --2- 172.21.15.2:0/1246174894 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fb9740a12c0 0x7fb9740a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:44.291 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.284+0000 7fb98d99a700 1 --2- 172.21.15.2:0/1246174894 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fb9740a12c0 0x7fb9740a3770 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb988197a30 tx=0x7fb97800b410 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:44.291 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.287+0000 7fb97f7fe700 1 -- 172.21.15.2:0/1246174894 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb984096200 con 0x7fb988100c50 2023-11-16T17:00:44.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:44 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1072221810' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T17:00:44.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:44 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3127355426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T17:00:44.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:44 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/4245607052' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T17:00:44.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:44 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1072221810' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T17:00:44.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:44 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3127355426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T17:00:44.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:44 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/4245607052' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T17:00:44.513 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.512+0000 7fb9903ff700 1 -- 172.21.15.2:0/1246174894 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb9881a3670 con 0x7fb988100c50 2023-11-16T17:00:44.514 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.512+0000 7fb97f7fe700 1 -- 172.21.15.2:0/1246174894 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v104) v1 ==== 81+0+926 (secure 0 0 0) 0x7fb984099820 con 0x7fb988100c50 2023-11-16T17:00:44.514 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/2834229980 2023-11-17T16:49:54.626188+0000 2023-11-16T17:00:44.515 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3820370584 2023-11-17T16:49:54.626188+0000 2023-11-16T17:00:44.515 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/1793067160 2023-11-17T16:49:54.626188+0000 2023-11-16T17:00:44.515 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/1793067160 2023-11-17T16:49:54.626188+0000 2023-11-16T17:00:44.515 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/2137541200 2023-11-17T16:40:43.705118+0000 2023-11-16T17:00:44.516 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/2525877070 2023-11-17T16:40:43.705118+0000 2023-11-16T17:00:44.516 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/30571981 2023-11-17T16:40:16.538166+0000 2023-11-16T17:00:44.516 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3784433150 2023-11-17T16:49:54.626188+0000 2023-11-16T17:00:44.516 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/3359672074 2023-11-17T16:40:43.705118+0000 2023-11-16T17:00:44.516 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/500255408 2023-11-17T16:40:16.538166+0000 2023-11-16T17:00:44.517 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/981081063 2023-11-17T16:40:43.705118+0000 2023-11-16T17:00:44.517 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/3453602765 2023-11-17T16:40:16.538166+0000 2023-11-16T17:00:44.517 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/1824419243 2023-11-17T16:49:54.626188+0000 2023-11-16T17:00:44.517 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/3359672074 2023-11-17T16:40:43.705118+0000 2023-11-16T17:00:44.517 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/3453602765 2023-11-17T16:40:16.538166+0000 2023-11-16T17:00:44.518 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3818551797 2023-11-17T16:40:16.538166+0000 2023-11-16T17:00:44.518 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:44 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1072221810' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-11-16T17:00:44.519 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:44 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3127355426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-11-16T17:00:44.519 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:44 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/4245607052' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-11-16T17:00:44.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.514+0000 7fb9903ff700 1 -- 172.21.15.2:0/1246174894 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fb9740a12c0 msgr2=0x7fb9740a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:44.520 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.514+0000 7fb9903ff700 1 --2- 172.21.15.2:0/1246174894 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fb9740a12c0 0x7fb9740a3770 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb988197a30 tx=0x7fb97800b410 comp rx=0 tx=0).stop 2023-11-16T17:00:44.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.514+0000 7fb9903ff700 1 -- 172.21.15.2:0/1246174894 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb988100c50 msgr2=0x7fb988196420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:44.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.514+0000 7fb9903ff700 1 --2- 172.21.15.2:0/1246174894 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb988100c50 0x7fb988196420 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb98400e9c0 tx=0x7fb98400ecd0 comp rx=0 tx=0).stop 2023-11-16T17:00:44.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.514+0000 7fb9903ff700 1 -- 172.21.15.2:0/1246174894 shutdown_connections 2023-11-16T17:00:44.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.514+0000 7fb9903ff700 1 --2- 172.21.15.2:0/1246174894 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fb9740a12c0 0x7fb9740a3770 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:44.521 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.514+0000 7fb9903ff700 1 --2- 172.21.15.2:0/1246174894 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fb9881055b0 0x7fb98819adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:44.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.514+0000 7fb9903ff700 1 --2- 172.21.15.2:0/1246174894 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb988100c50 0x7fb988196420 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:44.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.514+0000 7fb9903ff700 1 --2- 172.21.15.2:0/1246174894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9881015f0 0x7fb988196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:44.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.514+0000 7fb9903ff700 1 -- 172.21.15.2:0/1246174894 >> 172.21.15.2:0/1246174894 conn(0x7fb988077550 msgr2=0x7fb9880fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:44.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.515+0000 7fb9903ff700 1 -- 172.21.15.2:0/1246174894 shutdown_connections 2023-11-16T17:00:44.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:44.515+0000 7fb9903ff700 1 -- 172.21.15.2:0/1246174894 wait complete. 2023-11-16T17:00:44.523 INFO:teuthology.orchestra.run.smithi002.stderr:listed 16 entries 2023-11-16T17:00:44.535 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-16T17:00:44.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T17:00:44.536 DEBUG:teuthology.orchestra.run.smithi002:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-16T17:00:44.650 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T17:00:44.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T17:00:44.651 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-16T17:00:44.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T17:00:44.742 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2023-11-16T17:00:44.895 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-16T17:00:44.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T17:00:44.897 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-16T17:00:44.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-16T17:00:44.969 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link delete ceph-brx 2023-11-16T17:00:45.074 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-11-16T17:00:45.086 INFO:tasks.cephadm:Teardown begin 2023-11-16T17:00:45.087 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-16T17:00:45.087 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T17:00:45.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:45 smithi002 ceph-mon[128712]: pgmap v337: 129 pgs: 129 active+clean; 607 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T17:00:45.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:45 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1246174894' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T17:00:45.235 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:45 smithi002 ceph-mon[128712]: osdmap e105: 12 total, 12 up, 12 in 2023-11-16T17:00:45.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.296+0000 7f41b8bff700 1 -- 172.21.15.2:0/2917002630 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41b4072e30 msgr2=0x7f41b4073200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:45.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.296+0000 7f41b8bff700 1 --2- 172.21.15.2:0/2917002630 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41b4072e30 0x7f41b4073200 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f41a4009990 tx=0x7f41a4009ca0 comp rx=0 tx=0).stop 2023-11-16T17:00:45.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.297+0000 7f41b8bff700 1 -- 172.21.15.2:0/2917002630 shutdown_connections 2023-11-16T17:00:45.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.297+0000 7f41b8bff700 1 --2- 172.21.15.2:0/2917002630 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b410cb80 0x7f41b410ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:45.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.297+0000 7f41b8bff700 1 --2- 172.21.15.2:0/2917002630 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41b4072e30 0x7f41b4073200 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:45.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.297+0000 7f41b8bff700 1 --2- 172.21.15.2:0/2917002630 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41b4073740 0x7f41b410c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:45.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.297+0000 7f41b8bff700 1 -- 172.21.15.2:0/2917002630 >> 172.21.15.2:0/2917002630 conn(0x7f41b4077000 msgr2=0x7f41b4077400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:45.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.297+0000 7f41b8bff700 1 -- 172.21.15.2:0/2917002630 shutdown_connections 2023-11-16T17:00:45.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.297+0000 7f41b8bff700 1 -- 172.21.15.2:0/2917002630 wait complete. 2023-11-16T17:00:45.303 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.299+0000 7f41b8bff700 1 Processor -- start 2023-11-16T17:00:45.303 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.299+0000 7f41b8bff700 1 -- start start 2023-11-16T17:00:45.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.299+0000 7f41b8bff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41b4072e30 0x7f41b4198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:45.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.300+0000 7f41b8bff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41b4073740 0x7f41b4198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:45.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.300+0000 7f41b8bff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b410cb80 0x7f41b419cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:45.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.300+0000 7f41b8bff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b419d550 con 0x7f41b4072e30 2023-11-16T17:00:45.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.300+0000 7f41b8bff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b419d6c0 con 0x7f41b410cb80 2023-11-16T17:00:45.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.300+0000 7f41b8bff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b419d880 con 0x7f41b4073740 2023-11-16T17:00:45.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.300+0000 7f41b259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41b4072e30 0x7f41b4198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:45.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.300+0000 7f41b259c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41b4072e30 0x7f41b4198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46760/0 (socket says 172.21.15.2:46760) 2023-11-16T17:00:45.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.300+0000 7f41b259c700 1 -- 172.21.15.2:0/3161703278 learned_addr learned my addr 172.21.15.2:0/3161703278 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T17:00:45.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.300+0000 7f41abfff700 1 --2- 172.21.15.2:0/3161703278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41b4073740 0x7f41b4198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:45.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.300+0000 7f41b2d9d700 1 --2- 172.21.15.2:0/3161703278 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b410cb80 0x7f41b419cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:45.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.300+0000 7f41b2d9d700 1 -- 172.21.15.2:0/3161703278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41b4073740 msgr2=0x7f41b4198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:45.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.300+0000 7f41b2d9d700 1 --2- 172.21.15.2:0/3161703278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41b4073740 0x7f41b4198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:45.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.300+0000 7f41b2d9d700 1 -- 172.21.15.2:0/3161703278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41b4072e30 msgr2=0x7f41b4198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:45.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.300+0000 7f41b2d9d700 1 --2- 172.21.15.2:0/3161703278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41b4072e30 0x7f41b4198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:45.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.300+0000 7f41b2d9d700 1 -- 172.21.15.2:0/3161703278 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a40095f0 con 0x7f41b410cb80 2023-11-16T17:00:45.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.300+0000 7f41b259c700 1 --2- 172.21.15.2:0/3161703278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41b4072e30 0x7f41b4198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T17:00:45.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.301+0000 7f41b2d9d700 1 --2- 172.21.15.2:0/3161703278 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b410cb80 0x7f41b419cf10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f41ac00efc0 tx=0x7f41ac00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:45.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.301+0000 7f41ab7fe700 1 -- 172.21.15.2:0/3161703278 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac018070 con 0x7f41b410cb80 2023-11-16T17:00:45.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.301+0000 7f41ab7fe700 1 -- 172.21.15.2:0/3161703278 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41ac00f040 con 0x7f41b410cb80 2023-11-16T17:00:45.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.301+0000 7f41ab7fe700 1 -- 172.21.15.2:0/3161703278 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ac00cd40 con 0x7f41b410cb80 2023-11-16T17:00:45.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.301+0000 7f41b8bff700 1 -- 172.21.15.2:0/3161703278 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b4112500 con 0x7f41b410cb80 2023-11-16T17:00:45.308 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.301+0000 7f41b8bff700 1 -- 172.21.15.2:0/3161703278 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b4112370 con 0x7f41b410cb80 2023-11-16T17:00:45.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.303+0000 7f41b8bff700 1 -- 172.21.15.2:0/3161703278 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b4112690 con 0x7f41b410cb80 2023-11-16T17:00:45.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.304+0000 7f41ab7fe700 1 -- 172.21.15.2:0/3161703278 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f41ac003a00 con 0x7f41b410cb80 2023-11-16T17:00:45.309 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.305+0000 7f41ab7fe700 1 --2- 172.21.15.2:0/3161703278 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f41a00a1340 0x7f41a00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:45.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.305+0000 7f41ab7fe700 1 -- 172.21.15.2:0/3161703278 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9482+0+0 (secure 0 0 0) 0x7f41ac0ccbb0 con 0x7f41b410cb80 2023-11-16T17:00:45.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.306+0000 7f41b259c700 1 --2- 172.21.15.2:0/3161703278 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f41a00a1340 0x7f41a00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:45.310 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.306+0000 7f41b259c700 1 --2- 172.21.15.2:0/3161703278 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f41a00a1340 0x7f41a00a37f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f41a4006010 tx=0x7f41a400b260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:45.311 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.309+0000 7f41ab7fe700 1 -- 172.21.15.2:0/3161703278 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f41ac095c30 con 0x7f41b410cb80 2023-11-16T17:00:45.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:45 smithi155 ceph-mon[131607]: pgmap v337: 129 pgs: 129 active+clean; 607 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T17:00:45.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:45 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1246174894' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T17:00:45.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:45 smithi155 ceph-mon[131607]: osdmap e105: 12 total, 12 up, 12 in 2023-11-16T17:00:45.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:45 smithi141 ceph-mon[131184]: pgmap v337: 129 pgs: 129 active+clean; 607 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-11-16T17:00:45.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:45 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1246174894' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-11-16T17:00:45.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:45 smithi141 ceph-mon[131184]: osdmap e105: 12 total, 12 up, 12 in 2023-11-16T17:00:45.563 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.562+0000 7f41b8bff700 1 -- 172.21.15.2:0/3161703278 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41b41074a0 con 0x7f41b410cb80 2023-11-16T17:00:45.565 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.563+0000 7f41ab7fe700 1 -- 172.21.15.2:0/3161703278 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+10805 (secure 0 0 0) 0x7f41ac099250 con 0x7f41b410cb80 2023-11-16T17:00:45.565 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T17:00:45.565 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/1610211225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":1610211225},{"type":"v1","addr":"172.21.15.2:6837","nonce":1610211225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14538,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2583261818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2583261818},{"type":"v1","addr":"172.21.15.2:6839","nonce":2583261818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14550,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3170991923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3170991923},{"type":"v1","addr":"172.21.15.2:6841","nonce":3170991923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":41,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:26.349176+0000","modified":"2023-11-16T16:59:19.864440+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24544,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.141:6835/3516869188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3516869188},{"type":"v1","addr":"172.21.15.141:6835","nonce":3516869188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/2527313191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":2527313191},{"type":"v1","addr":"172.21.15.155:6835","nonce":2527313191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.141:6837/2934891475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6836","nonce":2934891475},{"type":"v1","addr":"172.21.15.141:6837","nonce":2934891475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24544":{"gid":24544,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1158457009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1158457009},{"type":"v1","addr":"172.21.15.155:6839","nonce":1158457009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6839/3916358272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6838","nonce":3916358272},{"type":"v1","addr":"172.21.15.141:6839","nonce":3916358272}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_25543":{"gid":25543,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/333094585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":333094585},{"type":"v1","addr":"172.21.15.155:6837","nonce":333094585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T17:00:45.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.565+0000 7f41b8bff700 1 -- 172.21.15.2:0/3161703278 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f41a00a1340 msgr2=0x7f41a00a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:45.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.565+0000 7f41b8bff700 1 --2- 172.21.15.2:0/3161703278 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f41a00a1340 0x7f41a00a37f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f41a4006010 tx=0x7f41a400b260 comp rx=0 tx=0).stop 2023-11-16T17:00:45.568 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.565+0000 7f41b8bff700 1 -- 172.21.15.2:0/3161703278 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b410cb80 msgr2=0x7f41b419cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:45.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.565+0000 7f41b8bff700 1 --2- 172.21.15.2:0/3161703278 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b410cb80 0x7f41b419cf10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f41ac00efc0 tx=0x7f41ac00c3a0 comp rx=0 tx=0).stop 2023-11-16T17:00:45.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.566+0000 7f41b8bff700 1 -- 172.21.15.2:0/3161703278 shutdown_connections 2023-11-16T17:00:45.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.566+0000 7f41b8bff700 1 --2- 172.21.15.2:0/3161703278 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f41a00a1340 0x7f41a00a37f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:45.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.566+0000 7f41b8bff700 1 --2- 172.21.15.2:0/3161703278 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f41b410cb80 0x7f41b419cf10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:45.569 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.566+0000 7f41b8bff700 1 --2- 172.21.15.2:0/3161703278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41b4072e30 0x7f41b4198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:45.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.566+0000 7f41b8bff700 1 --2- 172.21.15.2:0/3161703278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41b4073740 0x7f41b4198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:45.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.566+0000 7f41b8bff700 1 -- 172.21.15.2:0/3161703278 >> 172.21.15.2:0/3161703278 conn(0x7f41b4077000 msgr2=0x7f41b41045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:45.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.566+0000 7f41b8bff700 1 -- 172.21.15.2:0/3161703278 shutdown_connections 2023-11-16T17:00:45.570 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.566+0000 7f41b8bff700 1 -- 172.21.15.2:0/3161703278 wait complete. 2023-11-16T17:00:45.571 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 41 2023-11-16T17:00:45.587 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-16T17:00:45.795 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.792+0000 7ff83afff700 1 -- 172.21.15.2:0/3548224210 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff834105540 msgr2=0x7ff83410d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:45.795 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.792+0000 7ff83afff700 1 --2- 172.21.15.2:0/3548224210 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff834105540 0x7ff83410d110 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff82c00b0d0 tx=0x7ff82c00b3e0 comp rx=0 tx=0).stop 2023-11-16T17:00:45.795 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.793+0000 7ff83afff700 1 -- 172.21.15.2:0/3548224210 shutdown_connections 2023-11-16T17:00:45.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.793+0000 7ff83afff700 1 --2- 172.21.15.2:0/3548224210 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff834100550 0x7ff834100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:45.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.793+0000 7ff83afff700 1 --2- 172.21.15.2:0/3548224210 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff834105540 0x7ff83410d110 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:45.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.793+0000 7ff83afff700 1 --2- 172.21.15.2:0/3548224210 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff834100ef0 0x7ff834104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:45.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.793+0000 7ff83afff700 1 -- 172.21.15.2:0/3548224210 >> 172.21.15.2:0/3548224210 conn(0x7ff834077020 msgr2=0x7ff834077420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:45.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.793+0000 7ff83afff700 1 -- 172.21.15.2:0/3548224210 shutdown_connections 2023-11-16T17:00:45.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.793+0000 7ff83afff700 1 -- 172.21.15.2:0/3548224210 wait complete. 2023-11-16T17:00:45.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.794+0000 7ff83afff700 1 Processor -- start 2023-11-16T17:00:45.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.794+0000 7ff83afff700 1 -- start start 2023-11-16T17:00:45.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.794+0000 7ff83afff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff834100550 0x7ff834072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:45.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.794+0000 7ff83afff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff834100ef0 0x7ff834073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:45.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.794+0000 7ff83afff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff834105540 0x7ff834073870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:45.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.794+0000 7ff83afff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8341a4910 con 0x7ff834105540 2023-11-16T17:00:45.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.794+0000 7ff83afff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8341a4a80 con 0x7ff834100ef0 2023-11-16T17:00:45.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.794+0000 7ff83afff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8341a4c40 con 0x7ff834100550 2023-11-16T17:00:45.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.795+0000 7ff83959c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff834105540 0x7ff834073870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:45.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.795+0000 7ff83959c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff834105540 0x7ff834073870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46766/0 (socket says 172.21.15.2:46766) 2023-11-16T17:00:45.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.795+0000 7ff838d9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff834100550 0x7ff834072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:45.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.795+0000 7ff83959c700 1 -- 172.21.15.2:0/4045076031 learned_addr learned my addr 172.21.15.2:0/4045076031 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T17:00:45.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.795+0000 7ff833fff700 1 --2- 172.21.15.2:0/4045076031 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff834100ef0 0x7ff834073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:45.803 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.795+0000 7ff83959c700 1 -- 172.21.15.2:0/4045076031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff834100550 msgr2=0x7ff834072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:45.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.795+0000 7ff83959c700 1 --2- 172.21.15.2:0/4045076031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff834100550 0x7ff834072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:45.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.795+0000 7ff83959c700 1 -- 172.21.15.2:0/4045076031 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff834100ef0 msgr2=0x7ff834073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:45.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.795+0000 7ff83959c700 1 --2- 172.21.15.2:0/4045076031 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff834100ef0 0x7ff834073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:45.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.795+0000 7ff83959c700 1 -- 172.21.15.2:0/4045076031 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c009d00 con 0x7ff834105540 2023-11-16T17:00:45.804 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.795+0000 7ff83959c700 1 --2- 172.21.15.2:0/4045076031 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff834105540 0x7ff834073870 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ff82c007820 tx=0x7ff82c007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:45.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.795+0000 7ff831ffb700 1 -- 172.21.15.2:0/4045076031 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82c00e040 con 0x7ff834105540 2023-11-16T17:00:45.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.795+0000 7ff831ffb700 1 -- 172.21.15.2:0/4045076031 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff82c00bcb0 con 0x7ff834105540 2023-11-16T17:00:45.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.796+0000 7ff83afff700 1 -- 172.21.15.2:0/4045076031 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff834110360 con 0x7ff834105540 2023-11-16T17:00:45.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.796+0000 7ff831ffb700 1 -- 172.21.15.2:0/4045076031 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82c0058a0 con 0x7ff834105540 2023-11-16T17:00:45.805 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.796+0000 7ff83afff700 1 -- 172.21.15.2:0/4045076031 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8341101d0 con 0x7ff834105540 2023-11-16T17:00:45.806 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.797+0000 7ff83afff700 1 -- 172.21.15.2:0/4045076031 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8341104f0 con 0x7ff834105540 2023-11-16T17:00:45.806 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.798+0000 7ff831ffb700 1 -- 172.21.15.2:0/4045076031 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7ff82c004a50 con 0x7ff834105540 2023-11-16T17:00:45.807 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.800+0000 7ff831ffb700 1 --2- 172.21.15.2:0/4045076031 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ff8240a1340 0x7ff8240a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:45.807 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.800+0000 7ff831ffb700 1 -- 172.21.15.2:0/4045076031 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9482+0+0 (secure 0 0 0) 0x7ff82c0ce210 con 0x7ff834105540 2023-11-16T17:00:45.807 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.800+0000 7ff838d9b700 1 --2- 172.21.15.2:0/4045076031 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ff8240a1340 0x7ff8240a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:45.808 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.801+0000 7ff838d9b700 1 --2- 172.21.15.2:0/4045076031 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ff8240a1340 0x7ff8240a37f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff828009c60 tx=0x7ff828009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:45.808 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:45.804+0000 7ff831ffb700 1 -- 172.21.15.2:0/4045076031 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff82c097290 con 0x7ff834105540 2023-11-16T17:00:46.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.063+0000 7ff83afff700 1 -- 172.21.15.2:0/4045076031 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff834067a60 con 0x7ff834105540 2023-11-16T17:00:46.065 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.064+0000 7ff831ffb700 1 -- 172.21.15.2:0/4045076031 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+10805 (secure 0 0 0) 0x7ff82c09a8b0 con 0x7ff834105540 2023-11-16T17:00:46.066 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T17:00:46.066 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/4133614271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4133614271},{"type":"v1","addr":"172.21.15.2:6835","nonce":4133614271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14481,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/1610211225","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":1610211225},{"type":"v1","addr":"172.21.15.2:6837","nonce":1610211225}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14538,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/2583261818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":2583261818},{"type":"v1","addr":"172.21.15.2:6839","nonce":2583261818}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14550,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/3170991923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":3170991923},{"type":"v1","addr":"172.21.15.2:6841","nonce":3170991923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/4280048431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":4280048431},{"type":"v1","addr":"172.21.15.155:6833","nonce":4280048431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24470,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.141:6833/3738610702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6832","nonce":3738610702},{"type":"v1","addr":"172.21.15.141:6833","nonce":3738610702}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":41,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-11-16T16:47:26.349176+0000","modified":"2023-11-16T16:59:19.864440+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24544,"mds_2":24542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.141:6835/3516869188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6834","nonce":3516869188},{"type":"v1","addr":"172.21.15.141:6835","nonce":3516869188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/2527313191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":2527313191},{"type":"v1","addr":"172.21.15.155:6835","nonce":2527313191}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24542":{"gid":24542,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.141:6837/2934891475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6836","nonce":2934891475},{"type":"v1","addr":"172.21.15.141:6837","nonce":2934891475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24544":{"gid":24544,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1158457009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1158457009},{"type":"v1","addr":"172.21.15.155:6839","nonce":1158457009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.141:6839/3916358272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6838","nonce":3916358272},{"type":"v1","addr":"172.21.15.141:6839","nonce":3916358272}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_25543":{"gid":25543,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/333094585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":333094585},{"type":"v1","addr":"172.21.15.155:6837","nonce":333094585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-16T17:00:46.068 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.067+0000 7ff83afff700 1 -- 172.21.15.2:0/4045076031 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ff8240a1340 msgr2=0x7ff8240a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:46.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.067+0000 7ff83afff700 1 --2- 172.21.15.2:0/4045076031 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ff8240a1340 0x7ff8240a37f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff828009c60 tx=0x7ff828009260 comp rx=0 tx=0).stop 2023-11-16T17:00:46.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.067+0000 7ff83afff700 1 -- 172.21.15.2:0/4045076031 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff834105540 msgr2=0x7ff834073870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:46.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.067+0000 7ff83afff700 1 --2- 172.21.15.2:0/4045076031 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff834105540 0x7ff834073870 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ff82c007820 tx=0x7ff82c007850 comp rx=0 tx=0).stop 2023-11-16T17:00:46.069 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.067+0000 7ff83afff700 1 -- 172.21.15.2:0/4045076031 shutdown_connections 2023-11-16T17:00:46.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.067+0000 7ff83afff700 1 --2- 172.21.15.2:0/4045076031 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7ff8240a1340 0x7ff8240a37f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.067+0000 7ff83afff700 1 --2- 172.21.15.2:0/4045076031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff834100550 0x7ff834072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.067+0000 7ff83afff700 1 --2- 172.21.15.2:0/4045076031 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff834105540 0x7ff834073870 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.067+0000 7ff83afff700 1 --2- 172.21.15.2:0/4045076031 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7ff834100ef0 0x7ff834073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.070 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.067+0000 7ff83afff700 1 -- 172.21.15.2:0/4045076031 >> 172.21.15.2:0/4045076031 conn(0x7ff834077020 msgr2=0x7ff8340fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:46.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.067+0000 7ff83afff700 1 -- 172.21.15.2:0/4045076031 shutdown_connections 2023-11-16T17:00:46.071 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.067+0000 7ff83afff700 1 -- 172.21.15.2:0/4045076031 wait complete. 2023-11-16T17:00:46.071 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 41 2023-11-16T17:00:46.088 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-16T17:00:46.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.293+0000 7f4a5a7ff700 1 -- 172.21.15.2:0/4164258459 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a54100550 msgr2=0x7f4a54100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:46.295 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.293+0000 7f4a5a7ff700 1 --2- 172.21.15.2:0/4164258459 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a54100550 0x7f4a54100920 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4a440099c0 tx=0x7f4a44009cd0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.293+0000 7f4a5a7ff700 1 -- 172.21.15.2:0/4164258459 shutdown_connections 2023-11-16T17:00:46.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.293+0000 7f4a5a7ff700 1 --2- 172.21.15.2:0/4164258459 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a54105540 0x7f4a5410d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.293+0000 7f4a5a7ff700 1 --2- 172.21.15.2:0/4164258459 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a54100550 0x7f4a54100920 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.296 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.293+0000 7f4a5a7ff700 1 --2- 172.21.15.2:0/4164258459 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a54100ef0 0x7f4a54104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.293+0000 7f4a5a7ff700 1 -- 172.21.15.2:0/4164258459 >> 172.21.15.2:0/4164258459 conn(0x7f4a54077020 msgr2=0x7f4a54077420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:46.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.293+0000 7f4a5a7ff700 1 -- 172.21.15.2:0/4164258459 shutdown_connections 2023-11-16T17:00:46.297 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.293+0000 7f4a5a7ff700 1 -- 172.21.15.2:0/4164258459 wait complete. 2023-11-16T17:00:46.299 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.294+0000 7f4a5a7ff700 1 Processor -- start 2023-11-16T17:00:46.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.294+0000 7f4a5a7ff700 1 -- start start 2023-11-16T17:00:46.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.295+0000 7f4a5a7ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a54100550 0x7f4a54072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:46.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.295+0000 7f4a5a7ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a54100ef0 0x7f4a54073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:46.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.295+0000 7f4a5a7ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a54105540 0x7f4a541a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:46.300 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.295+0000 7f4a5a7ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a54073b60 con 0x7f4a54100550 2023-11-16T17:00:46.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.295+0000 7f4a5a7ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a541a4a90 con 0x7f4a54100ef0 2023-11-16T17:00:46.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.295+0000 7f4a5a7ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a541a4bd0 con 0x7f4a54105540 2023-11-16T17:00:46.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.295+0000 7f4a53fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a54100550 0x7f4a54072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:46.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.295+0000 7f4a53fff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a54100550 0x7f4a54072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46780/0 (socket says 172.21.15.2:46780) 2023-11-16T17:00:46.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.295+0000 7f4a53fff700 1 -- 172.21.15.2:0/653756345 learned_addr learned my addr 172.21.15.2:0/653756345 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T17:00:46.301 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.295+0000 7f4a537fe700 1 --2- 172.21.15.2:0/653756345 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a54100ef0 0x7f4a54073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:46.302 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.295+0000 7f4a58d9c700 1 --2- 172.21.15.2:0/653756345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a54105540 0x7f4a541a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:46.302 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.296+0000 7f4a53fff700 1 -- 172.21.15.2:0/653756345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a54105540 msgr2=0x7f4a541a4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:46.302 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.296+0000 7f4a53fff700 1 --2- 172.21.15.2:0/653756345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a54105540 0x7f4a541a4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.302 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.296+0000 7f4a53fff700 1 -- 172.21.15.2:0/653756345 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a54100ef0 msgr2=0x7f4a54073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:46.302 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.296+0000 7f4a53fff700 1 --2- 172.21.15.2:0/653756345 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a54100ef0 0x7f4a54073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.302 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.296+0000 7f4a53fff700 1 -- 172.21.15.2:0/653756345 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a440095f0 con 0x7f4a54100550 2023-11-16T17:00:46.303 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.296+0000 7f4a537fe700 1 --2- 172.21.15.2:0/653756345 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a54100ef0 0x7f4a54073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-16T17:00:46.303 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.296+0000 7f4a53fff700 1 --2- 172.21.15.2:0/653756345 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a54100550 0x7f4a54072df0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f4a4400b580 tx=0x7f4a4400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:46.303 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.296+0000 7f4a517fa700 1 -- 172.21.15.2:0/653756345 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4401d070 con 0x7f4a54100550 2023-11-16T17:00:46.303 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.296+0000 7f4a5a7ff700 1 -- 172.21.15.2:0/653756345 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a54110350 con 0x7f4a54100550 2023-11-16T17:00:46.303 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.296+0000 7f4a517fa700 1 -- 172.21.15.2:0/653756345 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a44005050 con 0x7f4a54100550 2023-11-16T17:00:46.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.296+0000 7f4a517fa700 1 -- 172.21.15.2:0/653756345 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a44004a40 con 0x7f4a54100550 2023-11-16T17:00:46.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.296+0000 7f4a5a7ff700 1 -- 172.21.15.2:0/653756345 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a541101c0 con 0x7f4a54100550 2023-11-16T17:00:46.304 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.297+0000 7f4a5a7ff700 1 -- 172.21.15.2:0/653756345 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a541104e0 con 0x7f4a54100550 2023-11-16T17:00:46.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.299+0000 7f4a517fa700 1 -- 172.21.15.2:0/653756345 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f4a44005430 con 0x7f4a54100550 2023-11-16T17:00:46.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.301+0000 7f4a517fa700 1 --2- 172.21.15.2:0/653756345 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f4a3c0a1340 0x7f4a3c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:46.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.302+0000 7f4a517fa700 1 -- 172.21.15.2:0/653756345 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9482+0+0 (secure 0 0 0) 0x7f4a440cde70 con 0x7f4a54100550 2023-11-16T17:00:46.305 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.302+0000 7f4a537fe700 1 --2- 172.21.15.2:0/653756345 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f4a3c0a1340 0x7f4a3c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:46.306 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.302+0000 7f4a537fe700 1 --2- 172.21.15.2:0/653756345 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f4a3c0a1340 0x7f4a3c0a37f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4a540745e0 tx=0x7f4a4800b410 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:46.307 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.305+0000 7f4a517fa700 1 -- 172.21.15.2:0/653756345 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4a44096ef0 con 0x7f4a54100550 2023-11-16T17:00:46.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:46 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3161703278' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T17:00:46.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:46 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/4045076031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T17:00:46.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:46 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3161703278' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T17:00:46.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:46 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/4045076031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T17:00:46.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:46 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3161703278' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T17:00:46.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:46 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/4045076031' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-11-16T17:00:46.522 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.521+0000 7f4a5a7ff700 1 -- 172.21.15.2:0/653756345 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a54067a60 con 0x7f4a54100550 2023-11-16T17:00:46.524 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.522+0000 7f4a517fa700 1 -- 172.21.15.2:0/653756345 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+25125 (secure 0 0 0) 0x7f4a4409a510 con 0x7f4a54100550 2023-11-16T17:00:46.525 INFO:teuthology.orchestra.run.smithi002.stdout: 2023-11-16T17:00:46.525 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":105,"fsid":"875fcd24-849e-11ee-8dbe-212e2dc638e7","created":"2023-11-16T16:39:43.029249+0000","modified":"2023-11-16T17:00:44.723543+0000","last_up_change":"2023-11-16T16:46:36.046623+0000","last_in_change":"2023-11-16T16:46:18.520667+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-16T16:43:04.306584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-16T16:47:24.458452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-16T16:47:25.786005+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-11-16T16:47:29.313181+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde47f9-e428-42b2-a86a-d1a39a7f8084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6803","nonce":3417839294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6805","nonce":3417839294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6809","nonce":3417839294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3417839294},{"type":"v1","addr":"172.21.15.2:6807","nonce":3417839294}]},"public_addr":"172.21.15.2:6803/3417839294","cluster_addr":"172.21.15.2:6805/3417839294","heartbeat_back_addr":"172.21.15.2:6809/3417839294","heartbeat_front_addr":"172.21.15.2:6807/3417839294","state":["exists","up"]},{"osd":1,"uuid":"fdef1dd6-b1c9-484c-9942-395e0213aa49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6801","nonce":1040978026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6803","nonce":1040978026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6807","nonce":1040978026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":1040978026},{"type":"v1","addr":"172.21.15.141:6805","nonce":1040978026}]},"public_addr":"172.21.15.141:6801/1040978026","cluster_addr":"172.21.15.141:6803/1040978026","heartbeat_back_addr":"172.21.15.141:6807/1040978026","heartbeat_front_addr":"172.21.15.141:6805/1040978026","state":["exists","up"]},{"osd":2,"uuid":"1ece6416-c9f0-4a49-9eaf-392ea9a4f0ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6801","nonce":844890823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6803","nonce":844890823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6807","nonce":844890823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":844890823},{"type":"v1","addr":"172.21.15.155:6805","nonce":844890823}]},"public_addr":"172.21.15.155:6801/844890823","cluster_addr":"172.21.15.155:6803/844890823","heartbeat_back_addr":"172.21.15.155:6807/844890823","heartbeat_front_addr":"172.21.15.155:6805/844890823","state":["exists","up"]},{"osd":3,"uuid":"374aea7d-57ed-49b1-921a-c78d31478c37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6811","nonce":2757259480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6813","nonce":2757259480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6817","nonce":2757259480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2757259480},{"type":"v1","addr":"172.21.15.2:6815","nonce":2757259480}]},"public_addr":"172.21.15.2:6811/2757259480","cluster_addr":"172.21.15.2:6813/2757259480","heartbeat_back_addr":"172.21.15.2:6817/2757259480","heartbeat_front_addr":"172.21.15.2:6815/2757259480","state":["exists","up"]},{"osd":4,"uuid":"4a4b549e-4eb4-4f67-829c-c37004403b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6809","nonce":1025476846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6811","nonce":1025476846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6815","nonce":1025476846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":1025476846},{"type":"v1","addr":"172.21.15.141:6813","nonce":1025476846}]},"public_addr":"172.21.15.141:6809/1025476846","cluster_addr":"172.21.15.141:6811/1025476846","heartbeat_back_addr":"172.21.15.141:6815/1025476846","heartbeat_front_addr":"172.21.15.141:6813/1025476846","state":["exists","up"]},{"osd":5,"uuid":"35c42f32-a4f7-49f4-ad1b-ca5ac8d4eafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6809","nonce":2988032938}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6811","nonce":2988032938}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6815","nonce":2988032938}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2988032938},{"type":"v1","addr":"172.21.15.155:6813","nonce":2988032938}]},"public_addr":"172.21.15.155:6809/2988032938","cluster_addr":"172.21.15.155:6811/2988032938","heartbeat_back_addr":"172.21.15.155:6815/2988032938","heartbeat_front_addr":"172.21.15.155:6813/2988032938","state":["exists","up"]},{"osd":6,"uuid":"e574f450-ac2d-445f-bcef-61ccdc6f6985","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6819","nonce":2881363640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6821","nonce":2881363640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6825","nonce":2881363640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2881363640},{"type":"v1","addr":"172.21.15.2:6823","nonce":2881363640}]},"public_addr":"172.21.15.2:6819/2881363640","cluster_addr":"172.21.15.2:6821/2881363640","heartbeat_back_addr":"172.21.15.2:6825/2881363640","heartbeat_front_addr":"172.21.15.2:6823/2881363640","state":["exists","up"]},{"osd":7,"uuid":"7f09b3c4-0d0f-4c6c-92e1-fd25e9726a46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6817","nonce":372138580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6819","nonce":372138580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6823","nonce":372138580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":372138580},{"type":"v1","addr":"172.21.15.141:6821","nonce":372138580}]},"public_addr":"172.21.15.141:6817/372138580","cluster_addr":"172.21.15.141:6819/372138580","heartbeat_back_addr":"172.21.15.141:6823/372138580","heartbeat_front_addr":"172.21.15.141:6821/372138580","state":["exists","up"]},{"osd":8,"uuid":"fd6d5d76-9af2-49f8-a769-3ef0fb0599dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6817","nonce":3808747539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6819","nonce":3808747539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6823","nonce":3808747539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3808747539},{"type":"v1","addr":"172.21.15.155:6821","nonce":3808747539}]},"public_addr":"172.21.15.155:6817/3808747539","cluster_addr":"172.21.15.155:6819/3808747539","heartbeat_back_addr":"172.21.15.155:6823/3808747539","heartbeat_front_addr":"172.21.15.155:6821/3808747539","state":["exists","up"]},{"osd":9,"uuid":"0a2692c3-2f49-4706-9a2b-0cb85882387d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6827","nonce":3506759823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6829","nonce":3506759823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6833","nonce":3506759823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":3506759823},{"type":"v1","addr":"172.21.15.2:6831","nonce":3506759823}]},"public_addr":"172.21.15.2:6827/3506759823","cluster_addr":"172.21.15.2:6829/3506759823","heartbeat_back_addr":"172.21.15.2:6833/3506759823","heartbeat_front_addr":"172.21.15.2:6831/3506759823","state":["exists","up"]},{"osd":10,"uuid":"4c9f53c3-c1bb-45ce-a05e-94d49b0c654f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6825","nonce":1084001742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6827","nonce":1084001742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6831","nonce":1084001742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":1084001742},{"type":"v1","addr":"172.21.15.141:6829","nonce":1084001742}]},"public_addr":"172.21.15.141:6825/1084001742","cluster_addr":"172.21.15.141:6827/1084001742","heartbeat_back_addr":"172.21.15.141:6831/1084001742","heartbeat_front_addr":"172.21.15.141:6829/1084001742","state":["exists","up"]},{"osd":11,"uuid":"52f27807-1649-4c8d-9182-d161fa526d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6825","nonce":1791652726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6827","nonce":1791652726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6831","nonce":1791652726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":1791652726},{"type":"v1","addr":"172.21.15.155:6829","nonce":1791652726}]},"public_addr":"172.21.15.155:6825/1791652726","cluster_addr":"172.21.15.155:6827/1791652726","heartbeat_back_addr":"172.21.15.155:6831/1791652726","heartbeat_front_addr":"172.21.15.155:6829/1791652726","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:18.926313+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:42:40.965617+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:01.838689+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:29.445149+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:43:51.573664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:12.567267+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:44:39.582179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:01.962845+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:23.398874+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:45:50.264261+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:12.790405+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-16T16:46:34.457934+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":8}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/2834229980":"2023-11-17T16:49:54.626188+0000","172.21.15.2:0/3820370584":"2023-11-17T16:49:54.626188+0000","172.21.15.2:6801/1793067160":"2023-11-17T16:49:54.626188+0000","172.21.15.2:6800/1793067160":"2023-11-17T16:49:54.626188+0000","172.21.15.2:0/2137541200":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/2525877070":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/30571981":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3784433150":"2023-11-17T16:49:54.626188+0000","172.21.15.2:6800/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:0/500255408":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/981081063":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6800/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/1824419243":"2023-11-17T16:49:54.626188+0000","172.21.15.2:6801/3359672074":"2023-11-17T16:40:43.705118+0000","172.21.15.2:6801/3453602765":"2023-11-17T16:40:16.538166+0000","172.21.15.2:0/3818551797":"2023-11-17T16:40:16.538166+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-16T17:00:46.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.525+0000 7f4a5a7ff700 1 -- 172.21.15.2:0/653756345 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f4a3c0a1340 msgr2=0x7f4a3c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:46.527 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.525+0000 7f4a5a7ff700 1 --2- 172.21.15.2:0/653756345 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f4a3c0a1340 0x7f4a3c0a37f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4a540745e0 tx=0x7f4a4800b410 comp rx=0 tx=0).stop 2023-11-16T17:00:46.528 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.525+0000 7f4a5a7ff700 1 -- 172.21.15.2:0/653756345 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a54100550 msgr2=0x7f4a54072df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:46.528 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.525+0000 7f4a5a7ff700 1 --2- 172.21.15.2:0/653756345 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a54100550 0x7f4a54072df0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f4a4400b580 tx=0x7f4a4400f690 comp rx=0 tx=0).stop 2023-11-16T17:00:46.528 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.525+0000 7f4a5a7ff700 1 -- 172.21.15.2:0/653756345 shutdown_connections 2023-11-16T17:00:46.528 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.525+0000 7f4a5a7ff700 1 --2- 172.21.15.2:0/653756345 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f4a3c0a1340 0x7f4a3c0a37f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.528 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.525+0000 7f4a5a7ff700 1 --2- 172.21.15.2:0/653756345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a54105540 0x7f4a541a4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.525+0000 7f4a5a7ff700 1 --2- 172.21.15.2:0/653756345 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a54100550 0x7f4a54072df0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.525+0000 7f4a5a7ff700 1 --2- 172.21.15.2:0/653756345 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4a54100ef0 0x7f4a54073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.525+0000 7f4a5a7ff700 1 -- 172.21.15.2:0/653756345 >> 172.21.15.2:0/653756345 conn(0x7f4a54077020 msgr2=0x7f4a540fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:46.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.525+0000 7f4a5a7ff700 1 -- 172.21.15.2:0/653756345 shutdown_connections 2023-11-16T17:00:46.529 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.525+0000 7f4a5a7ff700 1 -- 172.21.15.2:0/653756345 wait complete. 2023-11-16T17:00:46.547 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-16T17:00:46.755 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.753+0000 7f3c17dff700 1 -- 172.21.15.2:0/2508736972 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c10100450 msgr2=0x7f3c10100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:46.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.753+0000 7f3c17dff700 1 --2- 172.21.15.2:0/2508736972 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c10100450 0x7f3c10100820 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c0099c0 tx=0x7f3c0c009cd0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.790 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.753+0000 7f3c17dff700 1 -- 172.21.15.2:0/2508736972 shutdown_connections 2023-11-16T17:00:46.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.753+0000 7f3c17dff700 1 --2- 172.21.15.2:0/2508736972 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c10105490 0x7f3c1010d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.753+0000 7f3c17dff700 1 --2- 172.21.15.2:0/2508736972 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c10100450 0x7f3c10100820 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.753+0000 7f3c17dff700 1 --2- 172.21.15.2:0/2508736972 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3c10100df0 0x7f3c10104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.753+0000 7f3c17dff700 1 -- 172.21.15.2:0/2508736972 >> 172.21.15.2:0/2508736972 conn(0x7f3c100fbce0 msgr2=0x7f3c100fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:46.791 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.753+0000 7f3c17dff700 1 -- 172.21.15.2:0/2508736972 shutdown_connections 2023-11-16T17:00:46.792 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.753+0000 7f3c17dff700 1 -- 172.21.15.2:0/2508736972 wait complete. 2023-11-16T17:00:46.795 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.754+0000 7f3c17dff700 1 Processor -- start 2023-11-16T17:00:46.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.755+0000 7f3c17dff700 1 -- start start 2023-11-16T17:00:46.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.755+0000 7f3c17dff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3c10100450 0x7f3c10198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:46.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.755+0000 7f3c17dff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c10100df0 0x7f3c10198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:46.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.755+0000 7f3c17dff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c10105490 0x7f3c1019cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:46.796 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.755+0000 7f3c17dff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1019d530 con 0x7f3c10100df0 2023-11-16T17:00:46.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.755+0000 7f3c17dff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1019d6a0 con 0x7f3c10105490 2023-11-16T17:00:46.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.755+0000 7f3c17dff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1019d860 con 0x7f3c10100450 2023-11-16T17:00:46.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.755+0000 7f3c1539a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c10100df0 0x7f3c10198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:46.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.755+0000 7f3c1539a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c10100df0 0x7f3c10198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46802/0 (socket says 172.21.15.2:46802) 2023-11-16T17:00:46.797 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.755+0000 7f3c1539a700 1 -- 172.21.15.2:0/3166676880 learned_addr learned my addr 172.21.15.2:0/3166676880 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T17:00:46.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.755+0000 7f3c15b9b700 1 --2- 172.21.15.2:0/3166676880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3c10100450 0x7f3c10198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:46.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.755+0000 7f3c1639c700 1 --2- 172.21.15.2:0/3166676880 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c10105490 0x7f3c1019cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:46.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.756+0000 7f3c1539a700 1 -- 172.21.15.2:0/3166676880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3c10100450 msgr2=0x7f3c10198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:46.798 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.756+0000 7f3c1539a700 1 --2- 172.21.15.2:0/3166676880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3c10100450 0x7f3c10198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.756+0000 7f3c1539a700 1 -- 172.21.15.2:0/3166676880 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c10105490 msgr2=0x7f3c1019cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:46.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.756+0000 7f3c1539a700 1 --2- 172.21.15.2:0/3166676880 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c10105490 0x7f3c1019cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:46.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.756+0000 7f3c1539a700 1 -- 172.21.15.2:0/3166676880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c0095f0 con 0x7f3c10100df0 2023-11-16T17:00:46.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.756+0000 7f3c1539a700 1 --2- 172.21.15.2:0/3166676880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c10100df0 0x7f3c10198a90 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3c0000e940 tx=0x7f3c0000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:46.799 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.756+0000 7f3c06ffd700 1 -- 172.21.15.2:0/3166676880 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c00009730 con 0x7f3c10100df0 2023-11-16T17:00:46.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.756+0000 7f3c06ffd700 1 -- 172.21.15.2:0/3166676880 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c00010430 con 0x7f3c10100df0 2023-11-16T17:00:46.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.756+0000 7f3c17dff700 1 -- 172.21.15.2:0/3166676880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c101102a0 con 0x7f3c10100df0 2023-11-16T17:00:46.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.756+0000 7f3c06ffd700 1 -- 172.21.15.2:0/3166676880 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0000cea0 con 0x7f3c10100df0 2023-11-16T17:00:46.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.756+0000 7f3c17dff700 1 -- 172.21.15.2:0/3166676880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c10110110 con 0x7f3c10100df0 2023-11-16T17:00:46.800 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.757+0000 7f3c17dff700 1 -- 172.21.15.2:0/3166676880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c10110430 con 0x7f3c10100df0 2023-11-16T17:00:46.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.759+0000 7f3c06ffd700 1 -- 172.21.15.2:0/3166676880 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f3c00010810 con 0x7f3c10100df0 2023-11-16T17:00:46.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.762+0000 7f3c06ffd700 1 --2- 172.21.15.2:0/3166676880 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f3bfc0a12c0 0x7f3bfc0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:46.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.762+0000 7f3c06ffd700 1 -- 172.21.15.2:0/3166676880 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9482+0+0 (secure 0 0 0) 0x7f3c00014070 con 0x7f3c10100df0 2023-11-16T17:00:46.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.762+0000 7f3c15b9b700 1 --2- 172.21.15.2:0/3166676880 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f3bfc0a12c0 0x7f3bfc0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:46.801 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.763+0000 7f3c15b9b700 1 --2- 172.21.15.2:0/3166676880 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f3bfc0a12c0 0x7f3bfc0a3770 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009990 tx=0x7f3c0c005fb0 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:46.802 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:46.766+0000 7f3c06ffd700 1 -- 172.21.15.2:0/3166676880 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3c00051730 con 0x7f3c10100df0 2023-11-16T17:00:47.024 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.022+0000 7f3c17dff700 1 -- 172.21.15.2:0/3166676880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3c10067980 con 0x7f3c10100df0 2023-11-16T17:00:47.104 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.102+0000 7f3c06ffd700 1 -- 172.21.15.2:0/3166676880 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v42) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3c00096290 con 0x7f3c10100df0 2023-11-16T17:00:47.107 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.104+0000 7f3c17dff700 1 -- 172.21.15.2:0/3166676880 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f3bfc0a12c0 msgr2=0x7f3bfc0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:47.107 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.104+0000 7f3c17dff700 1 --2- 172.21.15.2:0/3166676880 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f3bfc0a12c0 0x7f3bfc0a3770 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009990 tx=0x7f3c0c005fb0 comp rx=0 tx=0).stop 2023-11-16T17:00:47.107 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.105+0000 7f3c17dff700 1 -- 172.21.15.2:0/3166676880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c10100df0 msgr2=0x7f3c10198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:47.107 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.105+0000 7f3c17dff700 1 --2- 172.21.15.2:0/3166676880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c10100df0 0x7f3c10198a90 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3c0000e940 tx=0x7f3c0000ec50 comp rx=0 tx=0).stop 2023-11-16T17:00:47.108 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.105+0000 7f3c17dff700 1 -- 172.21.15.2:0/3166676880 shutdown_connections 2023-11-16T17:00:47.108 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.105+0000 7f3c17dff700 1 --2- 172.21.15.2:0/3166676880 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f3bfc0a12c0 0x7f3bfc0a3770 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:47.108 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.105+0000 7f3c17dff700 1 --2- 172.21.15.2:0/3166676880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3c10100450 0x7f3c10198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:47.108 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.105+0000 7f3c17dff700 1 --2- 172.21.15.2:0/3166676880 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f3c10105490 0x7f3c1019cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:47.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.105+0000 7f3c17dff700 1 --2- 172.21.15.2:0/3166676880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c10100df0 0x7f3c10198a90 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:47.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.105+0000 7f3c17dff700 1 -- 172.21.15.2:0/3166676880 >> 172.21.15.2:0/3166676880 conn(0x7f3c100fbce0 msgr2=0x7f3c100fca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:47.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.105+0000 7f3c17dff700 1 -- 172.21.15.2:0/3166676880 shutdown_connections 2023-11-16T17:00:47.109 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.105+0000 7f3c17dff700 1 -- 172.21.15.2:0/3166676880 wait complete. 2023-11-16T17:00:47.110 INFO:teuthology.orchestra.run.smithi002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-16T17:00:47.126 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-16T17:00:47.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.336+0000 7fae979ff700 1 -- 172.21.15.2:0/3720965235 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae900746f0 msgr2=0x7fae90072d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:47.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.336+0000 7fae979ff700 1 --2- 172.21.15.2:0/3720965235 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae900746f0 0x7fae90072d20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fae8c009990 tx=0x7fae8c009ca0 comp rx=0 tx=0).stop 2023-11-16T17:00:47.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.336+0000 7fae979ff700 1 -- 172.21.15.2:0/3720965235 shutdown_connections 2023-11-16T17:00:47.339 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.336+0000 7fae979ff700 1 --2- 172.21.15.2:0/3720965235 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae900746f0 0x7fae90072d20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:47.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.336+0000 7fae979ff700 1 --2- 172.21.15.2:0/3720965235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae9010cec0 0x7fae9010f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:47.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.336+0000 7fae979ff700 1 --2- 172.21.15.2:0/3720965235 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae90074ac0 0x7fae90073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:47.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.336+0000 7fae979ff700 1 -- 172.21.15.2:0/3720965235 >> 172.21.15.2:0/3720965235 conn(0x7fae90077560 msgr2=0x7fae90077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:47.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.337+0000 7fae979ff700 1 -- 172.21.15.2:0/3720965235 shutdown_connections 2023-11-16T17:00:47.340 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.337+0000 7fae979ff700 1 -- 172.21.15.2:0/3720965235 wait complete. 2023-11-16T17:00:47.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.338+0000 7fae979ff700 1 Processor -- start 2023-11-16T17:00:47.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.338+0000 7fae979ff700 1 -- start start 2023-11-16T17:00:47.343 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.339+0000 7fae979ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae900746f0 0x7fae90198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:47.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.339+0000 7fae979ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae90074ac0 0x7fae90198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:47.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.339+0000 7fae979ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae9010cec0 0x7fae9019c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:47.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.339+0000 7fae979ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9019c600 con 0x7fae90074ac0 2023-11-16T17:00:47.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.339+0000 7fae979ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9019c770 con 0x7fae900746f0 2023-11-16T17:00:47.344 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.339+0000 7fae979ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9019c930 con 0x7fae9010cec0 2023-11-16T17:00:47.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.339+0000 7fae94f9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae90074ac0 0x7fae90198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:47.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.339+0000 7fae94f9a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae90074ac0 0x7fae90198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46826/0 (socket says 172.21.15.2:46826) 2023-11-16T17:00:47.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.339+0000 7fae94f9a700 1 -- 172.21.15.2:0/1530381261 learned_addr learned my addr 172.21.15.2:0/1530381261 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T17:00:47.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.339+0000 7fae9579b700 1 --2- 172.21.15.2:0/1530381261 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae900746f0 0x7fae90198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:47.345 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.339+0000 7fae95f9c700 1 --2- 172.21.15.2:0/1530381261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae9010cec0 0x7fae9019c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:47.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.340+0000 7fae94f9a700 1 -- 172.21.15.2:0/1530381261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae9010cec0 msgr2=0x7fae9019c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:47.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.340+0000 7fae94f9a700 1 --2- 172.21.15.2:0/1530381261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae9010cec0 0x7fae9019c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:47.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.340+0000 7fae94f9a700 1 -- 172.21.15.2:0/1530381261 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae900746f0 msgr2=0x7fae90198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:47.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.340+0000 7fae94f9a700 1 --2- 172.21.15.2:0/1530381261 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae900746f0 0x7fae90198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:47.346 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.340+0000 7fae94f9a700 1 -- 172.21.15.2:0/1530381261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae80009520 con 0x7fae90074ac0 2023-11-16T17:00:47.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.340+0000 7fae94f9a700 1 --2- 172.21.15.2:0/1530381261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae90074ac0 0x7fae90198c50 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fae8000ea30 tx=0x7fae8000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:47.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.340+0000 7fae867fc700 1 -- 172.21.15.2:0/1530381261 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8000cbd0 con 0x7fae90074ac0 2023-11-16T17:00:47.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.340+0000 7fae979ff700 1 -- 172.21.15.2:0/1530381261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae8c0095f0 con 0x7fae90074ac0 2023-11-16T17:00:47.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.340+0000 7fae867fc700 1 -- 172.21.15.2:0/1530381261 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fae8000eed0 con 0x7fae90074ac0 2023-11-16T17:00:47.347 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.340+0000 7fae867fc700 1 -- 172.21.15.2:0/1530381261 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae80003d20 con 0x7fae90074ac0 2023-11-16T17:00:47.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.340+0000 7fae979ff700 1 -- 172.21.15.2:0/1530381261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae901125a0 con 0x7fae90074ac0 2023-11-16T17:00:47.348 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.341+0000 7fae979ff700 1 -- 172.21.15.2:0/1530381261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae90112410 con 0x7fae90074ac0 2023-11-16T17:00:47.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.343+0000 7fae867fc700 1 -- 172.21.15.2:0/1530381261 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7fae80003ee0 con 0x7fae90074ac0 2023-11-16T17:00:47.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.345+0000 7fae867fc700 1 --2- 172.21.15.2:0/1530381261 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fae7c0a12c0 0x7fae7c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:47.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.345+0000 7fae867fc700 1 -- 172.21.15.2:0/1530381261 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9652+0+0 (secure 0 0 0) 0x7fae80014070 con 0x7fae90074ac0 2023-11-16T17:00:47.349 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.345+0000 7fae9579b700 1 --2- 172.21.15.2:0/1530381261 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fae7c0a12c0 0x7fae7c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:47.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.346+0000 7fae9579b700 1 --2- 172.21.15.2:0/1530381261 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fae7c0a12c0 0x7fae7c0a3770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fae8c00b2e0 tx=0x7fae8c005fd0 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:47.350 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.349+0000 7fae867fc700 1 -- 172.21.15.2:0/1530381261 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fae80096460 con 0x7fae90074ac0 2023-11-16T17:00:47.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:47 smithi155 ceph-mon[131607]: pgmap v339: 129 pgs: 129 active+clean; 607 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T17:00:47.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:47 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/653756345' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T17:00:47.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:47 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3166676880' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-11-16T17:00:47.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:47 smithi155 ceph-mon[131607]: osdmap e106: 12 total, 12 up, 12 in 2023-11-16T17:00:47.369 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 17:00:47 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[161168]: ignoring --setuser ceph since I am not root 2023-11-16T17:00:47.369 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 17:00:47 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[161168]: ignoring --setgroup ceph since I am not root 2023-11-16T17:00:47.369 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 17:00:47 smithi155 ceph-mds[161192]: -- 172.21.15.155:0/1042426750 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fb93f7b520 con 0x55fb93f50800 2023-11-16T17:00:47.369 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 17:00:47 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[161168]: starting mds.c at 2023-11-16T17:00:47.370 INFO:journalctl@ceph.mds.i.smithi155.stdout:Nov 16 17:00:47 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-i[157661]: ignoring --setuser ceph since I am not root 2023-11-16T17:00:47.370 INFO:journalctl@ceph.mds.i.smithi155.stdout:Nov 16 17:00:47 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-i[157661]: ignoring --setgroup ceph since I am not root 2023-11-16T17:00:47.371 INFO:journalctl@ceph.mds.i.smithi155.stdout:Nov 16 17:00:47 smithi155 ceph-mds[157685]: -- 172.21.15.155:0/1296445558 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55eb56799520 con 0x55eb5676f000 2023-11-16T17:00:47.371 INFO:journalctl@ceph.mds.i.smithi155.stdout:Nov 16 17:00:47 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-i[157661]: starting mds.i at 2023-11-16T17:00:47.371 INFO:journalctl@ceph.mds.f.smithi155.stdout:Nov 16 17:00:47 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-f[156980]: ignoring --setuser ceph since I am not root 2023-11-16T17:00:47.372 INFO:journalctl@ceph.mds.f.smithi155.stdout:Nov 16 17:00:47 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-f[156980]: ignoring --setgroup ceph since I am not root 2023-11-16T17:00:47.372 INFO:journalctl@ceph.mds.f.smithi155.stdout:Nov 16 17:00:47 smithi155 ceph-mds[157004]: -- 172.21.15.155:0/3300303206 <== mon.2 v2:172.21.15.141:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561712cfd520 con 0x561712cd3000 2023-11-16T17:00:47.372 INFO:journalctl@ceph.mds.f.smithi155.stdout:Nov 16 17:00:47 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-f[156980]: starting mds.f at 2023-11-16T17:00:47.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:47 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mon-a[128688]: 2023-11-16T17:00:47.098+0000 7fb47111f700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-16T17:00:47.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:47 smithi002 ceph-mon[128712]: pgmap v339: 129 pgs: 129 active+clean; 607 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T17:00:47.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:47 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/653756345' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T17:00:47.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:47 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3166676880' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-11-16T17:00:47.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:47 smithi002 ceph-mon[128712]: osdmap e106: 12 total, 12 up, 12 in 2023-11-16T17:00:47.515 INFO:journalctl@ceph.mds.b.smithi141.stdout:Nov 16 17:00:47 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-b[156414]: ignoring --setuser ceph since I am not root 2023-11-16T17:00:47.515 INFO:journalctl@ceph.mds.b.smithi141.stdout:Nov 16 17:00:47 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-b[156414]: ignoring --setgroup ceph since I am not root 2023-11-16T17:00:47.515 INFO:journalctl@ceph.mds.b.smithi141.stdout:Nov 16 17:00:47 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-b[156414]: starting mds.b at 2023-11-16T17:00:47.516 INFO:journalctl@ceph.mds.h.smithi141.stdout:Nov 16 17:00:47 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-h[156072]: ignoring --setuser ceph since I am not root 2023-11-16T17:00:47.516 INFO:journalctl@ceph.mds.h.smithi141.stdout:Nov 16 17:00:47 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-h[156072]: ignoring --setgroup ceph since I am not root 2023-11-16T17:00:47.516 INFO:journalctl@ceph.mds.h.smithi141.stdout:Nov 16 17:00:47 smithi141 ceph-mds[156097]: -- 172.21.15.141:0/918049492 <== mon.2 v2:172.21.15.141:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ee478fb520 con 0x55ee478d1000 2023-11-16T17:00:47.517 INFO:journalctl@ceph.mds.h.smithi141.stdout:Nov 16 17:00:47 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-h[156072]: starting mds.h at 2023-11-16T17:00:47.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:47 smithi141 ceph-mon[131184]: pgmap v339: 129 pgs: 129 active+clean; 607 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-11-16T17:00:47.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:47 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/653756345' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-16T17:00:47.518 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:47 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3166676880' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-11-16T17:00:47.518 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:47 smithi141 ceph-mon[131184]: osdmap e106: 12 total, 12 up, 12 in 2023-11-16T17:00:47.518 INFO:journalctl@ceph.mds.e.smithi141.stdout:Nov 16 17:00:47 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-e[155733]: ignoring --setuser ceph since I am not root 2023-11-16T17:00:47.519 INFO:journalctl@ceph.mds.e.smithi141.stdout:Nov 16 17:00:47 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-e[155733]: ignoring --setgroup ceph since I am not root 2023-11-16T17:00:47.519 INFO:journalctl@ceph.mds.e.smithi141.stdout:Nov 16 17:00:47 smithi141 ceph-mds[155757]: -- 172.21.15.141:0/973134734 <== mon.2 v2:172.21.15.141:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d371f0b520 con 0x55d371ee0800 2023-11-16T17:00:47.519 INFO:journalctl@ceph.mds.e.smithi141.stdout:Nov 16 17:00:47 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-e[155733]: starting mds.e at 2023-11-16T17:00:47.605 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:47.604+0000 7fae979ff700 1 -- 172.21.15.2:0/1530381261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fae90112730 con 0x7fae90074ac0 2023-11-16T17:00:48.116 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.115+0000 7fae867fc700 1 -- 172.21.15.2:0/1530381261 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7fae80099a80 con 0x7fae90074ac0 2023-11-16T17:00:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.118+0000 7fae979ff700 1 -- 172.21.15.2:0/1530381261 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fae7c0a12c0 msgr2=0x7fae7c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.118+0000 7fae979ff700 1 --2- 172.21.15.2:0/1530381261 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fae7c0a12c0 0x7fae7c0a3770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fae8c00b2e0 tx=0x7fae8c005fd0 comp rx=0 tx=0).stop 2023-11-16T17:00:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.118+0000 7fae979ff700 1 -- 172.21.15.2:0/1530381261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae90074ac0 msgr2=0x7fae90198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.118+0000 7fae979ff700 1 --2- 172.21.15.2:0/1530381261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae90074ac0 0x7fae90198c50 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fae8000ea30 tx=0x7fae8000ed40 comp rx=0 tx=0).stop 2023-11-16T17:00:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.118+0000 7fae979ff700 1 -- 172.21.15.2:0/1530381261 shutdown_connections 2023-11-16T17:00:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.118+0000 7fae979ff700 1 --2- 172.21.15.2:0/1530381261 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7fae7c0a12c0 0x7fae7c0a3770 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.118+0000 7fae979ff700 1 --2- 172.21.15.2:0/1530381261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae9010cec0 0x7fae9019c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.118+0000 7fae979ff700 1 --2- 172.21.15.2:0/1530381261 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7fae900746f0 0x7fae90198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.118+0000 7fae979ff700 1 --2- 172.21.15.2:0/1530381261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae90074ac0 0x7fae90198c50 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.118+0000 7fae979ff700 1 -- 172.21.15.2:0/1530381261 >> 172.21.15.2:0/1530381261 conn(0x7fae90077560 msgr2=0x7fae90104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.118+0000 7fae979ff700 1 -- 172.21.15.2:0/1530381261 shutdown_connections 2023-11-16T17:00:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.118+0000 7fae979ff700 1 -- 172.21.15.2:0/1530381261 wait complete. 2023-11-16T17:00:48.142 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-16T17:00:48.358 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.356+0000 7f07987ff700 1 -- 172.21.15.2:0/3831307057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0790100ee0 msgr2=0x7f0790104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:48.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.356+0000 7f07987ff700 1 --2- 172.21.15.2:0/3831307057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0790100ee0 0x7f0790104ea0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f07800099c0 tx=0x7f0780009cd0 comp rx=0 tx=0).stop 2023-11-16T17:00:48.359 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.356+0000 7f07987ff700 1 -- 172.21.15.2:0/3831307057 shutdown_connections 2023-11-16T17:00:48.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.356+0000 7f07987ff700 1 --2- 172.21.15.2:0/3831307057 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0790100540 0x7f0790100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:48.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.356+0000 7f07987ff700 1 --2- 172.21.15.2:0/3831307057 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0790105580 0x7f079010d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:48.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.356+0000 7f07987ff700 1 --2- 172.21.15.2:0/3831307057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0790100ee0 0x7f0790104ea0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:48.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.356+0000 7f07987ff700 1 -- 172.21.15.2:0/3831307057 >> 172.21.15.2:0/3831307057 conn(0x7f07900770a0 msgr2=0x7f07900774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:48.360 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.357+0000 7f07987ff700 1 -- 172.21.15.2:0/3831307057 shutdown_connections 2023-11-16T17:00:48.361 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.357+0000 7f07987ff700 1 -- 172.21.15.2:0/3831307057 wait complete. 2023-11-16T17:00:48.363 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.357+0000 7f07987ff700 1 Processor -- start 2023-11-16T17:00:48.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.358+0000 7f07987ff700 1 -- start start 2023-11-16T17:00:48.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.358+0000 7f07987ff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0790100540 0x7f0790072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:48.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.358+0000 7f07987ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0790100ee0 0x7f0790073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:48.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.358+0000 7f07987ff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0790105580 0x7f07901a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:48.364 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.358+0000 7f07987ff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0790073b60 con 0x7f0790105580 2023-11-16T17:00:48.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.358+0000 7f07987ff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07901a4b10 con 0x7f0790100540 2023-11-16T17:00:48.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.358+0000 7f07987ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07901a4c50 con 0x7f0790100ee0 2023-11-16T17:00:48.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.358+0000 7f0796d9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0790105580 0x7f07901a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:48.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.358+0000 7f0796d9c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0790105580 0x7f07901a45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46846/0 (socket says 172.21.15.2:46846) 2023-11-16T17:00:48.365 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.358+0000 7f0796d9c700 1 -- 172.21.15.2:0/2502517820 learned_addr learned my addr 172.21.15.2:0/2502517820 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T17:00:48.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.358+0000 7f079659b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0790100540 0x7f0790072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:48.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.358+0000 7f0795d9a700 1 --2- 172.21.15.2:0/2502517820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0790100ee0 0x7f0790073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:48.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.359+0000 7f0796d9c700 1 -- 172.21.15.2:0/2502517820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0790100ee0 msgr2=0x7f0790073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:48.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.359+0000 7f0796d9c700 1 --2- 172.21.15.2:0/2502517820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0790100ee0 0x7f0790073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:48.366 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.359+0000 7f0796d9c700 1 -- 172.21.15.2:0/2502517820 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0790100540 msgr2=0x7f0790072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:48.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.359+0000 7f0796d9c700 1 --2- 172.21.15.2:0/2502517820 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0790100540 0x7f0790072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:48.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.359+0000 7f0796d9c700 1 -- 172.21.15.2:0/2502517820 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07800095f0 con 0x7f0790105580 2023-11-16T17:00:48.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.359+0000 7f0796d9c700 1 --2- 172.21.15.2:0/2502517820 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0790105580 0x7f07901a45d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f078800ef30 tx=0x7f078800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:48.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.359+0000 7f07877fe700 1 -- 172.21.15.2:0/2502517820 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0788018070 con 0x7f0790105580 2023-11-16T17:00:48.367 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.359+0000 7f07877fe700 1 -- 172.21.15.2:0/2502517820 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f078800f040 con 0x7f0790105580 2023-11-16T17:00:48.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.359+0000 7f07987ff700 1 -- 172.21.15.2:0/2502517820 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0790110350 con 0x7f0790105580 2023-11-16T17:00:48.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.359+0000 7f07877fe700 1 -- 172.21.15.2:0/2502517820 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078800cd00 con 0x7f0790105580 2023-11-16T17:00:48.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.359+0000 7f07987ff700 1 -- 172.21.15.2:0/2502517820 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07901101c0 con 0x7f0790105580 2023-11-16T17:00:48.368 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.360+0000 7f07987ff700 1 -- 172.21.15.2:0/2502517820 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07901104e0 con 0x7f0790105580 2023-11-16T17:00:48.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:48 smithi155 ceph-mon[131607]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-11-16T17:00:48.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:48 smithi155 ceph-mon[131607]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-16T17:00:48.370 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:48 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3166676880' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-11-16T17:00:48.370 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:48 smithi155 ceph-mon[131607]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-11-16T17:00:48.370 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:48 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1530381261' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T17:00:48.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.362+0000 7f07877fe700 1 -- 172.21.15.2:0/2502517820 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f0788003a00 con 0x7f0790105580 2023-11-16T17:00:48.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.364+0000 7f07877fe700 1 --2- 172.21.15.2:0/2502517820 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f077c0a1340 0x7f077c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:48.371 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.364+0000 7f07877fe700 1 -- 172.21.15.2:0/2502517820 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9652+0+0 (secure 0 0 0) 0x7f07880ccc00 con 0x7f0790105580 2023-11-16T17:00:48.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.364+0000 7f079659b700 1 --2- 172.21.15.2:0/2502517820 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f077c0a1340 0x7f077c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:48.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.365+0000 7f079659b700 1 --2- 172.21.15.2:0/2502517820 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f077c0a1340 0x7f077c0a37f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f078c009c60 tx=0x7f078c009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:48.372 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.368+0000 7f07877fe700 1 -- 172.21.15.2:0/2502517820 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0788095bd0 con 0x7f0790105580 2023-11-16T17:00:48.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:48 smithi002 ceph-mon[128712]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-11-16T17:00:48.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:48 smithi002 ceph-mon[128712]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-16T17:00:48.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:48 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3166676880' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-11-16T17:00:48.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:48 smithi002 ceph-mon[128712]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-11-16T17:00:48.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:48 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1530381261' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T17:00:48.514 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:48 smithi141 ceph-mon[131184]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-11-16T17:00:48.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:48 smithi141 ceph-mon[131184]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-16T17:00:48.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:48 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3166676880' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-11-16T17:00:48.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:48 smithi141 ceph-mon[131184]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-11-16T17:00:48.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:48 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1530381261' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-11-16T17:00:48.577 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:48.575+0000 7f07987ff700 1 -- 172.21.15.2:0/2502517820 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0790067270 con 0x7f0790105580 2023-11-16T17:00:49.206 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.204+0000 7f07877fe700 1 -- 172.21.15.2:0/2502517820 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v107) v1 ==== 181+0+0 (secure 0 0 0) 0x7f07880991f0 con 0x7f0790105580 2023-11-16T17:00:49.210 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.209+0000 7f07987ff700 1 -- 172.21.15.2:0/2502517820 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f077c0a1340 msgr2=0x7f077c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:49.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.209+0000 7f07987ff700 1 --2- 172.21.15.2:0/2502517820 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f077c0a1340 0x7f077c0a37f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f078c009c60 tx=0x7f078c009260 comp rx=0 tx=0).stop 2023-11-16T17:00:49.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.209+0000 7f07987ff700 1 -- 172.21.15.2:0/2502517820 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0790105580 msgr2=0x7f07901a45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:49.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.209+0000 7f07987ff700 1 --2- 172.21.15.2:0/2502517820 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0790105580 0x7f07901a45d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f078800ef30 tx=0x7f078800c3a0 comp rx=0 tx=0).stop 2023-11-16T17:00:49.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.209+0000 7f07987ff700 1 -- 172.21.15.2:0/2502517820 shutdown_connections 2023-11-16T17:00:49.211 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.209+0000 7f07987ff700 1 --2- 172.21.15.2:0/2502517820 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f077c0a1340 0x7f077c0a37f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:49.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.209+0000 7f07987ff700 1 --2- 172.21.15.2:0/2502517820 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0790105580 0x7f07901a45d0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:49.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.209+0000 7f07987ff700 1 --2- 172.21.15.2:0/2502517820 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f0790100540 0x7f0790072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:49.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.209+0000 7f07987ff700 1 --2- 172.21.15.2:0/2502517820 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0790100ee0 0x7f0790073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:49.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.209+0000 7f07987ff700 1 -- 172.21.15.2:0/2502517820 >> 172.21.15.2:0/2502517820 conn(0x7f07900770a0 msgr2=0x7f07900fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:49.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.209+0000 7f07987ff700 1 -- 172.21.15.2:0/2502517820 shutdown_connections 2023-11-16T17:00:49.213 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.209+0000 7f07987ff700 1 -- 172.21.15.2:0/2502517820 wait complete. 2023-11-16T17:00:49.213 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_metadata' removed 2023-11-16T17:00:49.236 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-16T17:00:49.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:49 smithi155 ceph-mon[131607]: pgmap v341: 129 pgs: 129 active+clean; 607 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2023-11-16T17:00:49.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:49 smithi155 ceph-mon[131607]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-11-16T17:00:49.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:49 smithi155 ceph-mon[131607]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-11-16T17:00:49.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:49 smithi155 ceph-mon[131607]: Cluster is now healthy 2023-11-16T17:00:49.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:49 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.141:6834/4153256359,v1:172.21.15.141:6835/4153256359] up:boot 2023-11-16T17:00:49.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:49 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.155:6838/4257247018,v1:172.21.15.155:6839/4257247018] up:boot 2023-11-16T17:00:49.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:49 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.141:6836/844387741,v1:172.21.15.141:6837/844387741] up:boot 2023-11-16T17:00:49.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:49 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.155:6834/3741814455,v1:172.21.15.155:6835/3741814455] up:boot 2023-11-16T17:00:49.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:49 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.155:6836/475122095,v1:172.21.15.155:6837/475122095] up:boot 2023-11-16T17:00:49.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:49 smithi155 ceph-mon[131607]: mds.? [v2:172.21.15.141:6838/682792078,v1:172.21.15.141:6839/682792078] up:boot 2023-11-16T17:00:49.369 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:49 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/1530381261' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-11-16T17:00:49.370 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:49 smithi155 ceph-mon[131607]: fsmap 12 up:standby 2023-11-16T17:00:49.370 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:49 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T17:00:49.370 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:49 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T17:00:49.370 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:49 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T17:00:49.370 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:49 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T17:00:49.371 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:49 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T17:00:49.371 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:49 smithi155 ceph-mon[131607]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T17:00:49.371 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:49 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2502517820' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:00:49.405 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:49 smithi002 ceph-mon[128712]: pgmap v341: 129 pgs: 129 active+clean; 607 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2023-11-16T17:00:49.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:49 smithi002 ceph-mon[128712]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-11-16T17:00:49.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:49 smithi002 ceph-mon[128712]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-11-16T17:00:49.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:49 smithi002 ceph-mon[128712]: Cluster is now healthy 2023-11-16T17:00:49.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:49 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.141:6834/4153256359,v1:172.21.15.141:6835/4153256359] up:boot 2023-11-16T17:00:49.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:49 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.155:6838/4257247018,v1:172.21.15.155:6839/4257247018] up:boot 2023-11-16T17:00:49.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:49 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.141:6836/844387741,v1:172.21.15.141:6837/844387741] up:boot 2023-11-16T17:00:49.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:49 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.155:6834/3741814455,v1:172.21.15.155:6835/3741814455] up:boot 2023-11-16T17:00:49.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:49 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.155:6836/475122095,v1:172.21.15.155:6837/475122095] up:boot 2023-11-16T17:00:49.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:49 smithi002 ceph-mon[128712]: mds.? [v2:172.21.15.141:6838/682792078,v1:172.21.15.141:6839/682792078] up:boot 2023-11-16T17:00:49.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:49 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/1530381261' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-11-16T17:00:49.408 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:49 smithi002 ceph-mon[128712]: fsmap 12 up:standby 2023-11-16T17:00:49.408 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:49 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T17:00:49.408 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:49 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T17:00:49.408 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:49 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T17:00:49.408 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:49 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T17:00:49.408 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:49 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T17:00:49.409 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:49 smithi002 ceph-mon[128712]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T17:00:49.409 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:49 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2502517820' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:00:49.452 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.450+0000 7f4517bff700 1 -- 172.21.15.2:0/2162050732 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4510100cc0 msgr2=0x7f4510101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:49.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.450+0000 7f4517bff700 1 --2- 172.21.15.2:0/2162050732 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4510100cc0 0x7f4510101090 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f450c0099c0 tx=0x7f450c009cd0 comp rx=0 tx=0).stop 2023-11-16T17:00:49.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.451+0000 7f4517bff700 1 -- 172.21.15.2:0/2162050732 shutdown_connections 2023-11-16T17:00:49.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.451+0000 7f4517bff700 1 --2- 172.21.15.2:0/2162050732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4510105620 0x7f451010d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:49.453 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.451+0000 7f4517bff700 1 --2- 172.21.15.2:0/2162050732 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4510100cc0 0x7f4510101090 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:49.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.451+0000 7f4517bff700 1 --2- 172.21.15.2:0/2162050732 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4510101660 0x7f4510104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:49.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.451+0000 7f4517bff700 1 -- 172.21.15.2:0/2162050732 >> 172.21.15.2:0/2162050732 conn(0x7f4510077560 msgr2=0x7f4510077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:49.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.451+0000 7f4517bff700 1 -- 172.21.15.2:0/2162050732 shutdown_connections 2023-11-16T17:00:49.454 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.451+0000 7f4517bff700 1 -- 172.21.15.2:0/2162050732 wait complete. 2023-11-16T17:00:49.456 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.452+0000 7f4517bff700 1 Processor -- start 2023-11-16T17:00:49.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.452+0000 7f4517bff700 1 -- start start 2023-11-16T17:00:49.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.452+0000 7f4517bff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4510100cc0 0x7f451019e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:49.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.452+0000 7f4517bff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4510101660 0x7f451019e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:49.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.452+0000 7f4517bff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4510105620 0x7f4510198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:49.457 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.452+0000 7f4517bff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4510198b80 con 0x7f4510101660 2023-11-16T17:00:49.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.452+0000 7f4517bff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4510198cf0 con 0x7f4510100cc0 2023-11-16T17:00:49.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.452+0000 7f4517bff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4510198eb0 con 0x7f4510105620 2023-11-16T17:00:49.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.453+0000 7f451519a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4510101660 0x7f451019e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:49.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.453+0000 7f451519a700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4510101660 0x7f451019e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46864/0 (socket says 172.21.15.2:46864) 2023-11-16T17:00:49.458 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.453+0000 7f451519a700 1 -- 172.21.15.2:0/3480614880 learned_addr learned my addr 172.21.15.2:0/3480614880 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T17:00:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.453+0000 7f451599b700 1 --2- 172.21.15.2:0/3480614880 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4510100cc0 0x7f451019e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.453+0000 7f451619c700 1 --2- 172.21.15.2:0/3480614880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4510105620 0x7f4510198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.453+0000 7f451519a700 1 -- 172.21.15.2:0/3480614880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4510105620 msgr2=0x7f4510198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.453+0000 7f451519a700 1 --2- 172.21.15.2:0/3480614880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4510105620 0x7f4510198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.453+0000 7f451519a700 1 -- 172.21.15.2:0/3480614880 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4510100cc0 msgr2=0x7f451019e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.453+0000 7f451519a700 1 --2- 172.21.15.2:0/3480614880 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4510100cc0 0x7f451019e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:49.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.453+0000 7f451519a700 1 -- 172.21.15.2:0/3480614880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450c0095f0 con 0x7f4510101660 2023-11-16T17:00:49.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.453+0000 7f451519a700 1 --2- 172.21.15.2:0/3480614880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4510101660 0x7f451019e990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f450000e910 tx=0x7f450000ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:49.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.454+0000 7f4506ffd700 1 -- 172.21.15.2:0/3480614880 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450000cbd0 con 0x7f4510101660 2023-11-16T17:00:49.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.454+0000 7f4506ffd700 1 -- 172.21.15.2:0/3480614880 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f450000ee60 con 0x7f4510101660 2023-11-16T17:00:49.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.454+0000 7f4506ffd700 1 -- 172.21.15.2:0/3480614880 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45000055c0 con 0x7f4510101660 2023-11-16T17:00:49.460 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.454+0000 7f4517bff700 1 -- 172.21.15.2:0/3480614880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45101103d0 con 0x7f4510101660 2023-11-16T17:00:49.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.454+0000 7f4517bff700 1 -- 172.21.15.2:0/3480614880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4510110240 con 0x7f4510101660 2023-11-16T17:00:49.461 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.455+0000 7f4517bff700 1 -- 172.21.15.2:0/3480614880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4510110560 con 0x7f4510101660 2023-11-16T17:00:49.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.457+0000 7f4506ffd700 1 -- 172.21.15.2:0/3480614880 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f4500004750 con 0x7f4510101660 2023-11-16T17:00:49.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.459+0000 7f4506ffd700 1 --2- 172.21.15.2:0/3480614880 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f44fc0a1340 0x7f44fc0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:49.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.459+0000 7f4506ffd700 1 -- 172.21.15.2:0/3480614880 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9098+0+0 (secure 0 0 0) 0x7f4500014070 con 0x7f4510101660 2023-11-16T17:00:49.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.459+0000 7f451599b700 1 --2- 172.21.15.2:0/3480614880 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f44fc0a1340 0x7f44fc0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:49.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.460+0000 7f451599b700 1 --2- 172.21.15.2:0/3480614880 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f44fc0a1340 0x7f44fc0a37f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f450c009990 tx=0x7f450c005fb0 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:49.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.463+0000 7f4506ffd700 1 -- 172.21.15.2:0/3480614880 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4500096450 con 0x7f4510101660 2023-11-16T17:00:49.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:49 smithi141 ceph-mon[131184]: pgmap v341: 129 pgs: 129 active+clean; 607 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2023-11-16T17:00:49.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:49 smithi141 ceph-mon[131184]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-11-16T17:00:49.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:49 smithi141 ceph-mon[131184]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-11-16T17:00:49.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:49 smithi141 ceph-mon[131184]: Cluster is now healthy 2023-11-16T17:00:49.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:49 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.141:6834/4153256359,v1:172.21.15.141:6835/4153256359] up:boot 2023-11-16T17:00:49.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:49 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.155:6838/4257247018,v1:172.21.15.155:6839/4257247018] up:boot 2023-11-16T17:00:49.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:49 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.141:6836/844387741,v1:172.21.15.141:6837/844387741] up:boot 2023-11-16T17:00:49.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:49 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.155:6834/3741814455,v1:172.21.15.155:6835/3741814455] up:boot 2023-11-16T17:00:49.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:49 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.155:6836/475122095,v1:172.21.15.155:6837/475122095] up:boot 2023-11-16T17:00:49.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:49 smithi141 ceph-mon[131184]: mds.? [v2:172.21.15.141:6838/682792078,v1:172.21.15.141:6839/682792078] up:boot 2023-11-16T17:00:49.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:49 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/1530381261' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-11-16T17:00:49.516 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:49 smithi141 ceph-mon[131184]: fsmap 12 up:standby 2023-11-16T17:00:49.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:49 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-11-16T17:00:49.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:49 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-11-16T17:00:49.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:49 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-11-16T17:00:49.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:49 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-11-16T17:00:49.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:49 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-11-16T17:00:49.517 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:49 smithi141 ceph-mon[131184]: from='mgr.15372 172.21.15.2:0/4237691304' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-11-16T17:00:49.518 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:49 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2502517820' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:00:49.686 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:49.685+0000 7f4517bff700 1 -- 172.21.15.2:0/3480614880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4510067b00 con 0x7f4510101660 2023-11-16T17:00:50.212 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.210+0000 7f4506ffd700 1 -- 172.21.15.2:0/3480614880 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v108) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4500099a70 con 0x7f4510101660 2023-11-16T17:00:50.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.213+0000 7f4517bff700 1 -- 172.21.15.2:0/3480614880 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f44fc0a1340 msgr2=0x7f44fc0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:50.215 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.213+0000 7f4517bff700 1 --2- 172.21.15.2:0/3480614880 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f44fc0a1340 0x7f44fc0a37f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f450c009990 tx=0x7f450c005fb0 comp rx=0 tx=0).stop 2023-11-16T17:00:50.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.213+0000 7f4517bff700 1 -- 172.21.15.2:0/3480614880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4510101660 msgr2=0x7f451019e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:50.216 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.213+0000 7f4517bff700 1 --2- 172.21.15.2:0/3480614880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4510101660 0x7f451019e990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f450000e910 tx=0x7f450000ec20 comp rx=0 tx=0).stop 2023-11-16T17:00:50.217 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.214+0000 7f4517bff700 1 -- 172.21.15.2:0/3480614880 shutdown_connections 2023-11-16T17:00:50.217 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.214+0000 7f4517bff700 1 --2- 172.21.15.2:0/3480614880 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f44fc0a1340 0x7f44fc0a37f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:50.217 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.214+0000 7f4517bff700 1 --2- 172.21.15.2:0/3480614880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4510105620 0x7f4510198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:50.218 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.214+0000 7f4517bff700 1 --2- 172.21.15.2:0/3480614880 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f4510100cc0 0x7f451019e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:50.218 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.214+0000 7f4517bff700 1 --2- 172.21.15.2:0/3480614880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4510101660 0x7f451019e990 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:50.218 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.214+0000 7f4517bff700 1 -- 172.21.15.2:0/3480614880 >> 172.21.15.2:0/3480614880 conn(0x7f4510077560 msgr2=0x7f45100feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:50.218 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.214+0000 7f4517bff700 1 -- 172.21.15.2:0/3480614880 shutdown_connections 2023-11-16T17:00:50.218 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.214+0000 7f4517bff700 1 -- 172.21.15.2:0/3480614880 wait complete. 2023-11-16T17:00:50.219 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data' removed 2023-11-16T17:00:50.244 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-11-16T17:00:50.462 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.457+0000 7f686efff700 1 -- 172.21.15.2:0/693399624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6868100cc0 msgr2=0x7f6868101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:50.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.457+0000 7f686efff700 1 --2- 172.21.15.2:0/693399624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6868100cc0 0x7f6868101090 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f685c0099c0 tx=0x7f685c009cd0 comp rx=0 tx=0).stop 2023-11-16T17:00:50.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.457+0000 7f686efff700 1 -- 172.21.15.2:0/693399624 shutdown_connections 2023-11-16T17:00:50.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.457+0000 7f686efff700 1 --2- 172.21.15.2:0/693399624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6868105620 0x7f686810d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:50.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.457+0000 7f686efff700 1 --2- 172.21.15.2:0/693399624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6868100cc0 0x7f6868101090 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:50.463 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.457+0000 7f686efff700 1 --2- 172.21.15.2:0/693399624 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6868101660 0x7f6868104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:50.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.457+0000 7f686efff700 1 -- 172.21.15.2:0/693399624 >> 172.21.15.2:0/693399624 conn(0x7f6868077560 msgr2=0x7f6868077960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:50.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.457+0000 7f686efff700 1 -- 172.21.15.2:0/693399624 shutdown_connections 2023-11-16T17:00:50.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.457+0000 7f686efff700 1 -- 172.21.15.2:0/693399624 wait complete. 2023-11-16T17:00:50.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.458+0000 7f686efff700 1 Processor -- start 2023-11-16T17:00:50.464 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.458+0000 7f686efff700 1 -- start start 2023-11-16T17:00:50.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.459+0000 7f686efff700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6868100cc0 0x7f6868198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:50.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.459+0000 7f686efff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6868101660 0x7f6868198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:50.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.459+0000 7f686efff700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6868105620 0x7f686819d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:50.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.459+0000 7f686efff700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686819d660 con 0x7f6868105620 2023-11-16T17:00:50.465 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.459+0000 7f686efff700 1 -- --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686819d7d0 con 0x7f6868100cc0 2023-11-16T17:00:50.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.459+0000 7f686efff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686819d990 con 0x7f6868101660 2023-11-16T17:00:50.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.459+0000 7f686cd9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6868100cc0 0x7f6868198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:50.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.459+0000 7f686cd9b700 1 --2- >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6868100cc0 0x7f6868198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.141:3300/0 says I am v2:172.21.15.2:50700/0 (socket says 172.21.15.2:50700) 2023-11-16T17:00:50.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.459+0000 7f686cd9b700 1 -- 172.21.15.2:0/3846624792 learned_addr learned my addr 172.21.15.2:0/3846624792 (peer_addr_for_me v2:172.21.15.2:0/0) 2023-11-16T17:00:50.466 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.459+0000 7f686d59c700 1 --2- 172.21.15.2:0/3846624792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6868105620 0x7f686819d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:50.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.459+0000 7f6867fff700 1 --2- 172.21.15.2:0/3846624792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6868101660 0x7f6868198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:50.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.459+0000 7f686cd9b700 1 -- 172.21.15.2:0/3846624792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6868101660 msgr2=0x7f6868198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:50.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.459+0000 7f686cd9b700 1 --2- 172.21.15.2:0/3846624792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6868101660 0x7f6868198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:50.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.459+0000 7f686cd9b700 1 -- 172.21.15.2:0/3846624792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6868105620 msgr2=0x7f686819d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:50.467 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.459+0000 7f686cd9b700 1 --2- 172.21.15.2:0/3846624792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6868105620 0x7f686819d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:50.468 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.459+0000 7f686cd9b700 1 -- 172.21.15.2:0/3846624792 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f685c0095f0 con 0x7f6868100cc0 2023-11-16T17:00:50.468 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.460+0000 7f686cd9b700 1 --2- 172.21.15.2:0/3846624792 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6868100cc0 0x7f6868198680 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f685c00b580 tx=0x7f685c00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:50.468 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.460+0000 7f6865ffb700 1 -- 172.21.15.2:0/3846624792 <== mon.2 v2:172.21.15.141:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c01d070 con 0x7f6868100cc0 2023-11-16T17:00:50.468 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.460+0000 7f686efff700 1 -- 172.21.15.2:0/3846624792 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68681103d0 con 0x7f6868100cc0 2023-11-16T17:00:50.468 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.460+0000 7f6865ffb700 1 -- 172.21.15.2:0/3846624792 <== mon.2 v2:172.21.15.141:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f685c005050 con 0x7f6868100cc0 2023-11-16T17:00:50.468 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.460+0000 7f6865ffb700 1 -- 172.21.15.2:0/3846624792 <== mon.2 v2:172.21.15.141:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c004a40 con 0x7f6868100cc0 2023-11-16T17:00:50.469 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.460+0000 7f686efff700 1 -- 172.21.15.2:0/3846624792 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6868110240 con 0x7f6868100cc0 2023-11-16T17:00:50.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.461+0000 7f686efff700 1 -- 172.21.15.2:0/3846624792 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6868110560 con 0x7f6868100cc0 2023-11-16T17:00:50.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.463+0000 7f6865ffb700 1 -- 172.21.15.2:0/3846624792 <== mon.2 v2:172.21.15.141:3300/0 4 ==== mgrmap(e 25) v1 ==== 135845+0+0 (secure 0 0 0) 0x7f685c005430 con 0x7f6868100cc0 2023-11-16T17:00:50.470 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.465+0000 7f6865ffb700 1 --2- 172.21.15.2:0/3846624792 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f68580a12c0 0x7f68580a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-16T17:00:50.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.465+0000 7f6865ffb700 1 -- 172.21.15.2:0/3846624792 <== mon.2 v2:172.21.15.141:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8510+0+0 (secure 0 0 0) 0x7f685c0cdf60 con 0x7f6868100cc0 2023-11-16T17:00:50.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.465+0000 7f6867fff700 1 --2- 172.21.15.2:0/3846624792 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f68580a12c0 0x7f68580a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-16T17:00:50.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.466+0000 7f6867fff700 1 --2- 172.21.15.2:0/3846624792 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f68580a12c0 0x7f68580a3770 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6868199c90 tx=0x7f6854009260 comp rx=0 tx=0).ready entity=mgr.15372 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-16T17:00:50.471 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.467+0000 7f6865ffb700 1 -- 172.21.15.2:0/3846624792 <== mon.2 v2:172.21.15.141:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f685c0973b0 con 0x7f6868100cc0 2023-11-16T17:00:50.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:50 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/2502517820' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-11-16T17:00:50.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:50 smithi141 ceph-mon[131184]: osdmap e107: 12 total, 12 up, 12 in 2023-11-16T17:00:50.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:50 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3480614880' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:00:50.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:50 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/2502517820' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-11-16T17:00:50.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:50 smithi155 ceph-mon[131607]: osdmap e107: 12 total, 12 up, 12 in 2023-11-16T17:00:50.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:50 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3480614880' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:00:50.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:50 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/2502517820' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-11-16T17:00:50.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:50 smithi002 ceph-mon[128712]: osdmap e107: 12 total, 12 up, 12 in 2023-11-16T17:00:50.656 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:50 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3480614880' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:00:50.712 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:50.710+0000 7f686efff700 1 -- 172.21.15.2:0/3846624792 --> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6868199a70 con 0x7f6868100cc0 2023-11-16T17:00:51.217 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:51.215+0000 7f6865ffb700 1 -- 172.21.15.2:0/3846624792 <== mon.2 v2:172.21.15.141:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v109) v1 ==== 178+0+0 (secure 0 0 0) 0x7f685c09a9d0 con 0x7f6868100cc0 2023-11-16T17:00:51.221 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:51.219+0000 7f686efff700 1 -- 172.21.15.2:0/3846624792 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f68580a12c0 msgr2=0x7f68580a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:51.222 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:51.219+0000 7f686efff700 1 --2- 172.21.15.2:0/3846624792 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f68580a12c0 0x7f68580a3770 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6868199c90 tx=0x7f6854009260 comp rx=0 tx=0).stop 2023-11-16T17:00:51.222 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:51.219+0000 7f686efff700 1 -- 172.21.15.2:0/3846624792 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6868100cc0 msgr2=0x7f6868198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-16T17:00:51.222 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:51.219+0000 7f686efff700 1 --2- 172.21.15.2:0/3846624792 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6868100cc0 0x7f6868198680 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f685c00b580 tx=0x7f685c00f690 comp rx=0 tx=0).stop 2023-11-16T17:00:51.222 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:51.219+0000 7f686efff700 1 -- 172.21.15.2:0/3846624792 shutdown_connections 2023-11-16T17:00:51.223 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:51.219+0000 7f686efff700 1 --2- 172.21.15.2:0/3846624792 >> [v2:172.21.15.2:6800/1352003968,v1:172.21.15.2:6801/1352003968] conn(0x7f68580a12c0 0x7f68580a3770 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:51.223 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:51.219+0000 7f686efff700 1 --2- 172.21.15.2:0/3846624792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6868105620 0x7f686819d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:51.223 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:51.219+0000 7f686efff700 1 --2- 172.21.15.2:0/3846624792 >> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] conn(0x7f6868100cc0 0x7f6868198680 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:51.223 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:51.219+0000 7f686efff700 1 --2- 172.21.15.2:0/3846624792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6868101660 0x7f6868198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-16T17:00:51.223 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:51.219+0000 7f686efff700 1 -- 172.21.15.2:0/3846624792 >> 172.21.15.2:0/3846624792 conn(0x7f6868077560 msgr2=0x7f68680fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-16T17:00:51.224 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:51.219+0000 7f686efff700 1 -- 172.21.15.2:0/3846624792 shutdown_connections 2023-11-16T17:00:51.224 INFO:teuthology.orchestra.run.smithi002.stderr:2023-11-16T17:00:51.220+0000 7f686efff700 1 -- 172.21.15.2:0/3846624792 wait complete. 2023-11-16T17:00:51.224 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data_ec' removed 2023-11-16T17:00:51.251 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-16T17:00:51.341 DEBUG:teuthology.orchestra.run.smithi141:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-16T17:00:51.381 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-16T17:00:51.414 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-11-16T17:00:51.414 DEBUG:teuthology.orchestra.run.smithi002:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-16T17:00:51.458 DEBUG:teuthology.orchestra.run.smithi141:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-16T17:00:51.476 DEBUG:teuthology.orchestra.run.smithi155:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-16T17:00:51.494 INFO:tasks.cephadm:Stopping all daemons... 2023-11-16T17:00:51.494 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-11-16T17:00:51.494 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mon.a 2023-11-16T17:00:51.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:51 smithi141 ceph-mon[131184]: pgmap v343: 97 pgs: 97 active+clean; 577 KiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T17:00:51.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:51 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3480614880' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-11-16T17:00:51.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:51 smithi141 ceph-mon[131184]: osdmap e108: 12 total, 12 up, 12 in 2023-11-16T17:00:51.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:51 smithi141 ceph-mon[131184]: from='client.? 172.21.15.2:0/3846624792' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:00:51.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:51 smithi141 ceph-mon[131184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:00:51.559 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:51 smithi002 ceph-mon[128712]: pgmap v343: 97 pgs: 97 active+clean; 577 KiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T17:00:51.560 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:51 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3480614880' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-11-16T17:00:51.560 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:51 smithi002 ceph-mon[128712]: osdmap e108: 12 total, 12 up, 12 in 2023-11-16T17:00:51.560 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:51 smithi002 ceph-mon[128712]: from='client.? 172.21.15.2:0/3846624792' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:00:51.560 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:51 smithi002 ceph-mon[128712]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:00:51.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:51 smithi155 ceph-mon[131607]: pgmap v343: 97 pgs: 97 active+clean; 577 KiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail 2023-11-16T17:00:51.617 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:51 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3480614880' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-11-16T17:00:51.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:51 smithi155 ceph-mon[131607]: osdmap e108: 12 total, 12 up, 12 in 2023-11-16T17:00:51.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:51 smithi155 ceph-mon[131607]: from='client.? 172.21.15.2:0/3846624792' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:00:51.618 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:51 smithi155 ceph-mon[131607]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-11-16T17:00:51.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:51 smithi002 systemd[1]: Stopping Ceph mon.a for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:00:52.364 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:52 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mon-a[128688]: 2023-11-16T17:00:52.100+0000 7fb474125700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:00:52.364 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:52 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mon-a[128688]: 2023-11-16T17:00:52.100+0000 7fb474125700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-11-16T17:00:52.655 INFO:journalctl@ceph.mon.a.smithi002.stdout:Nov 16 17:00:52 smithi002 bash[287858]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mon-a 2023-11-16T17:00:53.015 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mon.a.service' 2023-11-16T17:00:53.140 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:00:53.140 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-11-16T17:00:53.141 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-11-16T17:00:53.141 DEBUG:teuthology.orchestra.run.smithi141:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mon.b 2023-11-16T17:00:53.515 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:53 smithi141 systemd[1]: Stopping Ceph mon.b for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:00:53.947 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:53 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mon-b[131160]: 2023-11-16T17:00:53.536+0000 7fb10d650700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:00:53.948 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:53 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mon-b[131160]: 2023-11-16T17:00:53.536+0000 7fb10d650700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-11-16T17:00:54.265 INFO:journalctl@ceph.mon.b.smithi141.stdout:Nov 16 17:00:53 smithi141 bash[162493]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mon-b 2023-11-16T17:00:54.347 DEBUG:teuthology.orchestra.run.smithi141:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mon.b.service' 2023-11-16T17:00:54.399 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:00:54.399 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-11-16T17:00:54.399 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-11-16T17:00:54.399 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mon.c 2023-11-16T17:00:54.747 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:54 smithi155 systemd[1]: Stopping Ceph mon.c for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:00:54.748 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:54 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mon-c[131565]: 2023-11-16T17:00:54.744+0000 7f229ea7b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:00:54.748 INFO:journalctl@ceph.mon.c.smithi155.stdout:Nov 16 17:00:54 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mon-c[131565]: 2023-11-16T17:00:54.744+0000 7f229ea7b700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-11-16T17:00:55.398 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mon.c.service' 2023-11-16T17:00:55.461 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:00:55.462 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-11-16T17:00:55.462 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-11-16T17:00:55.462 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mgr.x 2023-11-16T17:00:55.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 17:00:55 smithi002 systemd[1]: Stopping Ceph mgr.x for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:00:57.655 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 17:00:57 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x[128959]: 2023-11-16T17:00:57.395+0000 7fa64a8ec700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:00:58.905 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Nov 16 17:00:58 smithi002 bash[288005]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-x 2023-11-16T17:00:59.047 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mgr.x.service' 2023-11-16T17:00:59.136 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:00:59.136 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-11-16T17:00:59.136 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-11-16T17:00:59.136 DEBUG:teuthology.orchestra.run.smithi141:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mgr.y 2023-11-16T17:00:59.514 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 17:00:59 smithi141 systemd[1]: Stopping Ceph mgr.y for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:00:59.849 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 17:00:59 smithi141 bash[162640]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-y 2023-11-16T17:01:00.030 INFO:journalctl@ceph.mgr.y.smithi141.stdout:Nov 16 17:00:59 smithi141 systemd[1]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-11-16T17:01:00.035 DEBUG:teuthology.orchestra.run.smithi141:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mgr.y.service' 2023-11-16T17:01:00.072 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:01:00.072 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-11-16T17:01:00.072 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-11-16T17:01:00.073 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mgr.z 2023-11-16T17:01:00.367 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 17:01:00 smithi155 systemd[1]: Stopping Ceph mgr.z for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:01:01.486 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 17:01:01 smithi155 bash[164333]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mgr-z 2023-11-16T17:01:01.487 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Nov 16 17:01:01 smithi155 systemd[1]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-11-16T17:01:01.495 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mgr.z.service' 2023-11-16T17:01:01.533 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:01:01.534 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-11-16T17:01:01.534 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-11-16T17:01:01.534 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.0 2023-11-16T17:01:01.655 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 17:01:01 smithi002 systemd[1]: Stopping Ceph osd.0 for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:01:02.405 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 17:01:02 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-0[140855]: 2023-11-16T17:01:02.073+0000 7f67918d5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:01:02.405 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 17:01:02 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-0[140855]: 2023-11-16T17:01:02.073+0000 7f67918d5700 -1 osd.0 109 *** Got signal Terminated *** 2023-11-16T17:01:02.405 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 17:01:02 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-0[140855]: 2023-11-16T17:01:02.073+0000 7f67918d5700 -1 osd.0 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:01:08.156 INFO:journalctl@ceph.osd.0.smithi002.stdout:Nov 16 17:01:07 smithi002 bash[288163]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-0 2023-11-16T17:01:11.104 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.0.service' 2023-11-16T17:01:11.197 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:01:11.197 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-11-16T17:01:11.198 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-11-16T17:01:11.198 DEBUG:teuthology.orchestra.run.smithi141:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.1 2023-11-16T17:01:11.514 INFO:journalctl@ceph.osd.1.smithi141.stdout:Nov 16 17:01:11 smithi141 systemd[1]: Stopping Ceph osd.1 for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:01:12.015 INFO:journalctl@ceph.osd.1.smithi141.stdout:Nov 16 17:01:11 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-1[136252]: 2023-11-16T17:01:11.576+0000 7f45f3d9b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:01:12.015 INFO:journalctl@ceph.osd.1.smithi141.stdout:Nov 16 17:01:11 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-1[136252]: 2023-11-16T17:01:11.576+0000 7f45f3d9b700 -1 osd.1 109 *** Got signal Terminated *** 2023-11-16T17:01:12.015 INFO:journalctl@ceph.osd.1.smithi141.stdout:Nov 16 17:01:11 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-1[136252]: 2023-11-16T17:01:11.576+0000 7f45f3d9b700 -1 osd.1 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:01:17.439 INFO:journalctl@ceph.osd.1.smithi141.stdout:Nov 16 17:01:17 smithi141 bash[162799]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-1 2023-11-16T17:01:18.697 DEBUG:teuthology.orchestra.run.smithi141:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.1.service' 2023-11-16T17:01:18.737 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:01:18.737 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-11-16T17:01:18.737 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-11-16T17:01:18.737 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.2 2023-11-16T17:01:19.092 INFO:journalctl@ceph.osd.2.smithi155.stdout:Nov 16 17:01:18 smithi155 systemd[1]: Stopping Ceph osd.2 for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:01:19.367 INFO:journalctl@ceph.osd.2.smithi155.stdout:Nov 16 17:01:19 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-2[137415]: 2023-11-16T17:01:19.089+0000 7f4277fef700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:01:19.367 INFO:journalctl@ceph.osd.2.smithi155.stdout:Nov 16 17:01:19 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-2[137415]: 2023-11-16T17:01:19.089+0000 7f4277fef700 -1 osd.2 109 *** Got signal Terminated *** 2023-11-16T17:01:19.368 INFO:journalctl@ceph.osd.2.smithi155.stdout:Nov 16 17:01:19 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-2[137415]: 2023-11-16T17:01:19.089+0000 7f4277fef700 -1 osd.2 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:01:25.117 INFO:journalctl@ceph.osd.2.smithi155.stdout:Nov 16 17:01:24 smithi155 bash[164495]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-2 2023-11-16T17:01:26.027 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.2.service' 2023-11-16T17:01:26.067 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:01:26.067 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-11-16T17:01:26.067 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-11-16T17:01:26.068 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.3 2023-11-16T17:01:26.405 INFO:journalctl@ceph.osd.3.smithi002.stdout:Nov 16 17:01:26 smithi002 systemd[1]: Stopping Ceph osd.3 for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:01:26.905 INFO:journalctl@ceph.osd.3.smithi002.stdout:Nov 16 17:01:26 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-3[146408]: 2023-11-16T17:01:26.603+0000 7fbd7617d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:01:26.905 INFO:journalctl@ceph.osd.3.smithi002.stdout:Nov 16 17:01:26 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-3[146408]: 2023-11-16T17:01:26.603+0000 7fbd7617d700 -1 osd.3 109 *** Got signal Terminated *** 2023-11-16T17:01:26.905 INFO:journalctl@ceph.osd.3.smithi002.stdout:Nov 16 17:01:26 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-3[146408]: 2023-11-16T17:01:26.603+0000 7fbd7617d700 -1 osd.3 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:01:28.655 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:28 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:28.430+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:28.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:28 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:28.249+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:29.655 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:29 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:29.383+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:29.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:29 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:29.211+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:30.117 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:29 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:29.744+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:30.155 INFO:journalctl@ceph.osd.3.smithi002.stdout:Nov 16 17:01:29 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-3[146408]: 2023-11-16T17:01:29.867+0000 7fbd71771700 -1 osd.3 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.735631+0000 front 2023-11-16T17:01:05.735681+0000 (oldest deadline 2023-11-16T17:01:29.235744+0000) 2023-11-16T17:01:30.655 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:30 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:30.373+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:30.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:30 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:30.214+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:30.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:30 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:30.293+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:31.117 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:30 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:30.788+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:31.155 INFO:journalctl@ceph.osd.3.smithi002.stdout:Nov 16 17:01:30 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-3[146408]: 2023-11-16T17:01:30.876+0000 7fbd71771700 -1 osd.3 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.735631+0000 front 2023-11-16T17:01:05.735681+0000 (oldest deadline 2023-11-16T17:01:29.235744+0000) 2023-11-16T17:01:31.655 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:31 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:31.407+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:31.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:31 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:31.255+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:31.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:31 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:31.302+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:32.094 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:31 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:31.838+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:32.367 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:32 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:32.093+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.412907+0000 front 2023-11-16T17:01:05.412793+0000 (oldest deadline 2023-11-16T17:01:31.312793+0000) 2023-11-16T17:01:32.524 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:32 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:32.407+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:32.525 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:32 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:32.231+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:32.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:32 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:32.344+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:32.905 INFO:journalctl@ceph.osd.3.smithi002.stdout:Nov 16 17:01:32 smithi002 bash[288450]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-3 2023-11-16T17:01:33.117 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:33 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:33.064+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.412907+0000 front 2023-11-16T17:01:05.412793+0000 (oldest deadline 2023-11-16T17:01:31.312793+0000) 2023-11-16T17:01:33.118 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:32 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:32.814+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:33.449 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:33 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:33.436+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:33.449 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:33 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:33.196+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:33.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:33 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:33.319+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:34.117 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:34 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:34.058+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.412907+0000 front 2023-11-16T17:01:05.412793+0000 (oldest deadline 2023-11-16T17:01:31.312793+0000) 2023-11-16T17:01:34.118 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:33 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:33.852+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:34.338 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.3.service' 2023-11-16T17:01:34.432 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:01:34.432 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-11-16T17:01:34.432 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-11-16T17:01:34.433 DEBUG:teuthology.orchestra.run.smithi141:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.4 2023-11-16T17:01:34.655 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:34 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:34.420+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:34.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:34 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:34.225+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:34.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:34 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:34.363+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:34.765 INFO:journalctl@ceph.osd.4.smithi141.stdout:Nov 16 17:01:34 smithi141 systemd[1]: Stopping Ceph osd.4 for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:01:35.264 INFO:journalctl@ceph.osd.4.smithi141.stdout:Nov 16 17:01:34 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-4[141721]: 2023-11-16T17:01:34.786+0000 7f257f8b9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:01:35.264 INFO:journalctl@ceph.osd.4.smithi141.stdout:Nov 16 17:01:34 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-4[141721]: 2023-11-16T17:01:34.786+0000 7f257f8b9700 -1 osd.4 109 *** Got signal Terminated *** 2023-11-16T17:01:35.265 INFO:journalctl@ceph.osd.4.smithi141.stdout:Nov 16 17:01:34 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-4[141721]: 2023-11-16T17:01:34.786+0000 7f257f8b9700 -1 osd.4 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:01:35.367 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:35 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:35.017+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.412907+0000 front 2023-11-16T17:01:05.412793+0000 (oldest deadline 2023-11-16T17:01:31.312793+0000) 2023-11-16T17:01:35.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:34 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:34.867+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:35.655 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:35 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:35.389+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:35.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:35 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:35.205+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:35.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:35 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:35.408+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:36.367 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:35 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:35.997+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.412907+0000 front 2023-11-16T17:01:05.412793+0000 (oldest deadline 2023-11-16T17:01:31.312793+0000) 2023-11-16T17:01:36.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:35 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:35.886+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:36.655 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:36 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:36.428+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:36.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:36 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:36.218+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:36.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:36 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:36.367+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:37.367 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:36 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:36.960+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.412907+0000 front 2023-11-16T17:01:05.412793+0000 (oldest deadline 2023-11-16T17:01:31.312793+0000) 2023-11-16T17:01:37.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:36 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:36.925+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:37.655 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:37 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:37.397+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:37.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:37 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:37.207+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:37.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:37 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:37.392+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:37.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:37 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:37.392+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:38.367 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:38 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:38.010+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.412907+0000 front 2023-11-16T17:01:05.412793+0000 (oldest deadline 2023-11-16T17:01:31.312793+0000) 2023-11-16T17:01:38.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:37 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:37.912+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:38.434 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:38 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:38.182+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:38.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:38 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:38.430+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:38.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:38 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:38.430+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:38.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:38 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:38.432+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:38.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:38 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:38.432+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.997549+0000 front 2023-11-16T17:01:15.997475+0000 (oldest deadline 2023-11-16T17:01:38.297688+0000) 2023-11-16T17:01:39.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:38 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:38.932+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:39.368 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:39 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:39.010+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.412907+0000 front 2023-11-16T17:01:05.412793+0000 (oldest deadline 2023-11-16T17:01:31.312793+0000) 2023-11-16T17:01:39.368 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:39 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:39.010+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.413921+0000 front 2023-11-16T17:01:15.413872+0000 (oldest deadline 2023-11-16T17:01:38.913950+0000) 2023-11-16T17:01:39.405 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:39 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:39.135+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:39.703 INFO:journalctl@ceph.osd.4.smithi141.stdout:Nov 16 17:01:39 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-4[141721]: 2023-11-16T17:01:39.453+0000 7f2578ea9700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:14.708878+0000 front 2023-11-16T17:01:14.708837+0000 (oldest deadline 2023-11-16T17:01:38.809144+0000) 2023-11-16T17:01:39.703 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:39 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:39.399+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:39.703 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:39 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:39.399+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:39.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:39 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:39.461+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:39.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:39 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:39.461+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.997549+0000 front 2023-11-16T17:01:15.997475+0000 (oldest deadline 2023-11-16T17:01:38.297688+0000) 2023-11-16T17:01:40.014 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:39 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:39.701+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:40.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:39 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:39.959+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:40.368 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:39 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:39.987+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.412907+0000 front 2023-11-16T17:01:05.412793+0000 (oldest deadline 2023-11-16T17:01:31.312793+0000) 2023-11-16T17:01:40.368 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:39 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:39.987+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.413921+0000 front 2023-11-16T17:01:15.413872+0000 (oldest deadline 2023-11-16T17:01:38.913950+0000) 2023-11-16T17:01:40.405 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:40 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:40.129+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:40.405 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:40 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:40.129+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:40.566 INFO:journalctl@ceph.osd.4.smithi141.stdout:Nov 16 17:01:40 smithi141 bash[163060]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-4 2023-11-16T17:01:40.567 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:40 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:40.406+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:40.567 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:40 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:40.406+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:40.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:40 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:40.421+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:40.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:40 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:40.421+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.997549+0000 front 2023-11-16T17:01:15.997475+0000 (oldest deadline 2023-11-16T17:01:38.297688+0000) 2023-11-16T17:01:41.015 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:40 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:40.724+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:41.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:40 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:40.926+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:41.368 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:40 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:40.979+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.412907+0000 front 2023-11-16T17:01:05.412793+0000 (oldest deadline 2023-11-16T17:01:31.312793+0000) 2023-11-16T17:01:41.368 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:40 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:40.979+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.413921+0000 front 2023-11-16T17:01:15.413872+0000 (oldest deadline 2023-11-16T17:01:38.913950+0000) 2023-11-16T17:01:41.386 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:41 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:41.384+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:41.655 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:41 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:41.400+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:41.655 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:41 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:41.400+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.997549+0000 front 2023-11-16T17:01:15.997475+0000 (oldest deadline 2023-11-16T17:01:38.297688+0000) 2023-11-16T17:01:41.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:41 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:41.154+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:41.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:41 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:41.154+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:41.735 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:41 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:41.384+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:41.826 DEBUG:teuthology.orchestra.run.smithi141:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.4.service' 2023-11-16T17:01:41.866 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:01:41.866 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-11-16T17:01:41.867 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-11-16T17:01:41.867 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.5 2023-11-16T17:01:41.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:41 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:41.376+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:01:42.014 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:41 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:41.733+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:42.222 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:41 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:41.976+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:42.222 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:41 smithi155 systemd[1]: Stopping Ceph osd.5 for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:01:42.222 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:41 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:41.972+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.412907+0000 front 2023-11-16T17:01:05.412793+0000 (oldest deadline 2023-11-16T17:01:31.312793+0000) 2023-11-16T17:01:42.223 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:41 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:41.972+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.413921+0000 front 2023-11-16T17:01:15.413872+0000 (oldest deadline 2023-11-16T17:01:38.913950+0000) 2023-11-16T17:01:42.617 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:42 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:42.382+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:01:43.375 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:42 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:42.221+0000 7fbe39a64700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:01:43.375 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:42 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:42.221+0000 7fbe39a64700 -1 osd.5 109 *** Got signal Terminated *** 2023-11-16T17:01:43.375 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:42 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:42.221+0000 7fbe39a64700 -1 osd.5 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:01:43.376 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:42 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:42.972+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:43.376 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:42 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:42.966+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.412907+0000 front 2023-11-16T17:01:05.412793+0000 (oldest deadline 2023-11-16T17:01:31.312793+0000) 2023-11-16T17:01:43.376 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:42 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:42.966+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.413921+0000 front 2023-11-16T17:01:15.413872+0000 (oldest deadline 2023-11-16T17:01:38.913950+0000) 2023-11-16T17:01:43.377 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:42 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:42.418+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:43.377 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:42 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:42.418+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:43.378 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:42 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:42.404+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:43.378 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:42 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:42.404+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.997549+0000 front 2023-11-16T17:01:15.997475+0000 (oldest deadline 2023-11-16T17:01:38.297688+0000) 2023-11-16T17:01:43.378 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:42 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:42.696+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:43.379 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:42 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:42.203+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:43.379 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:42 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:42.203+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:43.656 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:43 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:43.432+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:43.656 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:43 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:43.432+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.997549+0000 front 2023-11-16T17:01:15.997475+0000 (oldest deadline 2023-11-16T17:01:38.297688+0000) 2023-11-16T17:01:43.657 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:43 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:43.217+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:43.657 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:43 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:43.217+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:43.696 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:43 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:43.388+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:43.696 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:43 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:43.388+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:43.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:43 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:43.426+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:01:44.014 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:43 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:43.694+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:44.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:44 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:44.012+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:44.368 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:43 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:43.958+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.412907+0000 front 2023-11-16T17:01:05.412793+0000 (oldest deadline 2023-11-16T17:01:31.312793+0000) 2023-11-16T17:01:44.369 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:43 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:43.958+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.413921+0000 front 2023-11-16T17:01:15.413872+0000 (oldest deadline 2023-11-16T17:01:38.913950+0000) 2023-11-16T17:01:44.655 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:44 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:44.429+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:44.656 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:44 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:44.429+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.997549+0000 front 2023-11-16T17:01:15.997475+0000 (oldest deadline 2023-11-16T17:01:38.297688+0000) 2023-11-16T17:01:44.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:44 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:44.219+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:44.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:44 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:44.219+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:44.727 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:44 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:44.423+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:44.728 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:44 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:44.423+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:44.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:44 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:44.405+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:01:45.015 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:44 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:44.726+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:45.367 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:44 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:44.974+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.412907+0000 front 2023-11-16T17:01:05.412793+0000 (oldest deadline 2023-11-16T17:01:31.312793+0000) 2023-11-16T17:01:45.368 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:44 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:44.974+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.413921+0000 front 2023-11-16T17:01:15.413872+0000 (oldest deadline 2023-11-16T17:01:38.913950+0000) 2023-11-16T17:01:45.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:45 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:45.044+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:45.655 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:45 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:45.456+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:45.656 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:45 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:45.456+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.997549+0000 front 2023-11-16T17:01:15.997475+0000 (oldest deadline 2023-11-16T17:01:38.297688+0000) 2023-11-16T17:01:45.656 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:45 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:45.456+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:18.298713+0000 front 2023-11-16T17:01:18.298410+0000 (oldest deadline 2023-11-16T17:01:44.699236+0000) 2023-11-16T17:01:45.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:45 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:45.246+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:45.657 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:45 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:45.246+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:45.756 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:45 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:45.458+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:45.756 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:45 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:45.459+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:45.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:45 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:45.430+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:01:46.015 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:45 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:45.755+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:46.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:46 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:46.060+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:46.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:46 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:46.060+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:01:46.369 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:45 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:45.947+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.412907+0000 front 2023-11-16T17:01:05.412793+0000 (oldest deadline 2023-11-16T17:01:31.312793+0000) 2023-11-16T17:01:46.369 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:45 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:45.947+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.413921+0000 front 2023-11-16T17:01:15.413872+0000 (oldest deadline 2023-11-16T17:01:38.913950+0000) 2023-11-16T17:01:46.370 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:45 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:45.947+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:21.214732+0000 front 2023-11-16T17:01:21.214750+0000 (oldest deadline 2023-11-16T17:01:45.314963+0000) 2023-11-16T17:01:46.655 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:46 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:46.462+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:46.655 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:46 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:46.462+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.997549+0000 front 2023-11-16T17:01:15.997475+0000 (oldest deadline 2023-11-16T17:01:38.297688+0000) 2023-11-16T17:01:46.655 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:46 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:46.462+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:18.298713+0000 front 2023-11-16T17:01:18.298410+0000 (oldest deadline 2023-11-16T17:01:44.699236+0000) 2023-11-16T17:01:46.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:46 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:46.274+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:46.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:46 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:46.274+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:46.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:46 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:46.274+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:01:46.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:46 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:46.447+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:46.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:46 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:46.447+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:46.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:46 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:46.445+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:01:47.264 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:46 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:46.788+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:47.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:47 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:47.086+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:47.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:47 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:47.086+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:01:47.368 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:46 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:46.952+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.412907+0000 front 2023-11-16T17:01:05.412793+0000 (oldest deadline 2023-11-16T17:01:31.312793+0000) 2023-11-16T17:01:47.368 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:46 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:46.952+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.413921+0000 front 2023-11-16T17:01:15.413872+0000 (oldest deadline 2023-11-16T17:01:38.913950+0000) 2023-11-16T17:01:47.368 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:46 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5[142841]: 2023-11-16T17:01:46.952+0000 7fbe35058700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:21.214732+0000 front 2023-11-16T17:01:21.214750+0000 (oldest deadline 2023-11-16T17:01:45.314963+0000) 2023-11-16T17:01:47.655 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:47 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:47.445+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:47.656 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:47 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:47.445+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.997549+0000 front 2023-11-16T17:01:15.997475+0000 (oldest deadline 2023-11-16T17:01:38.297688+0000) 2023-11-16T17:01:47.656 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:47 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:47.445+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:18.298713+0000 front 2023-11-16T17:01:18.298410+0000 (oldest deadline 2023-11-16T17:01:44.699236+0000) 2023-11-16T17:01:47.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:47 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:47.225+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:47.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:47 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:47.225+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:47.657 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:47 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:47.225+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:01:47.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:47 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:47.465+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:47.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:47 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:47.465+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:47.853 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:47 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:47.470+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:01:47.854 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:47 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:47.470+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:01:48.117 INFO:journalctl@ceph.osd.5.smithi155.stdout:Nov 16 17:01:47 smithi155 bash[164774]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-5 2023-11-16T17:01:48.118 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:48 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:48.079+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:48.118 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:48 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:48.079+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:01:48.264 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:47 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:47.766+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:48.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:47 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:47.766+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:22.076160+0000 front 2023-11-16T17:01:22.076183+0000 (oldest deadline 2023-11-16T17:01:47.376079+0000) 2023-11-16T17:01:48.655 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:48 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:48.468+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:48.655 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:48 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:48.468+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.997549+0000 front 2023-11-16T17:01:15.997475+0000 (oldest deadline 2023-11-16T17:01:38.297688+0000) 2023-11-16T17:01:48.655 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:48 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:48.468+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:18.298713+0000 front 2023-11-16T17:01:18.298410+0000 (oldest deadline 2023-11-16T17:01:44.699236+0000) 2023-11-16T17:01:48.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:48 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:48.255+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:48.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:48 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:48.255+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:48.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:48 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:48.255+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:01:48.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:48 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:48.452+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:48.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:48 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:48.452+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:48.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:48 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:48.452+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:01:48.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:48 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:48.488+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:01:48.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:48 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:48.488+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:01:49.186 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:49 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:49.079+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:49.187 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:49 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:49.079+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:01:49.194 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.5.service' 2023-11-16T17:01:49.234 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:01:49.235 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-11-16T17:01:49.235 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-11-16T17:01:49.235 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.6 2023-11-16T17:01:49.264 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:48 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:48.802+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:49.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:48 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:48.802+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:22.076160+0000 front 2023-11-16T17:01:22.076183+0000 (oldest deadline 2023-11-16T17:01:47.376079+0000) 2023-11-16T17:01:49.474 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:49 smithi002 systemd[1]: Stopping Ceph osd.6 for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:01:49.474 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:49 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:49.211+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:49.474 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:49 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:49.211+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:49.475 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:49 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:49.211+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:01:49.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:49 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:49.410+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:49.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:49 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:49.410+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:49.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:49 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:49.410+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:01:49.767 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:49 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:49.472+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:49.767 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:49 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:49.472+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.997549+0000 front 2023-11-16T17:01:15.997475+0000 (oldest deadline 2023-11-16T17:01:38.297688+0000) 2023-11-16T17:01:49.768 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:49 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:49.472+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:18.298713+0000 front 2023-11-16T17:01:18.298410+0000 (oldest deadline 2023-11-16T17:01:44.699236+0000) 2023-11-16T17:01:49.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:49 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:49.452+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:01:49.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:49 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:49.452+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:01:50.155 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:49 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:49.765+0000 7f69363a6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:01:50.155 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:49 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:49.765+0000 7f69363a6700 -1 osd.6 109 *** Got signal Terminated *** 2023-11-16T17:01:50.155 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:49 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:49.765+0000 7f69363a6700 -1 osd.6 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:01:50.264 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:49 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:49.791+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:50.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:49 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:49.791+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:22.076160+0000 front 2023-11-16T17:01:22.076183+0000 (oldest deadline 2023-11-16T17:01:47.376079+0000) 2023-11-16T17:01:50.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:50 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:50.072+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:50.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:50 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:50.072+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:01:50.498 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:50 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:50.167+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:50.498 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:50 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:50.167+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:50.498 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:50 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:50.167+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:01:50.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:50 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:50.439+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:50.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:50 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:50.439+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:50.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:50 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:50.439+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:01:50.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:50 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:50.461+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:01:50.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:50 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:50.461+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:01:50.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:50 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:50.496+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:50.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:50 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:50.496+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.997549+0000 front 2023-11-16T17:01:15.997475+0000 (oldest deadline 2023-11-16T17:01:38.297688+0000) 2023-11-16T17:01:50.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:50 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:50.496+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:18.298713+0000 front 2023-11-16T17:01:18.298410+0000 (oldest deadline 2023-11-16T17:01:44.699236+0000) 2023-11-16T17:01:51.264 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:50 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:50.832+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:51.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:50 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:50.832+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:22.076160+0000 front 2023-11-16T17:01:22.076183+0000 (oldest deadline 2023-11-16T17:01:47.376079+0000) 2023-11-16T17:01:51.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:51 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:51.046+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:51.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:51 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:51.046+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:01:51.503 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:51 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:51.191+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:51.503 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:51 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:51.191+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:51.503 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:51 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:51.191+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:01:51.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:51 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:51.395+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:51.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:51 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:51.395+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:51.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:51 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:51.395+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:01:51.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:51 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:51.429+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:01:51.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:51 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:51.429+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:01:51.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:51 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:51.502+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:51.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:51 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:51.502+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.997549+0000 front 2023-11-16T17:01:15.997475+0000 (oldest deadline 2023-11-16T17:01:38.297688+0000) 2023-11-16T17:01:51.906 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:51 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:51.502+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:18.298713+0000 front 2023-11-16T17:01:18.298410+0000 (oldest deadline 2023-11-16T17:01:44.699236+0000) 2023-11-16T17:01:52.264 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:51 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:51.841+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:52.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:51 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:51.841+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:22.076160+0000 front 2023-11-16T17:01:22.076183+0000 (oldest deadline 2023-11-16T17:01:47.376079+0000) 2023-11-16T17:01:52.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:52 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:52.023+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:52.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:52 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:52.023+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:01:52.471 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:52 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:52.154+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:52.471 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:52 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:52.154+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:52.472 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:52 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:52.154+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:01:52.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:52 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:52.394+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:52.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:52 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:52.394+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:52.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:52 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:52.394+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:01:52.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:52 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:52.477+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:01:52.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:52 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:52.477+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:01:52.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:52 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:52.477+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.645505+0000 front 2023-11-16T17:01:30.645605+0000 (oldest deadline 2023-11-16T17:01:52.345350+0000) 2023-11-16T17:01:52.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:52 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:52.470+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:52.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:52 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:52.470+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.997549+0000 front 2023-11-16T17:01:15.997475+0000 (oldest deadline 2023-11-16T17:01:38.297688+0000) 2023-11-16T17:01:52.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:52 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:52.470+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:18.298713+0000 front 2023-11-16T17:01:18.298410+0000 (oldest deadline 2023-11-16T17:01:44.699236+0000) 2023-11-16T17:01:53.264 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:52 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:52.869+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:53.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:52 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:52.869+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:22.076160+0000 front 2023-11-16T17:01:22.076183+0000 (oldest deadline 2023-11-16T17:01:47.376079+0000) 2023-11-16T17:01:53.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:53 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:53.031+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:53.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:53 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:53.031+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:01:53.405 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:53 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:53.151+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:53.405 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:53 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:53.151+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:53.405 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:53 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:53.151+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:01:53.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:53 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:53.414+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:53.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:53 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:53.414+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:53.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:53 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:53.414+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:01:53.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:53 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:53.459+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:01:53.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:53 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:53.459+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:01:53.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:53 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:53.459+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.645505+0000 front 2023-11-16T17:01:30.645605+0000 (oldest deadline 2023-11-16T17:01:52.345350+0000) 2023-11-16T17:01:53.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:53 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:53.510+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:53.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:53 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:53.510+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.997549+0000 front 2023-11-16T17:01:15.997475+0000 (oldest deadline 2023-11-16T17:01:38.297688+0000) 2023-11-16T17:01:53.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:53 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:53.510+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:18.298713+0000 front 2023-11-16T17:01:18.298410+0000 (oldest deadline 2023-11-16T17:01:44.699236+0000) 2023-11-16T17:01:54.264 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:53 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:53.847+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:54.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:53 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:53.847+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:22.076160+0000 front 2023-11-16T17:01:22.076183+0000 (oldest deadline 2023-11-16T17:01:47.376079+0000) 2023-11-16T17:01:54.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:53 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:53.992+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:54.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:53 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:53.992+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:01:54.550 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:54 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:54.191+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:54.550 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:54 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:54.191+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:54.551 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:54 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:54.191+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:01:54.551 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:54 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:54.191+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:01:54.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:54 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:54.452+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:54.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:54 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:54.452+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:54.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:54 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:54.452+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:01:54.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:54 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:54.430+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:01:54.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:54 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:54.430+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:01:54.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:54 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:54.430+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.645505+0000 front 2023-11-16T17:01:30.645605+0000 (oldest deadline 2023-11-16T17:01:52.345350+0000) 2023-11-16T17:01:54.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:54 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:54.549+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.894769+0000 front 2023-11-16T17:01:03.894724+0000 (oldest deadline 2023-11-16T17:01:27.994900+0000) 2023-11-16T17:01:54.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:54 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:54.549+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.997549+0000 front 2023-11-16T17:01:15.997475+0000 (oldest deadline 2023-11-16T17:01:38.297688+0000) 2023-11-16T17:01:54.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:54 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6[151950]: 2023-11-16T17:01:54.549+0000 7f692f996700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:18.298713+0000 front 2023-11-16T17:01:18.298410+0000 (oldest deadline 2023-11-16T17:01:44.699236+0000) 2023-11-16T17:01:55.264 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:54 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:54.896+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:55.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:54 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:54.896+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:22.076160+0000 front 2023-11-16T17:01:22.076183+0000 (oldest deadline 2023-11-16T17:01:47.376079+0000) 2023-11-16T17:01:55.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:55 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:55.010+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:55.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:55 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:55.010+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:01:55.531 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:55 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:55.198+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:55.532 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:55 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:55.198+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:55.532 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:55 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:55.198+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:01:55.532 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:55 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:55.198+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:01:55.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:55 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:55.465+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:55.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:55 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:55.465+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:55.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:55 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:55.465+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:01:55.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:55 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:55.389+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:01:55.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:55 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:55.389+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:01:55.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:55 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:55.389+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.645505+0000 front 2023-11-16T17:01:30.645605+0000 (oldest deadline 2023-11-16T17:01:52.345350+0000) 2023-11-16T17:01:55.905 INFO:journalctl@ceph.osd.6.smithi002.stdout:Nov 16 17:01:55 smithi002 bash[288716]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-6 2023-11-16T17:01:56.264 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:55 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:55.939+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:56.264 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:55 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:55.939+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:22.076160+0000 front 2023-11-16T17:01:22.076183+0000 (oldest deadline 2023-11-16T17:01:47.376079+0000) 2023-11-16T17:01:56.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:56 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:56.026+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:56.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:56 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:56.026+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:01:56.638 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:56 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:56.179+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:56.638 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:56 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:56.179+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:56.638 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:56 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:56.179+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:01:56.638 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:56 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:56.179+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:01:56.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:56 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:56.419+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:56.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:56 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:56.419+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:56.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:56 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:56.419+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:01:56.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:56 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:56.419+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:01:56.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:56 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:56.395+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:01:56.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:56 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:56.395+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:01:56.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:56 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:56.395+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.645505+0000 front 2023-11-16T17:01:30.645605+0000 (oldest deadline 2023-11-16T17:01:52.345350+0000) 2023-11-16T17:01:57.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:56 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:56.965+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:57.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:56 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:56.965+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:22.076160+0000 front 2023-11-16T17:01:22.076183+0000 (oldest deadline 2023-11-16T17:01:47.376079+0000) 2023-11-16T17:01:57.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:56 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:56.965+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:27.376908+0000 front 2023-11-16T17:01:27.376954+0000 (oldest deadline 2023-11-16T17:01:56.177433+0000) 2023-11-16T17:01:57.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:57 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:57.020+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:57.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:57 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:57.020+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:01:57.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:57 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:57.020+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:01:57.405 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:57 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:57.228+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:57.405 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:57 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:57.228+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:57.406 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:57 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:57.228+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:01:57.406 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:57 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:57.228+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:01:57.604 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.6.service' 2023-11-16T17:01:57.695 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:01:57.696 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-11-16T17:01:57.696 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-11-16T17:01:57.696 DEBUG:teuthology.orchestra.run.smithi141:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.7 2023-11-16T17:01:57.728 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:57 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:57.392+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:57.728 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:57 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:57.392+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:57.728 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:57 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:57.392+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:01:57.728 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:57 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:57.392+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:01:57.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:57 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:57.393+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:01:57.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:57 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:57.393+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:01:57.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:57 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:57.393+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.645505+0000 front 2023-11-16T17:01:30.645605+0000 (oldest deadline 2023-11-16T17:01:52.345350+0000) 2023-11-16T17:01:57.986 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:57 smithi141 systemd[1]: Stopping Ceph osd.7 for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:01:58.264 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:57 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:57.984+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:58.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:57 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:57.984+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:22.076160+0000 front 2023-11-16T17:01:22.076183+0000 (oldest deadline 2023-11-16T17:01:47.376079+0000) 2023-11-16T17:01:58.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:57 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:57.984+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:27.376908+0000 front 2023-11-16T17:01:27.376954+0000 (oldest deadline 2023-11-16T17:01:56.177433+0000) 2023-11-16T17:01:58.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:58 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:58.049+0000 7fd9a29c1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:01:58.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:58 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:58.049+0000 7fd9a29c1700 -1 osd.7 109 *** Got signal Terminated *** 2023-11-16T17:01:58.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:58 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:58.049+0000 7fd9a29c1700 -1 osd.7 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:01:58.344 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:58 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:58.034+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:58.345 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:58 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:58.034+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:01:58.345 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:58 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:58.034+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:01:58.617 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:58 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:58.344+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:01:58.617 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:58 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:58.344+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:01:58.618 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:58 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:58.344+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.645505+0000 front 2023-11-16T17:01:30.645605+0000 (oldest deadline 2023-11-16T17:01:52.345350+0000) 2023-11-16T17:01:58.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:58 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:58.226+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:58.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:58 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:58.226+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:58.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:58 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:58.226+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:01:58.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:58 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:58.226+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:01:58.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:58 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:58.362+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:58.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:58 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:58.362+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:58.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:58 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:58.362+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:01:58.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:58 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:58.362+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:01:59.264 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:58 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:58.998+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:01:59.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:58 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:58.998+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:22.076160+0000 front 2023-11-16T17:01:22.076183+0000 (oldest deadline 2023-11-16T17:01:47.376079+0000) 2023-11-16T17:01:59.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:58 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:58.998+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:27.376908+0000 front 2023-11-16T17:01:27.376954+0000 (oldest deadline 2023-11-16T17:01:56.177433+0000) 2023-11-16T17:01:59.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:59 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:59.048+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:01:59.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:59 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:59.048+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:01:59.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:01:59 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:01:59.048+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:01:59.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:59 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:59.251+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:01:59.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:59 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:59.251+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:01:59.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:59 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:59.251+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:01:59.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:01:59 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:01:59.251+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:01:59.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:59 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:59.347+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:01:59.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:59 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:59.347+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:01:59.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:59 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:59.347+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:01:59.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:01:59 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:01:59.347+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:01:59.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:59 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:59.371+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:01:59.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:59 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:59.371+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:01:59.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:01:59 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:01:59.371+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.645505+0000 front 2023-11-16T17:01:30.645605+0000 (oldest deadline 2023-11-16T17:01:52.345350+0000) 2023-11-16T17:02:00.264 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:59 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:59.977+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:02:00.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:59 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:59.977+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:22.076160+0000 front 2023-11-16T17:01:22.076183+0000 (oldest deadline 2023-11-16T17:01:47.376079+0000) 2023-11-16T17:02:00.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:01:59 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:01:59.977+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:27.376908+0000 front 2023-11-16T17:01:27.376954+0000 (oldest deadline 2023-11-16T17:01:56.177433+0000) 2023-11-16T17:02:00.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:00 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:00.086+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:00.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:00 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:00.086+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:00.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:00 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:00.086+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:00.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:00 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:00.265+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:02:00.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:00 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:00.265+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:02:00.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:00 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:00.265+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:02:00.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:00 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:00.265+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:02:00.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:00 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:00.388+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:00.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:00 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:00.388+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:00.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:00 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:00.388+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:00.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:00 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:00.388+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:00.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:00 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:00.372+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:02:00.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:00 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:00.372+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:02:00.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:00 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:00.372+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.645505+0000 front 2023-11-16T17:01:30.645605+0000 (oldest deadline 2023-11-16T17:01:52.345350+0000) 2023-11-16T17:02:01.353 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:01 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:01.043+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:01.353 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:01 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:01.043+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:01.353 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:01 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:01.043+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:01.353 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:01 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:01.043+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:01.355 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:02:01 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:02:01.014+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:02:01.356 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:02:01 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:02:01.014+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:22.076160+0000 front 2023-11-16T17:01:22.076183+0000 (oldest deadline 2023-11-16T17:01:47.376079+0000) 2023-11-16T17:02:01.356 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:02:01 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:02:01.014+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:27.376908+0000 front 2023-11-16T17:01:27.376954+0000 (oldest deadline 2023-11-16T17:01:56.177433+0000) 2023-11-16T17:02:01.356 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:02:01 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:02:01.014+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:36.177863+0000 front 2023-11-16T17:01:36.177883+0000 (oldest deadline 2023-11-16T17:02:00.877991+0000) 2023-11-16T17:02:01.617 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:01 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:01.352+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:02:01.617 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:01 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:01.352+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:02:01.618 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:01 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:01.352+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.645505+0000 front 2023-11-16T17:01:30.645605+0000 (oldest deadline 2023-11-16T17:01:52.345350+0000) 2023-11-16T17:02:01.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:01 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:01.218+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:02:01.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:01 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:01.218+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:02:01.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:01 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:01.218+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:02:01.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:01 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:01.218+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:02:01.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:01 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:01.355+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:01.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:01 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:01.355+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:01.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:01 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:01.355+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:01.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:01 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:01.355+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:02.264 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:02:02 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:02:02.011+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:02:02.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:02:02 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:02:02.011+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:22.076160+0000 front 2023-11-16T17:01:22.076183+0000 (oldest deadline 2023-11-16T17:01:47.376079+0000) 2023-11-16T17:02:02.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:02:02 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:02:02.011+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:27.376908+0000 front 2023-11-16T17:01:27.376954+0000 (oldest deadline 2023-11-16T17:01:56.177433+0000) 2023-11-16T17:02:02.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:02:02 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:02:02.011+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:36.177863+0000 front 2023-11-16T17:01:36.177883+0000 (oldest deadline 2023-11-16T17:02:00.877991+0000) 2023-11-16T17:02:02.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:02 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:02.064+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:02.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:02 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:02.064+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:02.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:02 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:02.064+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:02.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:02 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:02.064+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:02.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:02 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:02.210+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:02:02.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:02 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:02.210+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:02:02.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:02 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:02.210+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:02:02.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:02 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:02.210+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:02:02.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:02 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:02.210+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.131153+0000 front 2023-11-16T17:01:38.131237+0000 (oldest deadline 2023-11-16T17:02:01.631227+0000) 2023-11-16T17:02:02.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:02 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:02.394+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:02.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:02 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:02.394+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:02.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:02 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:02.394+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:02.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:02 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:02.394+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:02.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:02 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:02.392+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:02:02.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:02 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:02.392+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:02:02.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:02 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:02.392+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.645505+0000 front 2023-11-16T17:01:30.645605+0000 (oldest deadline 2023-11-16T17:01:52.345350+0000) 2023-11-16T17:02:03.264 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:02:02 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:02:02.992+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.873592+0000 front 2023-11-16T17:01:15.873620+0000 (oldest deadline 2023-11-16T17:01:39.274542+0000) 2023-11-16T17:02:03.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:02:02 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:02:02.992+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:22.076160+0000 front 2023-11-16T17:01:22.076183+0000 (oldest deadline 2023-11-16T17:01:47.376079+0000) 2023-11-16T17:02:03.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:02:02 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:02:02.992+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:27.376908+0000 front 2023-11-16T17:01:27.376954+0000 (oldest deadline 2023-11-16T17:01:56.177433+0000) 2023-11-16T17:02:03.265 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:02:02 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7[147160]: 2023-11-16T17:02:02.992+0000 7fd99cfb3700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:36.177863+0000 front 2023-11-16T17:01:36.177883+0000 (oldest deadline 2023-11-16T17:02:00.877991+0000) 2023-11-16T17:02:03.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:03 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:03.093+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:03.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:03 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:03.093+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:03.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:03 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:03.093+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:03.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:03 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:03.093+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:03.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:03 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:03.241+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:02:03.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:03 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:03.241+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:02:03.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:03 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:03.241+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:02:03.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:03 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:03.241+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:02:03.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:03 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:03.241+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.131153+0000 front 2023-11-16T17:01:38.131237+0000 (oldest deadline 2023-11-16T17:02:01.631227+0000) 2023-11-16T17:02:03.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:03 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:03.439+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:03.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:03 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:03.439+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:03.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:03 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:03.439+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:03.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:03 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:03.439+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:03.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:03 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:03.370+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:02:03.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:03 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:03.370+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:02:03.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:03 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:03.370+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.645505+0000 front 2023-11-16T17:01:30.645605+0000 (oldest deadline 2023-11-16T17:01:52.345350+0000) 2023-11-16T17:02:03.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:03 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:03.370+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.647612+0000 front 2023-11-16T17:01:38.647530+0000 (oldest deadline 2023-11-16T17:02:02.747492+0000) 2023-11-16T17:02:04.264 INFO:journalctl@ceph.osd.7.smithi141.stdout:Nov 16 17:02:03 smithi141 bash[163331]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-7 2023-11-16T17:02:04.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:04 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:04.108+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:04.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:04 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:04.108+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:04.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:04 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:04.108+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:04.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:04 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:04.108+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:04.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:04 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:04.206+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:02:04.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:04 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:04.206+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:02:04.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:04 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:04.206+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:02:04.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:04 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:04.206+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:02:04.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:04 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:04.206+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.131153+0000 front 2023-11-16T17:01:38.131237+0000 (oldest deadline 2023-11-16T17:02:01.631227+0000) 2023-11-16T17:02:04.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:04 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:04.481+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:04.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:04 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:04.481+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:04.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:04 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:04.481+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:04.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:04 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:04.481+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:04.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:04 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:04.481+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:04.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:04 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:04.420+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:02:04.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:04 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:04.420+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:02:04.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:04 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:04.420+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.645505+0000 front 2023-11-16T17:01:30.645605+0000 (oldest deadline 2023-11-16T17:01:52.345350+0000) 2023-11-16T17:02:04.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:04 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:04.420+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.647612+0000 front 2023-11-16T17:01:38.647530+0000 (oldest deadline 2023-11-16T17:02:02.747492+0000) 2023-11-16T17:02:05.094 DEBUG:teuthology.orchestra.run.smithi141:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.7.service' 2023-11-16T17:02:05.173 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:02:05.174 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-11-16T17:02:05.174 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-11-16T17:02:05.174 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.8 2023-11-16T17:02:05.368 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:05 smithi155 systemd[1]: Stopping Ceph osd.8 for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:02:05.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:05.104+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:05.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:05.104+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:05.369 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:05.105+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:05.369 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:05.105+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:05.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:05 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:05.224+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:02:05.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:05 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:05.224+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:02:05.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:05 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:05.224+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:02:05.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:05 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:05.224+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:02:05.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:05 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:05.224+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.131153+0000 front 2023-11-16T17:01:38.131237+0000 (oldest deadline 2023-11-16T17:02:01.631227+0000) 2023-11-16T17:02:05.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:05.402+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:02:05.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:05.402+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:02:05.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:05.402+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.645505+0000 front 2023-11-16T17:01:30.645605+0000 (oldest deadline 2023-11-16T17:01:52.345350+0000) 2023-11-16T17:02:05.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:05.402+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.647612+0000 front 2023-11-16T17:01:38.647530+0000 (oldest deadline 2023-11-16T17:02:02.747492+0000) 2023-11-16T17:02:05.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:05.524+0000 7f8d46a49700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:02:05.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:05.524+0000 7f8d46a49700 -1 osd.8 109 *** Got signal Terminated *** 2023-11-16T17:02:05.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:05 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:05.524+0000 7f8d46a49700 -1 osd.8 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:02:06.014 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:05 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:05.521+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:06.015 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:05 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:05.521+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:06.015 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:05 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:05.521+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:06.015 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:05 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:05.521+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:06.015 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:05 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:05.521+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:06.422 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:06 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:06.133+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:06.422 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:06 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:06.133+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:06.422 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:06 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:06.133+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:06.422 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:06 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:06.133+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:06.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:06 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:06.227+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:02:06.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:06 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:06.227+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:02:06.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:06 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:06.227+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:02:06.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:06 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:06.227+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:02:06.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:06 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:06.227+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.131153+0000 front 2023-11-16T17:01:38.131237+0000 (oldest deadline 2023-11-16T17:02:01.631227+0000) 2023-11-16T17:02:06.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:06 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:06.494+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:06.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:06 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:06.494+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:06.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:06 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:06.494+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:06.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:06 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:06.494+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:06.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:06 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:06.494+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:06.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:06 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:06.420+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:02:06.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:06 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:06.420+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:02:06.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:06 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:06.420+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.645505+0000 front 2023-11-16T17:01:30.645605+0000 (oldest deadline 2023-11-16T17:01:52.345350+0000) 2023-11-16T17:02:06.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:06 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:06.420+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.647612+0000 front 2023-11-16T17:01:38.647530+0000 (oldest deadline 2023-11-16T17:02:02.747492+0000) 2023-11-16T17:02:07.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:07 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:07.114+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:07.367 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:07 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:07.114+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:07.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:07 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:07.114+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:07.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:07 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:07.114+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:07.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:07 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:07.272+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:02:07.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:07 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:07.272+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:02:07.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:07 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:07.272+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:02:07.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:07 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:07.272+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:02:07.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:07 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:07.272+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.131153+0000 front 2023-11-16T17:01:38.131237+0000 (oldest deadline 2023-11-16T17:02:01.631227+0000) 2023-11-16T17:02:07.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:07 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:07.510+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:07.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:07 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:07.510+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:07.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:07 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:07.510+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:07.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:07 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:07.510+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:07.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:07 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:07.510+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:07.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:07 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:07.380+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:02:07.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:07 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:07.380+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:02:07.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:07 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:07.380+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.645505+0000 front 2023-11-16T17:01:30.645605+0000 (oldest deadline 2023-11-16T17:01:52.345350+0000) 2023-11-16T17:02:07.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:07 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:07.380+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.647612+0000 front 2023-11-16T17:01:38.647530+0000 (oldest deadline 2023-11-16T17:02:02.747492+0000) 2023-11-16T17:02:08.415 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:08 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:08.152+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:08.415 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:08 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:08.152+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:08.415 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:08 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:08.152+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:08.416 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:08 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:08.152+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:08.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:08 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:08.268+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:02:08.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:08 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:08.268+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:02:08.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:08 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:08.268+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:02:08.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:08 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:08.268+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:02:08.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:08 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:08.268+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.131153+0000 front 2023-11-16T17:01:38.131237+0000 (oldest deadline 2023-11-16T17:02:01.631227+0000) 2023-11-16T17:02:08.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:08 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:08.479+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:08.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:08 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:08.479+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:08.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:08 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:08.479+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:08.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:08 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:08.479+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:08.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:08 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:08.479+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:08.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:08 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:08.413+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:02:08.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:08 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:08.413+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:02:08.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:08 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:08.413+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.645505+0000 front 2023-11-16T17:01:30.645605+0000 (oldest deadline 2023-11-16T17:01:52.345350+0000) 2023-11-16T17:02:08.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:08 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:08.413+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.647612+0000 front 2023-11-16T17:01:38.647530+0000 (oldest deadline 2023-11-16T17:02:02.747492+0000) 2023-11-16T17:02:09.432 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:09 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:09.132+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:09.433 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:09 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:09.132+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:09.433 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:09 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:09.132+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:09.433 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:09 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:09.132+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:09.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:09 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:09.305+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:02:09.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:09 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:09.305+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:02:09.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:09 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:09.305+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:02:09.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:09 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:09.305+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:02:09.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:09 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:09.305+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.131153+0000 front 2023-11-16T17:01:38.131237+0000 (oldest deadline 2023-11-16T17:02:01.631227+0000) 2023-11-16T17:02:09.657 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:09 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:09.305+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:41.632002+0000 front 2023-11-16T17:01:41.631849+0000 (oldest deadline 2023-11-16T17:02:08.632314+0000) 2023-11-16T17:02:09.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:09 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:09.449+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:09.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:09 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:09.449+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:09.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:09 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:09.449+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:09.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:09 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:09.449+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:09.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:09 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:09.449+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:09.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:09 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:09.431+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:02:09.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:09 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:09.431+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:02:09.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:09 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:09.431+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.645505+0000 front 2023-11-16T17:01:30.645605+0000 (oldest deadline 2023-11-16T17:01:52.345350+0000) 2023-11-16T17:02:09.869 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:09 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:09.431+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.647612+0000 front 2023-11-16T17:01:38.647530+0000 (oldest deadline 2023-11-16T17:02:02.747492+0000) 2023-11-16T17:02:10.396 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:10 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:10.128+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:10.396 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:10 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:10.128+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:10.396 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:10 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:10.128+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:10.397 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:10 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:10.128+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:10.397 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:10 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:10.128+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:10.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:10 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:10.331+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:02:10.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:10 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:10.331+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:02:10.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:10 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:10.331+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:02:10.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:10 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:10.331+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:02:10.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:10 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:10.331+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.131153+0000 front 2023-11-16T17:01:38.131237+0000 (oldest deadline 2023-11-16T17:02:01.631227+0000) 2023-11-16T17:02:10.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:10 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:10.331+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:41.632002+0000 front 2023-11-16T17:01:41.631849+0000 (oldest deadline 2023-11-16T17:02:08.632314+0000) 2023-11-16T17:02:10.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:10 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:10.416+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:10.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:10 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:10.416+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:10.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:10 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:10.416+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:10.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:10 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:10.416+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:10.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:10 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:10.416+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:10.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:10 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:10.416+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:44.285712+0000 front 2023-11-16T17:01:44.285847+0000 (oldest deadline 2023-11-16T17:02:10.185685+0000) 2023-11-16T17:02:10.867 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:10 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:10.394+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.841833+0000 front 2023-11-16T17:01:13.841926+0000 (oldest deadline 2023-11-16T17:01:40.842462+0000) 2023-11-16T17:02:10.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:10 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:10.394+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.143557+0000 front 2023-11-16T17:01:23.143495+0000 (oldest deadline 2023-11-16T17:01:46.643724+0000) 2023-11-16T17:02:10.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:10 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:10.394+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.645505+0000 front 2023-11-16T17:01:30.645605+0000 (oldest deadline 2023-11-16T17:01:52.345350+0000) 2023-11-16T17:02:10.868 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:10 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8[148278]: 2023-11-16T17:02:10.394+0000 7f8d4303f700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.647612+0000 front 2023-11-16T17:01:38.647530+0000 (oldest deadline 2023-11-16T17:02:02.747492+0000) 2023-11-16T17:02:11.367 INFO:journalctl@ceph.osd.8.smithi155.stdout:Nov 16 17:02:11 smithi155 bash[165055]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-8 2023-11-16T17:02:11.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:11 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:11.137+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:11.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:11 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:11.137+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:11.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:11 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:11.137+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:11.368 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:11 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:11.137+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:11.369 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:11 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:11.137+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:11.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:11 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:11.313+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:02:11.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:11 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:11.313+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:02:11.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:11 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:11.313+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:02:11.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:11 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:11.313+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:02:11.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:11 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:11.313+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.131153+0000 front 2023-11-16T17:01:38.131237+0000 (oldest deadline 2023-11-16T17:02:01.631227+0000) 2023-11-16T17:02:11.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:11 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:11.313+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:41.632002+0000 front 2023-11-16T17:01:41.631849+0000 (oldest deadline 2023-11-16T17:02:08.632314+0000) 2023-11-16T17:02:11.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:11 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:11.383+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:11.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:11 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:11.383+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:11.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:11 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:11.383+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:11.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:11 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:11.383+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:11.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:11 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:11.383+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:11.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:11 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:11.383+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:44.285712+0000 front 2023-11-16T17:01:44.285847+0000 (oldest deadline 2023-11-16T17:02:10.185685+0000) 2023-11-16T17:02:12.373 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:12 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:12.181+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:12.373 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:12 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:12.181+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:12.374 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:12 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:12.181+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:12.374 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:12 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:12.181+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:12.374 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:12 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:12.181+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:12.381 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.8.service' 2023-11-16T17:02:12.422 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:02:12.422 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-11-16T17:02:12.422 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-11-16T17:02:12.423 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.9 2023-11-16T17:02:12.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:12 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:12.270+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:02:12.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:12 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:12.270+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:02:12.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:12 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:12.270+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:02:12.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:12 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:12.270+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:02:12.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:12 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:12.270+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.131153+0000 front 2023-11-16T17:01:38.131237+0000 (oldest deadline 2023-11-16T17:02:01.631227+0000) 2023-11-16T17:02:12.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:12 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:12.270+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:41.632002+0000 front 2023-11-16T17:01:41.631849+0000 (oldest deadline 2023-11-16T17:02:08.632314+0000) 2023-11-16T17:02:12.657 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:12 smithi002 systemd[1]: Stopping Ceph osd.9 for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:02:12.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:12 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:12.408+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:12.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:12 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:12.408+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:12.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:12 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:12.408+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:12.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:12 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:12.408+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:12.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:12 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:12.408+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:12.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:12 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:12.408+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:44.285712+0000 front 2023-11-16T17:01:44.285847+0000 (oldest deadline 2023-11-16T17:02:10.185685+0000) 2023-11-16T17:02:13.272 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:12 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:12.962+0000 7f770d670700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:02:13.273 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:12 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:12.962+0000 7f770d670700 -1 osd.9 109 *** Got signal Terminated *** 2023-11-16T17:02:13.273 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:12 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:12.962+0000 7f770d670700 -1 osd.9 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:02:13.617 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:13 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:13.230+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:13.617 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:13 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:13.230+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:13.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:13 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:13.230+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:13.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:13 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:13.230+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:13.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:13 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:13.230+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:13.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:13 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:13.270+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:02:13.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:13 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:13.270+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:02:13.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:13 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:13.270+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:02:13.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:13 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:13.270+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:02:13.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:13 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:13.270+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.131153+0000 front 2023-11-16T17:01:38.131237+0000 (oldest deadline 2023-11-16T17:02:01.631227+0000) 2023-11-16T17:02:13.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:13 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:13.270+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:41.632002+0000 front 2023-11-16T17:01:41.631849+0000 (oldest deadline 2023-11-16T17:02:08.632314+0000) 2023-11-16T17:02:13.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:13 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:13.400+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:13.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:13 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:13.400+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:13.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:13 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:13.400+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:13.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:13 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:13.400+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:13.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:13 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:13.400+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:13.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:13 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:13.400+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:44.285712+0000 front 2023-11-16T17:01:44.285847+0000 (oldest deadline 2023-11-16T17:02:10.185685+0000) 2023-11-16T17:02:14.617 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:14 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:14.275+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:14.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:14 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:14.275+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:14.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:14 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:14.275+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:14.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:14 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:14.275+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:14.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:14 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:14.275+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:14.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:14 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:14.307+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:02:14.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:14 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:14.307+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:02:14.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:14 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:14.307+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:02:14.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:14 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:14.307+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:02:14.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:14 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:14.307+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.131153+0000 front 2023-11-16T17:01:38.131237+0000 (oldest deadline 2023-11-16T17:02:01.631227+0000) 2023-11-16T17:02:14.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:14 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:14.307+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:41.632002+0000 front 2023-11-16T17:01:41.631849+0000 (oldest deadline 2023-11-16T17:02:08.632314+0000) 2023-11-16T17:02:14.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:14 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:14.383+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:14.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:14 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:14.383+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:14.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:14 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:14.383+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:14.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:14 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:14.383+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:14.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:14 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:14.383+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:14.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:14 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:14.383+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:44.285712+0000 front 2023-11-16T17:01:44.285847+0000 (oldest deadline 2023-11-16T17:02:10.185685+0000) 2023-11-16T17:02:15.617 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:15 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:15.233+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:15.617 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:15 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:15.233+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:15.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:15 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:15.233+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:15.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:15 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:15.233+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:15.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:15 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:15.233+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:15.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:15 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:15.308+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:02:15.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:15 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:15.308+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:02:15.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:15 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:15.308+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:02:15.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:15 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:15.308+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:02:15.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:15 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:15.308+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.131153+0000 front 2023-11-16T17:01:38.131237+0000 (oldest deadline 2023-11-16T17:02:01.631227+0000) 2023-11-16T17:02:15.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:15 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:15.308+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:41.632002+0000 front 2023-11-16T17:01:41.631849+0000 (oldest deadline 2023-11-16T17:02:08.632314+0000) 2023-11-16T17:02:15.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:15 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:15.371+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:15.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:15 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:15.371+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:15.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:15 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:15.371+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:15.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:15 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:15.371+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:15.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:15 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:15.371+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:15.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:15 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:15.371+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:44.285712+0000 front 2023-11-16T17:01:44.285847+0000 (oldest deadline 2023-11-16T17:02:10.185685+0000) 2023-11-16T17:02:16.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:16 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:16.255+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:16.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:16 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:16.255+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:16.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:16 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:16.255+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:16.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:16 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:16.255+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:16.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:16 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:16.255+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:16.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:16 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:16.338+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:02:16.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:16 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:16.338+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:02:16.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:16 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:16.338+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:02:16.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:16 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:16.338+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:02:16.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:16 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:16.338+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.131153+0000 front 2023-11-16T17:01:38.131237+0000 (oldest deadline 2023-11-16T17:02:01.631227+0000) 2023-11-16T17:02:16.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:16 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:16.338+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:41.632002+0000 front 2023-11-16T17:01:41.631849+0000 (oldest deadline 2023-11-16T17:02:08.632314+0000) 2023-11-16T17:02:16.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:16 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:16.330+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:16.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:16 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:16.330+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:16.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:16 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:16.330+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:16.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:16 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:16.330+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:16.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:16 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:16.330+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:16.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:16 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:16.330+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:44.285712+0000 front 2023-11-16T17:01:44.285847+0000 (oldest deadline 2023-11-16T17:02:10.185685+0000) 2023-11-16T17:02:16.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:16 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:16.330+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:51.886972+0000 front 2023-11-16T17:01:51.886954+0000 (oldest deadline 2023-11-16T17:02:15.986859+0000) 2023-11-16T17:02:17.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:17 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:17.249+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:17.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:17 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:17.249+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:17.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:17 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:17.249+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:17.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:17 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:17.249+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:17.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:17 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:17.249+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:17.655 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:17 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:17.350+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.223126+0000 front 2023-11-16T17:01:05.223177+0000 (oldest deadline 2023-11-16T17:01:28.123333+0000) 2023-11-16T17:02:17.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:17 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:17.350+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:15.526655+0000 front 2023-11-16T17:01:15.526624+0000 (oldest deadline 2023-11-16T17:01:39.527270+0000) 2023-11-16T17:02:17.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:17 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:17.350+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:20.028585+0000 front 2023-11-16T17:01:20.028661+0000 (oldest deadline 2023-11-16T17:01:45.928456+0000) 2023-11-16T17:02:17.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:17 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:17.350+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.929760+0000 front 2023-11-16T17:01:29.929810+0000 (oldest deadline 2023-11-16T17:01:53.429915+0000) 2023-11-16T17:02:17.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:17 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:17.350+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.131153+0000 front 2023-11-16T17:01:38.131237+0000 (oldest deadline 2023-11-16T17:02:01.631227+0000) 2023-11-16T17:02:17.656 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:17 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9[157528]: 2023-11-16T17:02:17.350+0000 7f7709465700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:41.632002+0000 front 2023-11-16T17:01:41.631849+0000 (oldest deadline 2023-11-16T17:02:08.632314+0000) 2023-11-16T17:02:17.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:17 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:17.295+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:17.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:17 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:17.295+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:17.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:17 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:17.295+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:17.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:17 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:17.295+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:17.767 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:17 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:17.295+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:17.767 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:17 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:17.295+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:44.285712+0000 front 2023-11-16T17:01:44.285847+0000 (oldest deadline 2023-11-16T17:02:10.185685+0000) 2023-11-16T17:02:17.767 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:17 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:17.295+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:51.886972+0000 front 2023-11-16T17:01:51.886954+0000 (oldest deadline 2023-11-16T17:02:15.986859+0000) 2023-11-16T17:02:18.617 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:18 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:18.224+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:18.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:18 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:18.224+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:18.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:18 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:18.224+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:18.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:18 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:18.224+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:18.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:18 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:18.224+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:18.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:18 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:18.224+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:54.655097+0000 front 2023-11-16T17:01:54.655181+0000 (oldest deadline 2023-11-16T17:02:18.154945+0000) 2023-11-16T17:02:18.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:18 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:18.322+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:18.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:18 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:18.322+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:18.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:18 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:18.322+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:18.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:18 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:18.322+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:18.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:18 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:18.322+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:18.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:18 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:18.322+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:44.285712+0000 front 2023-11-16T17:01:44.285847+0000 (oldest deadline 2023-11-16T17:02:10.185685+0000) 2023-11-16T17:02:18.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:18 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:18.322+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:51.886972+0000 front 2023-11-16T17:01:51.886954+0000 (oldest deadline 2023-11-16T17:02:15.986859+0000) 2023-11-16T17:02:19.156 INFO:journalctl@ceph.osd.9.smithi002.stdout:Nov 16 17:02:18 smithi002 bash[288995]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-9 2023-11-16T17:02:19.617 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:19 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:19.249+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:19.617 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:19 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:19.249+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:19.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:19 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:19.249+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:19.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:19 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:19.249+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:19.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:19 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:19.249+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:19.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:19 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:19.249+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:54.655097+0000 front 2023-11-16T17:01:54.655181+0000 (oldest deadline 2023-11-16T17:02:18.154945+0000) 2023-11-16T17:02:19.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:19 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:19.301+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:19.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:19 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:19.301+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:19.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:19 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:19.301+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:19.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:19 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:19.301+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:19.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:19 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:19.301+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:19.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:19 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:19.301+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:44.285712+0000 front 2023-11-16T17:01:44.285847+0000 (oldest deadline 2023-11-16T17:02:10.185685+0000) 2023-11-16T17:02:19.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:19 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:19.301+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:51.886972+0000 front 2023-11-16T17:01:51.886954+0000 (oldest deadline 2023-11-16T17:02:15.986859+0000) 2023-11-16T17:02:20.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:20 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:20.227+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:20.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:20 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:20.227+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:20.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:20 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:20.227+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:20.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:20 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:20.227+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:20.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:20 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:20.227+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:20.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:20 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:20.227+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:54.655097+0000 front 2023-11-16T17:01:54.655181+0000 (oldest deadline 2023-11-16T17:02:18.154945+0000) 2023-11-16T17:02:20.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:20 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:20.316+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:20.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:20 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:20.316+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:20.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:20 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:20.316+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:20.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:20 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:20.316+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:20.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:20 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:20.316+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:20.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:20 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:20.316+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:44.285712+0000 front 2023-11-16T17:01:44.285847+0000 (oldest deadline 2023-11-16T17:02:10.185685+0000) 2023-11-16T17:02:20.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:20 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:20.316+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:51.886972+0000 front 2023-11-16T17:01:51.886954+0000 (oldest deadline 2023-11-16T17:02:15.986859+0000) 2023-11-16T17:02:20.925 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.9.service' 2023-11-16T17:02:21.017 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:02:21.018 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-11-16T17:02:21.018 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-11-16T17:02:21.018 DEBUG:teuthology.orchestra.run.smithi141:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.10 2023-11-16T17:02:21.341 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:21 smithi141 systemd[1]: Stopping Ceph osd.10 for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:02:21.617 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:21 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:21.215+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:21.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:21 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:21.215+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:21.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:21 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:21.215+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:21.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:21 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:21.215+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:21.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:21 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:21.215+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:21.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:21 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:21.215+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:54.655097+0000 front 2023-11-16T17:01:54.655181+0000 (oldest deadline 2023-11-16T17:02:18.154945+0000) 2023-11-16T17:02:21.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:21 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:21.339+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:21.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:21 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:21.339+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:21.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:21 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:21.339+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:21.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:21 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:21.339+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:21.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:21 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:21.339+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:21.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:21 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:21.339+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:44.285712+0000 front 2023-11-16T17:01:44.285847+0000 (oldest deadline 2023-11-16T17:02:10.185685+0000) 2023-11-16T17:02:21.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:21 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:21.339+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:51.886972+0000 front 2023-11-16T17:01:51.886954+0000 (oldest deadline 2023-11-16T17:02:15.986859+0000) 2023-11-16T17:02:21.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:21 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:21.378+0000 7fa81efd4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:02:21.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:21 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:21.378+0000 7fa81efd4700 -1 osd.10 109 *** Got signal Terminated *** 2023-11-16T17:02:21.767 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:21 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:21.378+0000 7fa81efd4700 -1 osd.10 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:02:22.617 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:22 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:22.260+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:22.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:22 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:22.260+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:22.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:22 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:22.260+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:22.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:22 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:22.260+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:22.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:22 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:22.260+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:22.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:22 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:22.260+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:54.655097+0000 front 2023-11-16T17:01:54.655181+0000 (oldest deadline 2023-11-16T17:02:18.154945+0000) 2023-11-16T17:02:22.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:22 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:22.308+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:22.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:22 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:22.308+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:22.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:22 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:22.308+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:22.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:22 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:22.308+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:22.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:22 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:22.308+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:22.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:22 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:22.308+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:44.285712+0000 front 2023-11-16T17:01:44.285847+0000 (oldest deadline 2023-11-16T17:02:10.185685+0000) 2023-11-16T17:02:22.766 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:22 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:22.308+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:51.886972+0000 front 2023-11-16T17:01:51.886954+0000 (oldest deadline 2023-11-16T17:02:15.986859+0000) 2023-11-16T17:02:23.617 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:23 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:23.306+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:23.617 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:23 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:23.306+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:23.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:23 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:23.306+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:23.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:23 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:23.306+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:23.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:23 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:23.306+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:23.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:23 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:23.306+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:54.655097+0000 front 2023-11-16T17:01:54.655181+0000 (oldest deadline 2023-11-16T17:02:18.154945+0000) 2023-11-16T17:02:23.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:23 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:23.355+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:23.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:23 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:23.355+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:23.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:23 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:23.355+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:23.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:23 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:23.355+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:23.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:23 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:23.355+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:23.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:23 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:23.355+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:44.285712+0000 front 2023-11-16T17:01:44.285847+0000 (oldest deadline 2023-11-16T17:02:10.185685+0000) 2023-11-16T17:02:23.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:23 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:23.355+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:51.886972+0000 front 2023-11-16T17:01:51.886954+0000 (oldest deadline 2023-11-16T17:02:15.986859+0000) 2023-11-16T17:02:24.617 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:24 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:24.297+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:24.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:24 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:24.297+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:24.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:24 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:24.297+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:24.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:24 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:24.297+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:24.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:24 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:24.297+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:24.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:24 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:24.297+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:54.655097+0000 front 2023-11-16T17:01:54.655181+0000 (oldest deadline 2023-11-16T17:02:18.154945+0000) 2023-11-16T17:02:24.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:24 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:24.338+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:24.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:24 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:24.338+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:24.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:24 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:24.338+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:24.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:24 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:24.338+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:24.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:24 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:24.338+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:24.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:24 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:24.338+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:44.285712+0000 front 2023-11-16T17:01:44.285847+0000 (oldest deadline 2023-11-16T17:02:10.185685+0000) 2023-11-16T17:02:24.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:24 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:24.338+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:51.886972+0000 front 2023-11-16T17:01:51.886954+0000 (oldest deadline 2023-11-16T17:02:15.986859+0000) 2023-11-16T17:02:25.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:25 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:25.287+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:25.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:25 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:25.287+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:25.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:25 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:25.287+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:25.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:25 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:25.287+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:25.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:25 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:25.287+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:25.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:25 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:25.287+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:54.655097+0000 front 2023-11-16T17:01:54.655181+0000 (oldest deadline 2023-11-16T17:02:18.154945+0000) 2023-11-16T17:02:25.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:25 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:25.385+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:25.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:25 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:25.385+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:25.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:25 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:25.385+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:25.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:25 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:25.385+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:25.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:25 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:25.385+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:25.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:25 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:25.385+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:44.285712+0000 front 2023-11-16T17:01:44.285847+0000 (oldest deadline 2023-11-16T17:02:10.185685+0000) 2023-11-16T17:02:25.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:25 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:25.385+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:51.886972+0000 front 2023-11-16T17:01:51.886954+0000 (oldest deadline 2023-11-16T17:02:15.986859+0000) 2023-11-16T17:02:26.617 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:26 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:26.276+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:26.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:26 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:26.276+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:26.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:26 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:26.276+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:26.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:26 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:26.276+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:26.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:26 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:26.276+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:26.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:26 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:26.276+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:54.655097+0000 front 2023-11-16T17:01:54.655181+0000 (oldest deadline 2023-11-16T17:02:18.154945+0000) 2023-11-16T17:02:26.764 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:26 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:26.361+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:05.978387+0000 front 2023-11-16T17:01:05.978528+0000 (oldest deadline 2023-11-16T17:01:29.378954+0000) 2023-11-16T17:02:26.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:26 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:26.361+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6804 osd.1 since back 2023-11-16T17:01:13.881179+0000 front 2023-11-16T17:01:13.881246+0000 (oldest deadline 2023-11-16T17:01:37.381369+0000) 2023-11-16T17:02:26.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:26 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:26.361+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.782848+0000 front 2023-11-16T17:01:23.782797+0000 (oldest deadline 2023-11-16T17:01:47.882858+0000) 2023-11-16T17:02:26.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:26 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:26.361+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:29.584207+0000 front 2023-11-16T17:01:29.584243+0000 (oldest deadline 2023-11-16T17:01:55.484117+0000) 2023-11-16T17:02:26.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:26 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:26.361+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:38.384909+0000 front 2023-11-16T17:01:38.384957+0000 (oldest deadline 2023-11-16T17:02:04.285003+0000) 2023-11-16T17:02:26.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:26 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:26.361+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:44.285712+0000 front 2023-11-16T17:01:44.285847+0000 (oldest deadline 2023-11-16T17:02:10.185685+0000) 2023-11-16T17:02:26.765 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:26 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10[152641]: 2023-11-16T17:02:26.361+0000 7fa8175c2700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:51.886972+0000 front 2023-11-16T17:01:51.886954+0000 (oldest deadline 2023-11-16T17:02:15.986859+0000) 2023-11-16T17:02:27.264 INFO:journalctl@ceph.osd.10.smithi141.stdout:Nov 16 17:02:27 smithi141 bash[163595]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-10 2023-11-16T17:02:27.617 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:27 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:27.317+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:27.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:27 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:27.317+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:27.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:27 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:27.317+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:27.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:27 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:27.317+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:27.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:27 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:27.317+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:27.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:27 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:27.317+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:54.655097+0000 front 2023-11-16T17:01:54.655181+0000 (oldest deadline 2023-11-16T17:02:18.154945+0000) 2023-11-16T17:02:27.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:27 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:27.317+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6820 osd.7 since back 2023-11-16T17:02:01.656234+0000 front 2023-11-16T17:02:01.656190+0000 (oldest deadline 2023-11-16T17:02:26.955955+0000) 2023-11-16T17:02:28.413 DEBUG:teuthology.orchestra.run.smithi141:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.10.service' 2023-11-16T17:02:28.453 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:02:28.454 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-11-16T17:02:28.455 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-11-16T17:02:28.455 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.11 2023-11-16T17:02:28.617 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:28 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:28.322+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:28.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:28 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:28.322+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:28.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:28 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:28.322+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:28.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:28 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:28.322+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:28.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:28 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:28.322+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:28.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:28 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:28.322+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:54.655097+0000 front 2023-11-16T17:01:54.655181+0000 (oldest deadline 2023-11-16T17:02:18.154945+0000) 2023-11-16T17:02:28.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:28 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:28.322+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6820 osd.7 since back 2023-11-16T17:02:01.656234+0000 front 2023-11-16T17:02:01.656190+0000 (oldest deadline 2023-11-16T17:02:26.955955+0000) 2023-11-16T17:02:28.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:28 smithi155 systemd[1]: Stopping Ceph osd.11 for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:02:29.118 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:28 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:28.804+0000 7fe4a72bf700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:02:29.118 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:28 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:28.804+0000 7fe4a72bf700 -1 osd.11 109 *** Got signal Terminated *** 2023-11-16T17:02:29.118 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:28 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:28.804+0000 7fe4a72bf700 -1 osd.11 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-16T17:02:29.617 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:29 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:29.299+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:29.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:29 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:29.299+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:29.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:29 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:29.299+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:29.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:29 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:29.299+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:29.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:29 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:29.299+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:29.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:29 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:29.299+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:54.655097+0000 front 2023-11-16T17:01:54.655181+0000 (oldest deadline 2023-11-16T17:02:18.154945+0000) 2023-11-16T17:02:29.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:29 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:29.299+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6820 osd.7 since back 2023-11-16T17:02:01.656234+0000 front 2023-11-16T17:02:01.656190+0000 (oldest deadline 2023-11-16T17:02:26.955955+0000) 2023-11-16T17:02:30.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:30 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:30.252+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:30.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:30 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:30.252+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:30.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:30 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:30.252+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:30.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:30 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:30.252+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:30.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:30 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:30.252+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:30.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:30 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:30.252+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:54.655097+0000 front 2023-11-16T17:01:54.655181+0000 (oldest deadline 2023-11-16T17:02:18.154945+0000) 2023-11-16T17:02:30.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:30 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:30.252+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6820 osd.7 since back 2023-11-16T17:02:01.656234+0000 front 2023-11-16T17:02:01.656190+0000 (oldest deadline 2023-11-16T17:02:26.955955+0000) 2023-11-16T17:02:31.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:31 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:31.247+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:31.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:31 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:31.247+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:31.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:31 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:31.247+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:31.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:31 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:31.247+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:31.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:31 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:31.247+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:31.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:31 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:31.247+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:54.655097+0000 front 2023-11-16T17:01:54.655181+0000 (oldest deadline 2023-11-16T17:02:18.154945+0000) 2023-11-16T17:02:31.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:31 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:31.247+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6820 osd.7 since back 2023-11-16T17:02:01.656234+0000 front 2023-11-16T17:02:01.656190+0000 (oldest deadline 2023-11-16T17:02:26.955955+0000) 2023-11-16T17:02:31.620 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:31 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:31.247+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6820 osd.8 since back 2023-11-16T17:02:06.956325+0000 front 2023-11-16T17:02:06.956347+0000 (oldest deadline 2023-11-16T17:02:31.056388+0000) 2023-11-16T17:02:32.617 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:32 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:32.263+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:32.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:32 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:32.263+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:32.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:32 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:32.263+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:32.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:32 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:32.263+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:32.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:32 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:32.263+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:32.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:32 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:32.263+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:54.655097+0000 front 2023-11-16T17:01:54.655181+0000 (oldest deadline 2023-11-16T17:02:18.154945+0000) 2023-11-16T17:02:32.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:32 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:32.263+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6820 osd.7 since back 2023-11-16T17:02:01.656234+0000 front 2023-11-16T17:02:01.656190+0000 (oldest deadline 2023-11-16T17:02:26.955955+0000) 2023-11-16T17:02:32.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:32 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:32.263+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6820 osd.8 since back 2023-11-16T17:02:06.956325+0000 front 2023-11-16T17:02:06.956347+0000 (oldest deadline 2023-11-16T17:02:31.056388+0000) 2023-11-16T17:02:33.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:33 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:33.313+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2023-11-16T17:01:03.544485+0000 front 2023-11-16T17:01:03.544519+0000 (oldest deadline 2023-11-16T17:01:29.444732+0000) 2023-11-16T17:02:33.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:33 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:33.313+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2023-11-16T17:01:23.948127+0000 front 2023-11-16T17:01:23.948145+0000 (oldest deadline 2023-11-16T17:01:45.048299+0000) 2023-11-16T17:02:33.618 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:33 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:33.313+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2023-11-16T17:01:30.249992+0000 front 2023-11-16T17:01:30.249923+0000 (oldest deadline 2023-11-16T17:01:56.149964+0000) 2023-11-16T17:02:33.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:33 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:33.313+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6812 osd.4 since back 2023-11-16T17:01:37.851236+0000 front 2023-11-16T17:01:37.851259+0000 (oldest deadline 2023-11-16T17:02:00.751384+0000) 2023-11-16T17:02:33.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:33 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:33.313+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2023-11-16T17:01:45.953007+0000 front 2023-11-16T17:01:45.953023+0000 (oldest deadline 2023-11-16T17:02:09.453249+0000) 2023-11-16T17:02:33.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:33 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:33.313+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2023-11-16T17:01:54.655097+0000 front 2023-11-16T17:01:54.655181+0000 (oldest deadline 2023-11-16T17:02:18.154945+0000) 2023-11-16T17:02:33.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:33 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:33.313+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.141:6820 osd.7 since back 2023-11-16T17:02:01.656234+0000 front 2023-11-16T17:02:01.656190+0000 (oldest deadline 2023-11-16T17:02:26.955955+0000) 2023-11-16T17:02:33.619 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:33 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11[153732]: 2023-11-16T17:02:33.313+0000 7fe4a08af700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.155:6820 osd.8 since back 2023-11-16T17:02:06.956325+0000 front 2023-11-16T17:02:06.956347+0000 (oldest deadline 2023-11-16T17:02:31.056388+0000) 2023-11-16T17:02:34.867 INFO:journalctl@ceph.osd.11.smithi155.stdout:Nov 16 17:02:34 smithi155 bash[165318]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-osd-11 2023-11-16T17:02:35.752 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@osd.11.service' 2023-11-16T17:02:35.792 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:02:35.792 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-11-16T17:02:35.793 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-11-16T17:02:35.793 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.a 2023-11-16T17:02:36.155 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 17:02:35 smithi002 systemd[1]: Stopping Ceph mds.a for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:02:36.655 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 17:02:36 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-a[162965]: 2023-11-16T17:02:36.317+0000 7f1bebea4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:02:36.655 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 17:02:36 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-a[162965]: 2023-11-16T17:02:36.317+0000 7f1bebea4700 -1 mds.a *** got signal Terminated *** 2023-11-16T17:02:38.655 INFO:journalctl@ceph.mds.a.smithi002.stdout:Nov 16 17:02:38 smithi002 bash[289269]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-a 2023-11-16T17:02:38.915 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.a.service' 2023-11-16T17:02:39.001 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:02:39.001 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-11-16T17:02:39.001 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-11-16T17:02:39.001 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.d 2023-11-16T17:02:39.405 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 17:02:39 smithi002 systemd[1]: Stopping Ceph mds.d for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:02:39.905 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 17:02:39 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-d[161956]: 2023-11-16T17:02:39.500+0000 7f1c3b71f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:02:39.905 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 17:02:39 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-d[161956]: 2023-11-16T17:02:39.500+0000 7f1c3b71f700 -1 mds.d *** got signal Terminated *** 2023-11-16T17:02:41.405 INFO:journalctl@ceph.mds.d.smithi002.stdout:Nov 16 17:02:41 smithi002 bash[289413]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-d 2023-11-16T17:02:41.525 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.d.service' 2023-11-16T17:02:41.609 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:02:41.609 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-11-16T17:02:41.609 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-11-16T17:02:41.610 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.g 2023-11-16T17:02:41.905 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 17:02:41 smithi002 systemd[1]: Stopping Ceph mds.g for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:02:42.405 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 17:02:42 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-g[162576]: 2023-11-16T17:02:42.141+0000 7fcb47e0f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:02:42.405 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 17:02:42 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-g[162576]: 2023-11-16T17:02:42.141+0000 7fcb47e0f700 -1 mds.g *** got signal Terminated *** 2023-11-16T17:02:46.655 INFO:journalctl@ceph.mds.g.smithi002.stdout:Nov 16 17:02:46 smithi002 bash[289557]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-g 2023-11-16T17:02:46.878 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.g.service' 2023-11-16T17:02:46.961 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:02:46.961 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-11-16T17:02:46.962 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-11-16T17:02:46.962 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.j 2023-11-16T17:02:47.155 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 17:02:47 smithi002 systemd[1]: Stopping Ceph mds.j for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:02:47.905 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 17:02:47 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-j[161565]: 2023-11-16T17:02:47.482+0000 7f56433fe700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:02:47.905 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 17:02:47 smithi002 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-j[161565]: 2023-11-16T17:02:47.482+0000 7f56433fe700 -1 mds.j *** got signal Terminated *** 2023-11-16T17:02:50.331 INFO:journalctl@ceph.mds.j.smithi002.stdout:Nov 16 17:02:49 smithi002 bash[289701]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-j 2023-11-16T17:02:50.354 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.j.service' 2023-11-16T17:02:50.437 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:02:50.438 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-11-16T17:02:50.438 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-11-16T17:02:50.438 DEBUG:teuthology.orchestra.run.smithi141:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.b 2023-11-16T17:02:50.764 INFO:journalctl@ceph.mds.b.smithi141.stdout:Nov 16 17:02:50 smithi141 systemd[1]: Stopping Ceph mds.b for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:02:51.264 INFO:journalctl@ceph.mds.b.smithi141.stdout:Nov 16 17:02:50 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-b[156414]: 2023-11-16T17:02:50.798+0000 7f4fd2789700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:02:51.265 INFO:journalctl@ceph.mds.b.smithi141.stdout:Nov 16 17:02:50 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-b[156414]: 2023-11-16T17:02:50.798+0000 7f4fd2789700 -1 mds.b *** got signal Terminated *** 2023-11-16T17:02:55.170 INFO:journalctl@ceph.mds.b.smithi141.stdout:Nov 16 17:02:55 smithi141 bash[163871]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-b 2023-11-16T17:02:55.357 DEBUG:teuthology.orchestra.run.smithi141:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.b.service' 2023-11-16T17:02:55.394 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:02:55.394 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-11-16T17:02:55.394 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-11-16T17:02:55.394 DEBUG:teuthology.orchestra.run.smithi141:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.e 2023-11-16T17:02:55.764 INFO:journalctl@ceph.mds.e.smithi141.stdout:Nov 16 17:02:55 smithi141 systemd[1]: Stopping Ceph mds.e for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:02:56.264 INFO:journalctl@ceph.mds.e.smithi141.stdout:Nov 16 17:02:55 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-e[155733]: 2023-11-16T17:02:55.782+0000 7f0e17aa6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:02:56.265 INFO:journalctl@ceph.mds.e.smithi141.stdout:Nov 16 17:02:55 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-e[155733]: 2023-11-16T17:02:55.782+0000 7f0e17aa6700 -1 mds.e *** got signal Terminated *** 2023-11-16T17:02:59.697 INFO:journalctl@ceph.mds.e.smithi141.stdout:Nov 16 17:02:59 smithi141 bash[164010]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-e 2023-11-16T17:02:59.703 DEBUG:teuthology.orchestra.run.smithi141:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.e.service' 2023-11-16T17:02:59.736 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:02:59.736 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-11-16T17:02:59.737 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-11-16T17:02:59.737 DEBUG:teuthology.orchestra.run.smithi141:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.h 2023-11-16T17:03:00.014 INFO:journalctl@ceph.mds.h.smithi141.stdout:Nov 16 17:02:59 smithi141 systemd[1]: Stopping Ceph mds.h for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:03:00.514 INFO:journalctl@ceph.mds.h.smithi141.stdout:Nov 16 17:03:00 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-h[156072]: 2023-11-16T17:03:00.113+0000 7f2adc3bb700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:03:00.515 INFO:journalctl@ceph.mds.h.smithi141.stdout:Nov 16 17:03:00 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-h[156072]: 2023-11-16T17:03:00.113+0000 7f2adc3bb700 -1 mds.h *** got signal Terminated *** 2023-11-16T17:03:03.481 INFO:journalctl@ceph.mds.h.smithi141.stdout:Nov 16 17:03:03 smithi141 bash[164157]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-h 2023-11-16T17:03:03.668 DEBUG:teuthology.orchestra.run.smithi141:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.h.service' 2023-11-16T17:03:03.703 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:03:03.704 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-11-16T17:03:03.704 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-11-16T17:03:03.704 DEBUG:teuthology.orchestra.run.smithi141:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.k 2023-11-16T17:03:04.014 INFO:journalctl@ceph.mds.k.smithi141.stdout:Nov 16 17:03:03 smithi141 systemd[1]: Stopping Ceph mds.k for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:03:04.514 INFO:journalctl@ceph.mds.k.smithi141.stdout:Nov 16 17:03:04 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-k[155395]: 2023-11-16T17:03:04.087+0000 7f913fac8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:03:04.515 INFO:journalctl@ceph.mds.k.smithi141.stdout:Nov 16 17:03:04 smithi141 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-k[155395]: 2023-11-16T17:03:04.087+0000 7f913fac8700 -1 mds.k *** got signal Terminated *** 2023-11-16T17:03:06.846 INFO:journalctl@ceph.mds.k.smithi141.stdout:Nov 16 17:03:06 smithi141 bash[164297]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-k 2023-11-16T17:03:07.041 DEBUG:teuthology.orchestra.run.smithi141:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.k.service' 2023-11-16T17:03:07.075 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:03:07.075 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-11-16T17:03:07.075 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-11-16T17:03:07.076 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.c 2023-11-16T17:03:07.368 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 17:03:07 smithi155 systemd[1]: Stopping Ceph mds.c for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:03:07.867 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 17:03:07 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[161168]: 2023-11-16T17:03:07.422+0000 7f665712c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:03:07.868 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 17:03:07 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c[161168]: 2023-11-16T17:03:07.422+0000 7f665712c700 -1 mds.c *** got signal Terminated *** 2023-11-16T17:03:11.582 INFO:journalctl@ceph.mds.c.smithi155.stdout:Nov 16 17:03:11 smithi155 bash[165587]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-c 2023-11-16T17:03:11.590 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.c.service' 2023-11-16T17:03:11.627 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:03:11.627 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-11-16T17:03:11.627 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-11-16T17:03:11.628 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.f 2023-11-16T17:03:11.867 INFO:journalctl@ceph.mds.f.smithi155.stdout:Nov 16 17:03:11 smithi155 systemd[1]: Stopping Ceph mds.f for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:03:12.367 INFO:journalctl@ceph.mds.f.smithi155.stdout:Nov 16 17:03:11 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-f[156980]: 2023-11-16T17:03:11.996+0000 7f4dae9c3700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:03:12.368 INFO:journalctl@ceph.mds.f.smithi155.stdout:Nov 16 17:03:11 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-f[156980]: 2023-11-16T17:03:11.996+0000 7f4dae9c3700 -1 mds.f *** got signal Terminated *** 2023-11-16T17:03:15.617 INFO:journalctl@ceph.mds.f.smithi155.stdout:Nov 16 17:03:15 smithi155 bash[165727]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-f 2023-11-16T17:03:15.702 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.f.service' 2023-11-16T17:03:15.736 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:03:15.736 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-11-16T17:03:15.736 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-11-16T17:03:15.736 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.i 2023-11-16T17:03:16.114 INFO:journalctl@ceph.mds.i.smithi155.stdout:Nov 16 17:03:15 smithi155 systemd[1]: Stopping Ceph mds.i for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:03:16.368 INFO:journalctl@ceph.mds.i.smithi155.stdout:Nov 16 17:03:16 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-i[157661]: 2023-11-16T17:03:16.113+0000 7fd07f403700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:03:16.368 INFO:journalctl@ceph.mds.i.smithi155.stdout:Nov 16 17:03:16 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-i[157661]: 2023-11-16T17:03:16.113+0000 7fd07f403700 -1 mds.i *** got signal Terminated *** 2023-11-16T17:03:19.481 INFO:journalctl@ceph.mds.i.smithi155.stdout:Nov 16 17:03:19 smithi155 bash[165874]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-i 2023-11-16T17:03:19.663 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.i.service' 2023-11-16T17:03:19.697 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:03:19.698 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-11-16T17:03:19.698 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-11-16T17:03:19.698 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.l 2023-11-16T17:03:20.071 INFO:journalctl@ceph.mds.l.smithi155.stdout:Nov 16 17:03:19 smithi155 systemd[1]: Stopping Ceph mds.l for 875fcd24-849e-11ee-8dbe-212e2dc638e7... 2023-11-16T17:03:20.367 INFO:journalctl@ceph.mds.l.smithi155.stdout:Nov 16 17:03:20 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-l[156632]: 2023-11-16T17:03:20.070+0000 7f71d768d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-16T17:03:20.368 INFO:journalctl@ceph.mds.l.smithi155.stdout:Nov 16 17:03:20 smithi155 ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-l[156632]: 2023-11-16T17:03:20.070+0000 7f71d768d700 -1 mds.l *** got signal Terminated *** 2023-11-16T17:03:24.780 INFO:journalctl@ceph.mds.l.smithi155.stdout:Nov 16 17:03:24 smithi155 bash[166015]: ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7-mds-l 2023-11-16T17:03:24.785 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-875fcd24-849e-11ee-8dbe-212e2dc638e7@mds.l.service' 2023-11-16T17:03:24.819 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-16T17:03:24.820 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-11-16T17:03:24.820 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 --force --keep-logs 2023-11-16T17:03:29.272 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 --force --keep-logs 2023-11-16T17:03:31.231 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 --force --keep-logs 2023-11-16T17:03:33.192 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-16T17:03:33.273 DEBUG:teuthology.orchestra.run.smithi141:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-16T17:03:33.310 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-16T17:03:33.339 INFO:tasks.cephadm:Archiving crash dumps... 2023-11-16T17:03:33.342 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460916/remote/smithi002/crash 2023-11-16T17:03:33.343 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-16T17:03:33.429 INFO:teuthology.orchestra.run.smithi002.stderr:tar: /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-16T17:03:33.430 INFO:teuthology.orchestra.run.smithi002.stderr:tar: Error is not recoverable: exiting now 2023-11-16T17:03:33.433 DEBUG:teuthology.misc:Transferring archived files from smithi141:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460916/remote/smithi141/crash 2023-11-16T17:03:33.435 DEBUG:teuthology.orchestra.run.smithi141:> sudo tar c -f - -C /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-16T17:03:33.490 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-16T17:03:33.490 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2023-11-16T17:03:33.492 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460916/remote/smithi155/crash 2023-11-16T17:03:33.493 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-16T17:03:33.538 INFO:teuthology.orchestra.run.smithi155.stderr:tar: /var/lib/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-16T17:03:33.538 INFO:teuthology.orchestra.run.smithi155.stderr:tar: Error is not recoverable: exiting now 2023-11-16T17:03:33.539 INFO:tasks.cephadm:Checking cluster log for badness... 2023-11-16T17:03:33.540 DEBUG:teuthology.orchestra.run.smithi002:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-11-16T17:03:33.619 INFO:teuthology.orchestra.run.smithi002.stderr:grep: /var/log/ceph/875fcd24-849e-11ee-8dbe-212e2dc638e7/ceph.log: No such file or directory 2023-11-16T17:03:33.624 INFO:tasks.cephadm:Compressing logs... 2023-11-16T17:03:33.624 DEBUG:teuthology.orchestra.run.smithi002:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T17:03:33.629 DEBUG:teuthology.orchestra.run.smithi141:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T17:03:33.632 DEBUG:teuthology.orchestra.run.smithi155:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T17:03:33.658 INFO:teuthology.orchestra.run.smithi155.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-11-16T17:03:33.660 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-11-16T17:03:33.746 INFO:teuthology.orchestra.run.smithi002.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-11-16T17:07:17.395 INFO:tasks.cephadm:Archiving logs... 2023-11-16T17:07:17.395 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/log/ceph to /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460916/remote/smithi002/log 2023-11-16T17:07:17.397 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-16T17:07:29.068 DEBUG:teuthology.misc:Transferring archived files from smithi141:/var/log/ceph to /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460916/remote/smithi141/log 2023-11-16T17:07:29.069 DEBUG:teuthology.orchestra.run.smithi141:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-16T17:07:55.284 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/log/ceph to /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460916/remote/smithi155/log 2023-11-16T17:07:55.285 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-16T17:08:17.249 INFO:tasks.cephadm:Removing cluster... 2023-11-16T17:08:17.250 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 --force 2023-11-16T17:08:18.795 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 --force 2023-11-16T17:08:19.885 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 875fcd24-849e-11ee-8dbe-212e2dc638e7 --force 2023-11-16T17:08:20.927 INFO:tasks.cephadm:Removing cephadm ... 2023-11-16T17:08:20.928 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-16T17:08:20.977 DEBUG:teuthology.orchestra.run.smithi141:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-16T17:08:20.998 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-16T17:08:21.017 INFO:tasks.cephadm:Teardown complete 2023-11-16T17:08:21.017 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-16T17:08:21.029 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-16T17:08:21.029 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-16T17:08:21.034 DEBUG:teuthology.orchestra.run.smithi141:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-16T17:08:21.042 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-16T17:08:21.176 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-16T17:08:21.187 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-16T17:08:21.188 DEBUG:teuthology.orchestra.run.smithi002:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T17:08:21.193 DEBUG:teuthology.orchestra.run.smithi141:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T17:08:21.195 DEBUG:teuthology.orchestra.run.smithi155:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-16T17:08:21.211 INFO:teuthology.orchestra.run.smithi155.stderr:bash: ntpq: command not found 2023-11-16T17:08:21.212 INFO:teuthology.orchestra.run.smithi141.stderr:bash: ntpq: command not found 2023-11-16T17:08:21.217 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T17:08:21.217 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2023-11-16T17:08:21.218 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 2 +1260us[+1260us] +/- 75ms 2023-11-16T17:08:21.218 INFO:teuthology.orchestra.run.smithi155.stdout:^* hv02.front.sepia.ceph.com 3 6 377 3 -512us[ -521us] +/- 76ms 2023-11-16T17:08:21.218 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 -735us[ -735us] +/- 72ms 2023-11-16T17:08:21.218 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T17:08:21.219 INFO:teuthology.orchestra.run.smithi141.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T17:08:21.220 INFO:teuthology.orchestra.run.smithi141.stdout:=============================================================================== 2023-11-16T17:08:21.220 INFO:teuthology.orchestra.run.smithi141.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 4 +1260us[+1243us] +/- 75ms 2023-11-16T17:08:21.220 INFO:teuthology.orchestra.run.smithi141.stdout:^* hv02.front.sepia.ceph.com 3 6 377 3 -495us[ -512us] +/- 76ms 2023-11-16T17:08:21.220 INFO:teuthology.orchestra.run.smithi141.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 -727us[ -744us] +/- 72ms 2023-11-16T17:08:21.220 INFO:teuthology.orchestra.run.smithi141.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T17:08:21.231 INFO:teuthology.orchestra.run.smithi002.stderr:bash: ntpq: command not found 2023-11-16T17:08:21.258 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-16T17:08:21.258 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2023-11-16T17:08:21.258 INFO:teuthology.orchestra.run.smithi002.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 3 +1265us[+1265us] +/- 75ms 2023-11-16T17:08:21.258 INFO:teuthology.orchestra.run.smithi002.stdout:^* hv02.front.sepia.ceph.com 3 6 377 63 -493us[ -487us] +/- 75ms 2023-11-16T17:08:21.259 INFO:teuthology.orchestra.run.smithi002.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 -712us[ -712us] +/- 72ms 2023-11-16T17:08:21.259 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-11-16T17:08:21.260 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-16T17:08:21.271 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-16T17:08:21.272 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-16T17:08:21.282 DEBUG:teuthology.orchestra.run.smithi002:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-16T17:08:21.585 DEBUG:teuthology.orchestra.run.smithi141:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-16T17:08:21.690 DEBUG:teuthology.orchestra.run.smithi155:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-16T17:08:21.788 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T17:08:21.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T17:08:21.867 DEBUG:teuthology.orchestra.run.smithi141:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T17:08:21.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T17:08:21.902 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-16T17:08:21.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T17:08:21.934 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-16T17:08:21.947 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-16T17:08:21.957 INFO:teuthology.task.internal:Duration was 2856.919847 seconds 2023-11-16T17:08:21.958 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-16T17:08:21.968 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-16T17:08:21.968 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-16T17:08:21.973 DEBUG:teuthology.orchestra.run.smithi141:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-16T17:08:21.975 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-16T17:08:22.020 INFO:teuthology.orchestra.run.smithi141.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T17:08:22.022 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T17:08:22.141 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-16T17:08:22.380 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-16T17:08:22.380 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi002.front.sepia.ceph.com 2023-11-16T17:08:22.381 DEBUG:teuthology.orchestra.run.smithi002:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-16T17:08:22.468 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi141.front.sepia.ceph.com 2023-11-16T17:08:22.468 DEBUG:teuthology.orchestra.run.smithi141:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-16T17:08:22.491 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi155.front.sepia.ceph.com 2023-11-16T17:08:22.491 DEBUG:teuthology.orchestra.run.smithi155:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-16T17:08:22.515 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-16T17:08:22.515 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T17:08:22.520 DEBUG:teuthology.orchestra.run.smithi141:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T17:08:22.534 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-16T17:08:22.663 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-16T17:08:22.664 DEBUG:teuthology.orchestra.run.smithi002:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-16T17:08:22.669 DEBUG:teuthology.orchestra.run.smithi141:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-16T17:08:22.672 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-16T17:08:22.855 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-16T17:08:22.867 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-16T17:08:22.868 DEBUG:teuthology.orchestra.run.smithi002:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-16T17:08:22.949 DEBUG:teuthology.orchestra.run.smithi141:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-16T17:08:22.977 DEBUG:teuthology.orchestra.run.smithi155:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-16T17:08:23.004 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-16T17:08:23.018 DEBUG:teuthology.orchestra.run.smithi002:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-16T17:08:23.023 DEBUG:teuthology.orchestra.run.smithi141:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-16T17:08:23.026 DEBUG:teuthology.orchestra.run.smithi155:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-16T17:08:23.049 INFO:teuthology.orchestra.run.smithi141.stdout:kernel.core_pattern = core 2023-11-16T17:08:23.071 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = core 2023-11-16T17:08:23.108 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = core 2023-11-16T17:08:23.146 DEBUG:teuthology.orchestra.run.smithi002:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-16T17:08:23.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T17:08:23.189 DEBUG:teuthology.orchestra.run.smithi141:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-16T17:08:23.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T17:08:23.207 DEBUG:teuthology.orchestra.run.smithi155:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-16T17:08:23.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-16T17:08:23.224 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-16T17:08:23.237 INFO:teuthology.task.internal:Transferring archived files... 2023-11-16T17:08:23.240 DEBUG:teuthology.misc:Transferring archived files from smithi002:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460916/remote/smithi002 2023-11-16T17:08:23.240 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-16T17:08:23.330 DEBUG:teuthology.misc:Transferring archived files from smithi141:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460916/remote/smithi141 2023-11-16T17:08:23.330 DEBUG:teuthology.orchestra.run.smithi141:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-16T17:08:23.373 DEBUG:teuthology.misc:Transferring archived files from smithi155:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460916/remote/smithi155 2023-11-16T17:08:23.373 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-16T17:08:23.415 INFO:teuthology.task.internal:Removing archive directory... 2023-11-16T17:08:23.416 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-16T17:08:23.421 DEBUG:teuthology.orchestra.run.smithi141:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-16T17:08:23.423 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-16T17:08:23.464 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-16T17:08:23.478 INFO:teuthology.task.internal:Not uploading archives. 2023-11-16T17:08:23.479 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-16T17:08:23.490 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-16T17:08:23.490 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-16T17:08:23.505 DEBUG:teuthology.orchestra.run.smithi141:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-16T17:08:23.507 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-16T17:08:23.522 INFO:teuthology.orchestra.run.smithi141.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 16 17:08 /home/ubuntu/cephtest 2023-11-16T17:08:23.527 INFO:teuthology.orchestra.run.smithi155.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 16 17:08 /home/ubuntu/cephtest 2023-11-16T17:08:23.544 INFO:teuthology.orchestra.run.smithi002.stdout: 524293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 16 17:08 /home/ubuntu/cephtest 2023-11-16T17:08:23.548 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-16T17:08:23.560 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-16T17:08:23.583 INFO:teuthology.nuke:Checking targets against current locks 2023-11-16T17:08:23.610 DEBUG:teuthology.nuke:shortname: smithi002 2023-11-16T17:08:23.611 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T17:08:23.632 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460916', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 16:10:25.045193', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T17:08:23.636 DEBUG:teuthology.nuke:shortname: smithi141 2023-11-16T17:08:23.637 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T17:08:23.656 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi141.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460916', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 16:10:25.046973', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T17:08:23.662 DEBUG:teuthology.nuke:shortname: smithi155 2023-11-16T17:08:23.663 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-16T17:08:23.689 INFO:teuthology.orchestra.console.smithi002:Power off 2023-11-16T17:08:23.690 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-16T17:08:23.706 INFO:teuthology.orchestra.console.smithi141:Power off 2023-11-16T17:08:23.707 DEBUG:teuthology.orchestra.console.smithi141:pexpect command: ipmitool -H smithi141.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-16T17:08:23.714 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-11-16_16:09:17-fs:workload-wip-choffman-62793-distro-default-smithi/7460916', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-16 16:10:25.048360', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-16T17:08:23.718 DEBUG:teuthology.orchestra.console.smithi002:power off output: Chassis Power Control: Down/Off 2023-11-16T17:08:23.718 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:08:23.729 DEBUG:teuthology.orchestra.console.smithi141:power off output: Chassis Power Control: Down/Off 2023-11-16T17:08:23.730 DEBUG:teuthology.orchestra.console.smithi141:pexpect command: ipmitool -H smithi141.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:08:23.744 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2023-11-16T17:08:23.753 DEBUG:teuthology.orchestra.console.smithi141:check power output: Chassis Power is on 2023-11-16T17:08:23.768 INFO:teuthology.orchestra.console.smithi155:Power off 2023-11-16T17:08:23.769 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-16T17:08:23.789 DEBUG:teuthology.orchestra.console.smithi155:power off output: Chassis Power Control: Down/Off 2023-11-16T17:08:23.790 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:08:23.812 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2023-11-16T17:08:27.745 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:08:27.762 DEBUG:teuthology.orchestra.console.smithi141:pexpect command: ipmitool -H smithi141.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:08:27.812 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:08:27.865 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2023-11-16T17:08:27.877 DEBUG:teuthology.orchestra.console.smithi141:check power output: Chassis Power is on 2023-11-16T17:08:27.923 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2023-11-16T17:08:31.872 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:08:31.894 DEBUG:teuthology.orchestra.console.smithi141:pexpect command: ipmitool -H smithi141.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:08:31.925 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-16T17:08:31.997 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is off 2023-11-16T17:08:32.008 DEBUG:teuthology.orchestra.console.smithi141:check power output: Chassis Power is off 2023-11-16T17:08:32.037 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is off 2023-11-16T17:08:32.098 INFO:teuthology.orchestra.console.smithi002:Power off completed 2023-11-16T17:08:32.109 INFO:teuthology.orchestra.console.smithi141:Power off completed 2023-11-16T17:08:32.137 INFO:teuthology.orchestra.console.smithi155:Power off completed 2023-11-16T17:08:32.247 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 5b73517e0e2fa132ea79290e5fc06ea5349b2288 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} duration: 2856.9198467731476 failure_reason: 'Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi002 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e4a743ad7fd1422ebd6c4dc950a21a303279207 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh''' flavor: default owner: scheduled_choffman@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f9d9071e83e849048fc413a6cc6d58cd status: fail success: false 2023-11-16T17:08:32.248 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-16T17:08:32.343 INFO:teuthology.run:FAIL